Ep 25: James Beswick AWS Senior Developer Advocate

Talking Serverless: With Ryan Jones and James Beswick

Ryan Jones: Welcome to the Talking Serverless podcast. I'm your host, Ryan Jones, and I'm joined today by one of the most well-known serverless developer advocates, James Beswick. James is a serverless builder, Product Manager and currently a four-time AWS certified serverless evangelist, also known as a senior developer advocate for Amazon Web Services. James has had a long career in the IT field and is a frequent speaker at conferences, meetups, community days and AWS events. He is well known for his blogs, workshops, proof of concept applications and white papers surrounding serverless and broad AWS concepts.


Background
Ryan Jones: Q: How did you get started in serverless and how did you pursuits help you blossom into where you are today?
James Beswick: I started 19/20 years ago, and I've been working through building lots of applications over the years, first as a software developer, then as a product manager. I got interested in the challenges of IT and building things. Before I joined AWS, I ran a company that built applications for start-ups. Through that, I discovered that we were building lots of applications over and over with common, boilerplate code. And all of this was on Elastic Beanstalk on ECS. So after doing 30/40 applications, we were finding that the management of these instances was taking up a lot of time.
I came across serverless completely by accident; I discovered it at serverless comm, while I took part in a hacking day where I went through and built some applications. I saw some really eye-opening things from people like Ben Kehoe, and I started exploring and realised there was a huge opportunity to simplify what we were building. That was three years ago, which seems like quite a long time now. Since then, I have switched everything over to building serverlessly. I trained a lot of our developers in how to build things this way. I ended up writing so many blogs and doing things in the community that I was offered the role I am now in. I took it because I thought it was a great opportunity to continue talking about serverless with the community. So it was totally by accident, but I'm really happy to be where I am.

Q: Which serverless conference was it that introduced you?
James Beswick: I think it was 2017 from memory, in New York. It was really interesting, because I remember seeing the ACloudGuru brothers talking about how they build their platform serverlessly. And up until that point, all I knew about it was that you could do image resizing, because I've seen it at ReInvent. I couldn't believe the capabilities of the platform when I started digging into it more and more. So for start-ups that we were working with, one of the great benefits was that scalability is handled for you. We found that about 2 in 10 of the start-ups we were working with had success with their applications, but we were never sure which 2 in 10 would be successful. When we are building these MVPs, part of the problem is that building scalability into MVP is really a contradiction in terms, because if you're doing that you're not building an MVP. But then if the product takes off and becomes very popular and you haven't built scalability, in then you end up scrambling to rewrite things. And so this was a really good fit for us; we could build these applications and if they were suddenly popular, the scalability side of it was managed automatically. That was the point where I realised the value of serverless.

General Serverless
Q: When you started adding serverless into start-ups, what was the result in terms of building scalability and lightening overheads?
James Beswick: There was definitely a learning curve we had to go through, because you have to do a few things differently. Not many people were talking about the process of learning at that time, so we're right in the cutting edge of learning it. That's when I realised there were some real changes ahead. One of the biggest changes is writing a lot less code; you're not writing authentication layers, or database connectivity libraries, or anything like that. Getting used to writing less code takes a bit of a practice. Then you start to get more into the idea of building things that jump across a range of different services, and you can see that you're more in the distributed architecture design side of things. So there were definitely some steps we had to go through to learn how to do it properly, but it was truly amazing. Some of the applications we put out that went to medium scale fairly quickly just worked seamlessly out of the box. To me that was an amazing payoff, because initially when you're learning all these things, like with all technology, you're never quite sure how it's going to work and release until you sit and practice.

Q: What makes less code take more time to adjust to?
James Beswick: I think a lot of the time you get used to building things with frameworks you've become very comfortable with, and you take for granted that those frameworks are going to do certain things. You develop practices that you feel comfortable with, and when you've got these new services coming in that can do that work for you, it feels a little bit counterintuitive. But in a well designed lambda project functions are typically fairly small, and you get some real benefits from that, including easier testing. As a developer, it's easier to see what 50-100 lines of code are doing instead of poring through tens of thousands. Also, when you get really good at it, you can mode code from project to project very easily. So, if you had one function for one specialised piece of work, it'd be fairly easy using environment variables to lift and shift that other project, which really makes the code reuse side of it very beneficial.

Q: How do you think serverless and the cloud might look in one or two years from now?
James Beswick: Over the last few years the pattern has been that IT teams are under enormous amounts of stress. We don't typically get lots of headcount, but we do get more features to build and more systems to manage this. Generally in the industry there has been a trend towards having developers learn more and more. You have this concept of the full stack developer, which is essentially, a person who knows a lot of different things. So we have to do things that make it easier for these developers to do more without working 24 hours a day. When we talk to customers about what they want to see in lambda and other services, a lot of what we are driven towards is lifting that load and making things easier for people so they can build things without having to add more people or infrastructure. And so really, when you look at what lambdas have been releasing over the last couple of years, it's been constantly driving towards more integrations, better developer experience, and better tooling. I think those are the trends that you'll see continuing.

AWS Role
Q: Do you travel as part of your day-to-day role as senior developer advocate for AWS, or is your work mostly remote at the moment?
James Beswick: No, we're not travelling at the moment, everything is virtual. Before all of this we went to lots of conferences, summits, events, and local meetups. These are a great way to meet people in the community and have conversations with developers. Now, everything is moved to an online environment, and there's some pros and cons to that. A pro is we can go to meetups anywhere. I recently spoke to a Brazilian meetup group, and I wouldn't have necessarily had that opportunity when we were travelling. So we can meet more people through the virtual events, and talk to people more directly using all sorts of tools. But it is definitely a different environment right now; I think everybody's just trying to get to grips with how to do this. We all hope that the travel comes back sometime soon.

Q: What is the serverless developer advocate role in virtual meetups? Have you always leaned towards community actions, and what led you to writing your blog articles?
James Beswick: I've always been a writer. When I was a teenager I used to write game reviews for magazines where I lived in England. I've been an early blogger for a long time, and I've always found that I best connect with an audience when I want to discuss something. So for me, a lot of technology is about exploring things you don't understand well; learning about the best way to do these things and then talking about them in a blog to engage with the audience. When I discovered that the developer advocate role actually existed it was almost a perfect, dream job, because I enjoy this. I get the opportunity to play with some of the world's greatest services, build things that are really interesting and talk to developers. It's not a marketing job where I'm telling people what to use; I actually get to talk to developers about what works, what doesn't work, and what they would like to see in the products. I really engage quite a lot of people in this role.

Q: You're a huge writer on the AWS blog. How can people find you on there?
James Beswick: AWS has a number of different blogs. There's one called the Compute blog, and you can get to it from my Twitter account, which is just Twitter/JBESW. The compute blog represents lots of different writings by DAs, account managers, systems architects and all sorts of people in AWS who want to write about different topics. Quite a few of my team write quite frequently for the Compute blog, and it enables us to talk about different subjects, how services fit together, what we're currently thinking about certain things and also link out to code examples. When I saw this job of 18 months ago, I wanted to build some large format examples. I didn't feel that had been done before, or not very frequently, so I started down the road of building some of these projects I've done recently, like Ask Around Me and Happy Path, that show you how you can build full scale examples using this. A lot of this actually comes down to planning, because obviously there's a lot of writing and coding involved, and you have to try and organise your time to do it. My aim is to have something that goes out each week, that is talking about topics that I see as being relevant at the time.

Q: How do you break down your time to put out that much content? Do you have any tips for time management?
James Beswick: I've always been a whiteboard person with post-its. I find as long as I have a post-it with a task on it and that goes onto the board, I won't forget to do it. But I'm also pretty rigid in terms of how I manage my time. When I had the company before this, how you manage your time really matters when you're building projects. You want to be on time and on budget. So I bring a lot of that with me, but AWS is a very busy place with lots of different things going on. We're involved in launches, internal testing, and lots of meetings about what we'd like to see in products according to what we're hearing from customers. We also engage a lot of customers; they email me and send me DMS all the time on Twitter. So a lot of the time, we're having to balance lots of different things. I find as long as I'm making progress in my larger scale projects at a pace where I'm staying relatively on track, it's okay, but there are definitely a lot of different things going on at the same time. So you have to be fairly flexible and be able to switch tasks fairly frequently.

AWS EventBridge
Q: AWS EventBridge came out and people were raving over it. What is EventBridge?
James Beswick: EventBridge is a really interesting service. It was launched at the New York summit last year, and it's an evolution of CloudWatch Events, which has been around for years. EventBridge adds new features/functionality and extends the capabilities of the program further. All the future development is going into EventBridge. Essentially, it's an Event Bus at core. You can use and Event Bus to decouple your applications and microservices very seamlessly, and it's one of those things where you have to play with it a little bit to realise how powerful it is. If you work in a medium or large-scale project, you realise how difficult it is to coordinate all these different pieces and microservices. Very quickly, you need some sort of messaging layer that can decouple these things. We've always had things like SQS and SNS but EventBridge really takes it to a whole new level, because you can ingest events from third parties like SaaS providers, other AWS accounts, or other Event Buses. You can also publish events from your own applications, so it actually becomes a very important extensibility layer. If you're building larger applications, your applications can emit events into Event Buses, and other people who want to integrate with your application can do so from the Event Bus without having to directly integrate their application. So you have an Event Bus in your account automatically as default, and you configure rules on that bus, which filter the ways of filtering events that you care about. You set up targets for those rules, so if an event happens that you're interested in, it can invoke a target such as a lambda function. There's not much to it beyond that; the events are simply JSON, and the pattern matching happens with JSON as well.
A lot of people who work in enterprises have seen some of the things before with Event Buses, so this is familiar but different. Back in the time when Event Buses were popular, corporations often saw them as a single point of failure. Event Bridge is a highly available service that currently handles trillions of events a month, so it's on a massive scale. We've also introduced other features where, if you look at some of the problems in managing events or keeping track of schemas, we introduced a service called schema registry that automates some of that, and lets you download code bindings directly into ID so you can automate the creation of classes directly from events. It's actually fairly simple to get your hands on and use some of the some of the sample applications. It's not difficult to get started, but the capabilities are really pretty astounding.

Ryan Jones: "... having a centralised location where people can understand how this whole thing works reminds me a lot of black box development; somebody creates an easy to instance SSH into it and downloads a whole bunch of stuff, then that person leaves six months later and no one has any idea what's going on."

James Beswick: JSON is very simple to understand and read when you look at it. It has a complete inventory of all the events you see within AWS, so it's a very simple place to go and find what an event looks like. For example, you can pull one up from S3 and take a look at it. JSON is actually a much simpler way to to manage development, especially if you're working with lambda, where your unit testing can become very robust through this approach.

Ryan Jones: Q: You recently wrote an article, 'Reducing Custom Code by using Advanced Rules and Amazon EventBridge'. Can you talk a bit about that?
James Beswick: Yeah, so we've had filters and services like SNS before, but the filters within the EventBridge are fairly sophisticated. You can take an event coming in, an S3 event for example, where you then invoke a lambda function and you might only care about objects that have a certain naming convention. You can do that currently with an S3 to lambda integration, but what you can't do is really complicated things. Imagine having a dozen buckets where you put all your sales in a different bucket via the month. Each time an object arrives, you want to invoke a lambda function, but the bucket may not exist yet. Because the March/April/May bucket isn't there, you can use EventBridge to manage that kind of integration, and suddenly you're decoupling the buckets and objects from downstream processing. It gives you a lot of capabilities. Anything that's within the S3 event, which includes things like principal ID, IP address, or object size, can be used by these filters and EventBridge. It makes it much easier to then figure out which events you care about, and you're not charged to those events because any events coming into EventBridge from AWS services are free. It's much more cost-efficient to do the filtering at the EventBridge stage, instead of passing off to the lambda function first.

Q: I was reading that you can have five targets per rule. How does that compare with SNS or SQS? When would you use one over the other, or would you use them together?
James Beswick: When you look at SQS, SNS and EventBridge they have some apparent similarities, so there are times when you might use one or another it doesn't make much of a difference. SNS, Cloud and EventBridge are fairly similar in many respects, but SNS has a different scale and list of targets. SNS has up to 12.5 million subscribers in a single topic, whereas EventBridge is limited to 5 million, but SNS has a lower latency than EventBridge when it's passing across these events. What's interesting is you can use them in combinations; there's nothing stopping you making an SNS topic a target for EventBridge. Once you start getting into these advanced patterns, using the combination and bringing cues in that to build real resilience in the architecture, the three together become very powerful.

Q: Does Sam support EventBridge out of the box?
James Beswick: Yes, Sam supports all three of the services. In some of those articles you're referring, I put some links to GitHub repos that people may find useful. I've put Sam templates on everything, so you can very quickly copy one, clone the repo and see how it works. The critical thing for Sam is setting up the rules and permissions correctly, so if you see that from a basic template it's much easier to start, rather than building everything from scratch.

Q: Another article that you wrote on EventBridge, was 'Decoupling larger Applications with Amazon EventBridge. You talk about decoupling these larger applications; can you give an idea of what that looks like?
James Beswick: There's a couple of really interesting ideas here that both hit me around the same time. The first was when you build these cloud formation applications or Sam applications, typically you're required to deploy theS3 bucket, they both have to be in the same stack when you're launching them. Often that's not really going to be possible, especially in enterprise situations where resources already exist. You want to be able to combine your application with those, and decoupling through EventBridge makes that possible. You can work with pre-existing, real resources just for part of your application.
The second idea was based upon my experience; you're never quite sure what you'll be building as a developer three months from now. You build these architectures at the time, and you think you've covered all the likely features that will be needed, but then you find a couple of months away requests that you've never heard of before, and it changes the direction of what you're building. This type of approach lets you emit events early on in your designs, for consumers that may not even exist yet. You may be publishing events into the ether. Later on you'll be very happy you did, because you can essentially build these smaller microservices from those events. I got really interested in this idea of decoupling, simply because it means you're building things that are scoped down in a much smaller way. They're much easier to manage. You're not always dealing with massive amounts of infrastructure just to roll out an application.
The example in the in the blog post was from an application that I'd published a few weeks previously, showing how you could index enterprise documents into Elastic Search. Even as a fairly simple application, you have lots of extensibility options at EventBridge. In my example, I only indexed three different types of file, but you could start to build an adaptive framework based upon EventBridge, where if you want to add new different types of file, it can be done easily. The architecture from the original diagram actually becomes much easier to understand and maintain when you bring EventBridge into the middle; it's this coordinating feature around different services that makes the application work.

Q: You mentioned you're going to share the GitHub repos. Are there any other resources that people can look at? Are there any online courses on EventBridge yet?
James Beswick: At the moment we have a number of different learning paths. They're collections of different videos, articles, and resources, and we have one for EventBridge. For this S3 to lambda pattern, I wrote a series on those two, we've started to build videos, articles and GitHub repos that you can refer to. In terms general learning resources, there's lots of information you can also find on the serverless website, which is AWS.amazon.com/serverless. We're starting to build more and more learning resources on those pages there.

Q: What do you think the adoption rate looks like for? Is it where you thought it would be, comparing a few years ago to now? What do you think might increase adoption in the future?
James Beswick: Adoption has been staggering. Just in the last couple of years at AWS, there are hundreds of 1000s of customers using lambda every week, running trillions of invocations. The scale is absolutely staggering. It's interesting when you look at the distribution, because we see large companies like Dunelm, or Fannie Mae using it for enormous scale applications. We've got machine learning shops using it all the time, people using it for data processing, web app development, front end or back end. And then you've got smaller start-ups that are using it for building things quickly and being able to experiment without spending an enormous amount of money. Globally, we see patterns where there's accelerating adoptions in places like South America and in Europe. There are lots of people using it in the US, but it's really accelerating outside of the US. It's just amazing how quickly it's changing. The hardest problem is keeping up with everything that's happening, because AWS is releasing so many features and so many different services that benefit serverless developers. A lot of it is just making sure that you can keep pace with the rate of change.
Within the community itself, we've got some really amazing community partners, when you look at Serverless Days and what Anne Stanley has built there with the organisation, and some of the newsletters like Jeremy Daily's dailies off by non newsletter, which is really fantastic. Then you look at what the AWS service heroes have been doing contributing to the community. I think they've all done an amazing job keeping us in contact with developers and helping new developers who are just coming into serverless really keep up with what's going on. It's become a lot easier today than it was three years ago to build these applications. The tooling is a lot better, we have better observability capabilities. When you look at tools like AWS and Sam, it makes it truly easy to deploy and manage these sorts of applications. The growth curve is definitely up and to the right, and this acceleration of adoption really shows no sign of slowing down. I think within a few years it will become the standard way to build most applications.

Conclusion
Q: If this is what is going to be the standard way of building things in the cloud, what happens to the word serverless? Do we have to keep juggling all these terms?
James Beswick: In some respects, the term serverless is helpful and unhelpful. Because we in the community know generally what serverless is, even though there's some debate. We know that you need to have a compute layer like lambda, we know that generally you have this pay-for-value model where, if you're not using it, you're not paying for it. And we know there are features such as things being fully managed or automatically scaled for you. Obviously you start to bring in more services that are fully managed, and some of them are not always AWS. We've got customers and partners like Orth Zero and Stripe who these applications reach across and integrate with; those SAS providers that more of a managed architecture. I'm sure we'll start to see the growth of applications on Fargate as well, with lambda becoming closer as time continues. So serverless will become one of those terms like electronic was in the 1980s; we know what it means as we're starting to adopt to it, but once you're there, the term might become redundant because it's the way that you're doing everything.

Q: Where can we find you?
James Beswick: You can DM me anytime on Twitter. My handle is JBESW. You can also email me at Amazon. I'm JBeswick@amazon.com. I really enjoy hearing from everybody. If you have any thoughts or ideas about what we should be building or problems you're having or just interesting things you've built, feel free to drop me a line.

Ryan Jones: Awesome, thanks so much for coming on. To those listening, this has been the Talking Serverless podcast with Ryan Jones. If you like our show and want to learn more, check out talking serverless.io. Please leave us a review on iTunes, Spotify or Google podcasts, and join us next time as we sit down with another fantastic serverless guest!