Ep 26: Eric Johnson AWS Senior Developer Advocate

Ryan Jones: Welcome to the Talking Serverless podcast. I'm your host, Ryan Jones. I am joined today by Eric Johnson, a senior developer advocate at Amazon Web Services based out of Northern Colorado. Eric is a fanatic about serverless and helping developers understand serverless. He believes that serverless technologies introduce a major paradigm shift in building and running applications at a massive scale, and that moving to serverless results in less code and infrastructure overhead.

Background
Ryan Jones: Q: You began advocating for cloud around 2016. How did this transition happen and how did you get into the cloud to begin with?
Eric Johnson: That's an interesting story. For people who don't know this, I was a pastor for 12 years, so I was a public speaker, but on the side I had always done technology. As I stepped down, I started moving technology into the forefront. I learned at night, I have no formal training in computers, I do not have a pure degree. I have a Master's in Divinity, which is quite the opposite. I started out as a designer. Turns out, I was a really poor designer, so I moved into development, and just fell in love with development in general. In 2014, I was working for a company called Show Pitch as a Senior Architect, Front-end Architect and Back-end architect, and we were doing a lot of different things. I remember coming across the announcement for serverless, and I just sat down and decided there was no other way. That was the way to do it, it seemed, and I had already become a cloud fanatic. I really believed, why would you have your own data centre and build your own system when you can move that heavy lifting off to someone else?
In every company I worked for, I was the R&D guy. Hey, go figure out the cloud. Hey, figure out how to do this, etc. So I was always kind of a cloud admin. So in 2016 I went to work for a company called Rackspace as an AWS Evangelist. Rackspace was stepping into the cloud management, cloud support arena, so they hired me to talk about AWS and that was a dream come true. I'm an AWS fanatic and have been since 2006 when I was doing some early stuff on AWS. The rest is history. I got into the cloud and I saw serverless, and that was my target from then on. Coming forward to 2021, that's when I found you, I met with Chris and I had always been at Rackspace. Even though I was the AWS evangelist, I'd always been the serverless guy. Oh, you want to do serverless? Talk to Eric. Because I just loved it. It was the way to go. And so it was just a very natural progression for me to move to a targeted serverless Evangelist or DA role.

Q: How do you think that being a pastor in your former life has affected or supported your work?
Eric Johnson: I speak sometimes on the side, so I'm still involved in my church. I have five kids, so I had to step down and do something else, to make sure my family's getting more of my attention. Pastoring is a hard job. But the Developer Advocate role is this marriage of technology and speaking, and the way I see it is, there are speakers who are more technical. My speaking ability is my higher skill of the two skill sets. So I'm higher on the speaking, and I try to stay as technical as I can. Whereas we have some folks that are crazy technical that can also speak very well.
Q: How did you build up this knowledge around serverless? What types of things were you building?
Eric Johnson: It started internally. I was looking at some ways we were tracking things, and I knew, I could do this better. There are 300 million applications out there, so probably not better, but I didn't like the way we were tracking something. I always loved serverless so I decided, you know what, I'm gonna apply that. So on my off time at night and on weekends, I started building a tracking system using just serverless, and I got it to prove a concept. I went to my boss at the time, and I said, hey, I think we can do this better. Here's a proof of concept I've got. He loved it. We ended up hiring a team of developers that made my code much better, but stayed serverless with it. I believe that serverless can handle most workloads. There are some edge cases where I would look at others, but for most workloads, I feel like serverless is a good fit. That's where I started; let me try this using serverless to do this perfect concept. And then if I feel like I need to step out of serverless, I can for some of this, but the perfect concept idea of serverless is so fast. I'm not spinning up instances, I'm not spinning up databases, I'm just writing code. I did this proof of concept in about a week, and most of that was me learning how to code better and figure out how I want to do some Front-end stuff.
So what we would do is say, we built that in serverless, where else can we do it? I worked as a solutions architect. So I worked all across the AWS spectrum, but because serverless is where I became the 'pinpoint guy', I got taken to meetings and I got to help a lot of customers look at serverless workloads more. Usually, it was a case of integrating serverless into what they were already doing, like taking some ops stuff or doing someone-off CRON work.

Ryan Jones: "We've seen serverless mature quite a lot over the past few years."

Q: When you're talking about most workloads fitting into serverless, how do you explain that to people?
Eric Johnson: That's the challenge, right? I think one of the struggles that we have and a lot of the pushback we get is from it not being not the way people did it. They think they need to bring in a lot of stuff, which you don't need to with serverless. I'm going to write my own authentication, I'm going to write my own pub sub, I'm going to write my own whatever, versus understanding. If you think of putting 100 people in a room and defining serverless, we have lots of different angles, but I look at it as using managed services and not reinventing the wheel. You put together managed services that build an application.
It comes down to awareness; I didn't know that a bit bridge did what it does, I didn't know SQL existed, I didn't know SNS could fan out etc. So they write a lot of that code, or write a lot of that functionality into their code, and then it doesn't really fit. It takes them longer than the 15 minute time frame to do what they're trying to do. So the first question I often ask is, how are you approaching it? How are you doing that? Explain to me what you're doing, talk me through it. It is very linear; I did this, then I did this, then I did this. There's 17,000 lines of code and my lambda to handle all this. And you're right, that doesn't fit in serverless. But really, that shouldn't fit anywhere.

Ryan Jones: "... serverless forces you to expose different problems that may have been pushed underneath the covers"

Code VS No Code
Q: I've talked to some people on the podcast and they've described the future as no code, or any code is overhead. What do you think about that?
Eric Johnson: I don't know if I'd say any code is overhead. If you follow me at all, you'll know that I push the direct integrations from API gateway. So instead of running a lambda function between API gateway and DynamoDB, I often opt for just connecting to DynamoDB. You remove the code, but essentially what you're doing is moving that code or moving some of that logic to API gateway. So it's not really a removal of code, so to speak. It lessens the code, but I think a better way to say it is it's up to you to decide whether it's better or not. Another way that I would say that is reducing code and offloading tasks to services that are built for those tasks.
Any time I'm writing code to reinvent the wheel, that's wasted code. The code I'm writing should only apply to what differentiates me and my application from the company next to me. I shouldn't be mastering queueing systems, I shouldn't be mastering databases, I shouldn't be mastering AWS zero. I should just use the services that are in place. What I should be mastering is the code that makes my app do what makes it unique.

Ryan Jones: "... when you're focusing on not reinventing the wheel, you're focusing on what matters."

Q: So what does that look like, for those for those listening that don't do API integrations or API gateway integrations to other services? Is that a good experience?
Eric Johnson: VTL to replace all code? No, you want to delete that. I have a URL shortener. We can post a link or something, but it's a three part blog. It builds a URL shortener, because the world needs another URL shortener right, that's entirely API gateway, and DynamoDB. Then we put CloudFront in front of it, and it works great. It's lightning fast, very reliable and very secure. But the VTL is not pretty, so there's a balance there. And Jinair, my director of lambda services, says you shouldn't use lambda to transport, you should use lambda to transform. I really hold to that when I evaluate; what am I doing with the data? If the data is coming in and I'm simply putting it in a format that can be injected straight into DynamoDB, then I really don't need a lambda function to do that. But if I'm doing a little more than that, like doing some logic, I'm not going to waste time trying to figure out all the VTL for that. So there's a balance, right?
It all comes down to what makes us as developers efficient, and what makes our applications efficient. So directly integrating into DynamoDB is going to save me a lambda invocation, and when you're talking about millions of hits, that's a little bit of money saving, and you're also reducing a very minor bit of latency. But is that worth the large VTL files that it can sometimes get? In some places I say yes, in some places I say no. That's the discussion I have in my head when I'm approaching this is; is this more efficient for me? Is it more efficient for the application as a whole? Sometimes it's yes but oftentimes it's no.

Examples
Ryan Jones: You mentioned that you did a VTL. Crazy project for shortening URLs. I've played quite a lot with VTL for a while now, and I know how quickly those things can get out of hand.
Eric Johnson: Sometimes it's not about no code, but about when to code. Part of that pattern wasn't so much driving the removal of the lambda function; I'm a huge advocate of lambdas and lambda functions. But what I'm an advocate for is using them in the right place. So for instance, I do a presentation called Thinking Asynchronously. The whole idea behind this is, if we think about when we process data, do we process that data between our API gateway; our API gateway is the front door, right? For me it's going to be DynamoDB, so we'll use that in our example. Do I process that data before I put it into Dynamodb? And here's the risk there. I've said this 100 times. The most brutal part of an application is going to be my code. That's just the truth. 100 developers in the room, I might be in the top 50, I'm certainly not in the top 20. I do a good job but it's not a guarantee. It's not going to be as well tested and as bulletproof as what's behind SQS, Dynamo or API gateway.
So what I'm doing is introducing a brittleness between receiving the data from the client and storing that data. What I tend to do is I say, hey, let's store the data first and get a response back from the customer to say they have the data. And now I'm going to do some work for them using streams. This is a DynamoDB thing, but you can also SQS or whatever. Using DynamoDB streams, I trigger a lambda to do whatever processing I need to do on that data. I have an application where I synchronously translate; I have four or five cultures and I give it a string. It'll create five requests against Amazon to translate that, then I'll use Polly to convert it to audio, and then I'll save it to a bucket. So once I get the response back, the whole time my lambda is spinning and doing that, and there's always a chance for something to break in my code. If I shoot that data into DynamoDB and start triggering actions after the fact, if my code breaks somewhere, I've still got the customer data in DynamoDB. I don't have to go back to the customer and say, hey, I lost your data, you want to trust me again? I promise I'll get it right this time. So you're building a robustness to your applications this way, and it really makes us think asynchronously versus synchronously.

Q: Is that API gateway lambda to DynamoDB, or is this service integration?
Eric Johnson: Yes, in that example I'm going direct to DynamoDB. I'm not doing much modification. I'm just doing a CRUD, right? Create, read, update, delete. Then my heavy logic after the fact can be done with a lambda function, based on the string.

Ryan Jones: "You said your code is the most brutal part. And I think everyone is that way ... when we have AWS, with 1000s of engineers building services and hardening them over 5/10 years."

Code Quality
Q: Can we ever get bulletproof code the same way we have bulletproof SQS stuff?
Eric Johnson: With AWS we build heavily based on what customers are struggling with and ask us for. For a long time the infrastructure that was easy to do that with was SWF. But you see us moving further up the stack, lambda functions are a great example. While I don't have any 'roadmap' secrets to give you, I can tell you that our goal is to remove the brittleness and help customers get to the application. They want to build security, resilience and scalability. Those are job one for us. So yes, I would say you'll see us continuing to move up the stack, to help customers out in that way.

Q: For a while the CloudWatch offerings were somewhat lagging behind some of the third-party observability tools, where they had focused teams building this stuff out. How do you think about that when you're approaching somebody that's starting to do stuff? Do you lean completely into CloudWatch? What do you think about observability when it comes to serverless?
Eric Johnson: It's going to be the same. Many customers, I worked with that in Rackspace, as a solutions architect. I love this, because I'll ask them, are you monitoring your application? And they'll say, we're monitoring the heck out of it. We're monitoring that bad boy. Okay, that's great. What are you doing? We're storing that data. Okay, great, where are you storing that? We're storing that data in S3. Okay, so what do you do with that data? We store it, and if there's a problem, we have it to go back and read it. That's a great start, right? Yes, we've wrapped in some logging and metrics and things like that.
For me, from the beginning, I believe you should write observability into your system. CloudWatch is a phenomenal tool for that. If you can take, with Insights or Xray, especially with serverless applications using X Ray, I can give you the story around that. We have partners that we use and we love them. If I'm sitting down talking to a customer and say, if you've got someone that can sit down and build your observability into your application, then cloud watch is the way to go to build your alerts and metrics and everything you need, they understand that. But a lot of companies don't, especially start-ups and even bigger companies. That's where these 'out of the box' solutions come from. Some of our partners are really good, because they're going to tap into that and say, we've already got your alerts and metrics in place, literally just turn our system on. That's where I draw the line. A lot of a lot of our partners rely on cloud watch as well to get that information. But as far as having the technical know-how to build that, we give you these 'Lego blocks' in cloud watches that build an amazing story around your application.

Education
Q: When we talk about going from one lambda function to a production serverless application, what does that look like? How long does it take and how would you encourage people to move to the serverless application?
Eric Johnson: We have a lot of folks that start in the console and they build a lambda they trigger with a CRON event or cloud watch event. The first thing I tell folks is, that's great, you're in the console, you built that, but now let's get out of the console. Because the console is not scalable. How do I approach enterprise grade applications? I don't build them on the production server; I start thinking about them in pipelines and start thinking about how to break it down just like any other architecture I'm building. If I'm going to build an application, I'm going to start thinking about my domain objects. What are the environments that I need? How is this going to break apart? That's the same in serverless as it would be in non-serverless.
The cool thing about serverless is, if we get out of the console, the first thing we tell folks is to look at a framework. I love the CMC a lot. Another great one is Serverless.com, or GoServerless. To really get serious about building serverless, you need to get into one of the frameworks. This takes a lot of the deployment and the debugging locally; looking at logs, different things like that. You start thinking more about just trying it out and having a plan to build an application. So that's the first thing we say; get off the console and look at getting into the framework. The second thing I would say is look at some of the workshops that are out there. This is going to be a shameless plug but I'm throwing it out there. James Bezwick, one of our developer advocates, created an amazing workshop called Innovator Island. It walks you through building a full-on application that monitors a theme park and how many people are on the rides. You take pictures and green screens and all kinds of stuff. It gives you this idea of building a full application that is mobile, client and website ready. So you kind of get into your head, how do I build serverless applications past that?
If you're anything like me, you've got to break it to learn it, so my encouragement for building service applications is grab hold of one of these frameworks, get started and then build a proof of concept. That's one of the beauties of serverless; you can get a proof of concept up immediately. You can do the ever popular image manipulation, lambda function, or you can do the Hello World, or whatever. As you get that proof of concept going, you're able to grow out from that.

Ryan Jones: "... you're not throwing out the entire book. If you have established processes, most likely, they just need to be adapted to serverless."

Conclusion
Q: Are there specific backgrounds for people to have, that might actually be a really great fit for something like serverless, if they're interested in serverless or a career change?
Eric Johnson: Developer backgrounds. I think there's sometimes a mystery of serverless that people see, that's really not there. If you can code, you can do node. Go Ruby, Java, Python, Swift... That's what I've been working on lately, running Swift in a lambda function. If you can do that, then you can do serverless. Pretty straightforward. If you're not a developer, and I see this a lot, I've got some buddies that have kids that are learning to code. When I show them, you're doing a lambda function and there's no computer, they love that. So if you're learning to code, I think you can pull serverless in, you know? I could just run this in a lambda function versus having to bring up another server. So yeah, all walks of life. You're talking to an ex-pastor with a Master’s of Divinity degree who's doing coding. Everything I learned, I learned on Google at night. After the kids were in bed, I went downstairs and just started breaking things until one day it worked.

Q: Where can we find you?
Eric Johnson: The developer advocates for AWS, we have a YouTube serverless channel. The link to that is youtube.com/c/AWSserver. You can also follow me at EdJGeek on Twitter. I also answer a tonne of serverless questions there, so that is a great way to reach out to me and my team.

Ryan Jones: Thanks so much for coming on the show, Eric. This has been the talking circles podcast with Ryan Jones. If you like our show and want to learn more, check out TalkingServerless.io. Please leave us a review on iTunes, Spotify or Google podcasts, and join us next time as we sit down with another fantastic serverless guest.