Ep 33: Yan Cui Returns!

Ryan Jones: Welcome to the Talking Serverless Podcast. I'm your host, Ryan Jones, and I'm joined today by Yan Cui. Yan is one of the most notable people in the serverless industry; an AWS serverless hero, independent consultant, host of the popular podcast Real World Serverless. He is the author of Production Ready Serverless, and more recently, the AWS AppSync Masterclass.

AppSync Masterclass
Ryan Jones: Q: What has changed for you in the past three to six months?
Yan Cui: The biggest change over the last couple months has been the AppSync Masterclass, which has been doing really well. We opened the early access in October, so it's been just under three months now. It's going really well, more than 700 people have signed up and we have something like 100 thousand dollars total revenue already, which is pretty crazy. I didn't really expect that. It's been keeping me really busy; I'm almost doing double as much content as I thought I would up to this point. So far we've done the first four modules. So you've learned about the basics of Graph QL, AppSync and how that compares to Rest and API gateway. And you've built a complete and functional back end for a Twitter clone that lets you sign in, sign out, edit your profile, post things like tweets, follow other people, re-tweet, reply, and all of that. It's got loads of unit tests around the PTL templates, lambda functions, and engine tests. My co-instructor Jura Sans, who used to work on the Amplify team, is now busy doing the front end lessons. I think we will be able to release a few of his early lessons pretty soon. So it's been pretty full on the last couple months because of that, as well as all the consulting work that I'm doing on the side as well.

Q: What does the future of AppSync Masterclass look like? Do you have any plans to continue growing it underneath a certain brand and add more courses?
Yan Cui: For the Masterclass itself, we still have lots of content ready to produce. We've got a basic working backend for the Twitter clone, but we are still going to add integration with our Agolia to implement the search features, and we're going to introduce Graph QL subscriptions. When we do the direct messaging support we're going to introduce how it knows common things users would like to do, like Analytics tracking, and we're going to do things like progressive web application and mobile friendly UI. That's more Jura's expertise. All the stuff we're doing on Vue JS and VCSS, I did an initial version but he spent a bit of time just refactoring my terribly written back end developer front end code. So I think we will be busy during this AppSync Masterclass for at least the next four or five months, trying to wrap up everything we have in the agenda, which is quite substantial. After that, I might do another course later in the year. But we will see how much energy I have left after doing this one.
After Masterclass we will probably turn it into a workshop format as well, which is what I did for the Production Ready Serverless course I did with Manning. When you do a video course it becomes really hard to keep up to date, because maybe you changed one thing early on in the course, and that's going to have a ripple effect on everything else you've done. So it took me almost a year to finish the Production Ready Serverless for Manning, and it's been really hard trying to find the changes that we could update without having to re-record the entire thing. So after I've got that basic content, it's turning that into a workshop that I can more easily recycle and reiterate on new things that are coming out from AWS. After every re-invent, I have to change a lot of the lectures and also maybe some of the code examples. Without having to record everything is a lot easier to keep updated as time goes by and introduce new leading practices, tools and libraries as they become available.
Going forward, I may do the same thing with the AppSync Masterclass, because Production Ready Serverless has worked super well since it went live almost three years ago now. Time really flies. We've also been running it as a workshop or for public service officer in-house training for a number of years now and it's been really well received. We've been continuously refreshing the content as new things become available that we need to teach people. I think that's the future of AppSync Masterclass; once we finish all the contents that we already have planned, there's still plenty there to do.

Q: Who would you pitch AppSync Masterclass to and what skills would they take from it?
Yan Cui: We actually thought really long and hard about that. One of the things that's quite unique about this particular course compared to other things I've done is that it's not just about the backend, it's not just about AWS, there is also quite a lot of front end as well. And with Graph QL, I think that makes a lot of sense, because Graph QL as a technology is really friendly to front end developers. Instead of having bespoke BFFs, you can have just a Graph QL API that unifies a lot of your back end services into something that is easy to consume from the front end application. So it's really well suited for teams that have got that full stack mentality. With this, if you're a front end developer, you learn enough about AWS in terms of the services that you likely would end up using like lambda, AppSync, DynamoDB, Cognito for authentication, CloudWatch logs for monitoring and things like that. So it gives you enough exposure to a lot of the common tools that you probably end up using in a full stack environment, where you're doing a lot of back end work using serverless components, without it being so laser focused on lambda ins, outs and best practices.
With this, there's also a lot of content around the front-end side of things. Not just how you do things with View but also linking things together, how to connect to Cognito and a Graph QL backend, and practices that are common in the Graph QL community like using fragments or using other things like that. So you should have a tool belt with a lot of knowledge from both the front end and back end, but also in terms of the practices that are useful in the production environment like how to monitor things and not just asking API to deploy it and hope for the best. Also, thinking about performance and caching strategy, reacting to problems in production, testing and how to think about testing for serverless, and things like that. So there's a lot of different topics. Even an experienced serverless developer should come away with a lot of new things that they may not have thought about in the past. So there's a mindset to everything I teach that it's not just about teaching you how to write code. That's the easy bit. It's thinking about the whole application more holistically from the code, what services you should use, and the practices and the processes you have in place for dealing with things in production.

Ryan Jones: "(serverless) courses tend to lean towards the back end with a little bit of front end. It doesn't really get deep into best practices that have front (and backend) at the same time."

Build VS Die Mentality
Ryan Jones: Alex O'Brien recommended a book called Ask Your Developer. Have you read it?
Yan Cui: No, I haven't even heard about it. What's it about?
Ryan Jones: It's written by Jeff Lawson, the CEO of Twilio. I love it so far, because it talks about the power of software developers and unleashing that creativity across the entire company. One concept that he gets into quite heavily is 'build versus die'. I've previously heard people say 'adapt or die', when they're talking about legacy, or companies that are trying to transition and keep up with the start-ups of today. But from Jeff Lawson's perspective, companies that learn to build things will outpace their competitors. It sounds like a natural transition from what you were just talking about. Your course is teaching people how to build things and build them quickly, and Jeff Lawson is talking about how companies that learn to build things are going to outpace their competitors and survive this period of massive disruption.

Q: What are your thoughts on this? What do you think about the 'build versus die' mentality, and about asking the developer in terms of unleashing trapped creativity?
Yan Cui: Well the serverless mindset is that, as much as possible, we want to offload non-differentiating concerns to someone else. Looking after infrastructures shouldn't be my concern if a device can do it better for me and I can just rent computer units for running my application code. So from that perspective, I definitely think you should be buying instead of building. But if there are things that are not what your core business is for, and that's why your customers are coming to you, then you should definitely be building those things. I don't know if you follow Simon Wardley. He's the one that created the whole value mapping chain, where you can look at your application and the different levels of components you have, and see how it maps to a guest at the top, so you've got the things that the customer actually wants. To deliver that you need the computer, the networking, and maybe some CRM stuff. Then you can look at the maturity of each component. If there's something that is already commoditized, like a computer, then you probably shouldn't be building your own platform. For something that is a new idea or everyone's still building their own thing, I think you should have a set of features and those are probably things you should be potentially building yourself, or looking at something off the shelf to get you started for now. If it's not your core business, maybe you should buy instead.
Simon's value chain mapping is about answering those questions for specific components, not just as a whole. Thinking that we should build everything, I don't think it's going to do anyone any good, because you're going to be doing a lot of things that don't differentiate for your business. But at the same time, you also don't want to buy everything, because somewhere you're going to have to introduce a value for your customers. Otherwise, why would they come to your business over somebody else, if you're just going to offer something generic?

Ryan Jones: "...we're at a point now in this evolution of technology and software, where the AppSync Masterclass is built on so many layers of abstraction. Like there's VTLs in between AppSync to a fully managed database that's completely being handled by AWS in the background, or lambda functions. It's handling the compute and the scaling and all those things in the background as well."

Future
Q: What are your thoughts on building applications today and being able to do what you're building with AppSync Masterclass versus everything we had to think about ten years ago?
Yan Cui: It's night and day. I recently wrote a blog post that details a client project of mine where I built a back end for a new social network, working part time for a couple of weeks. That sort of thing just wouldn't have been possible ten years ago, because I would have been spending that time just configuring the VPCs, the database, setting up the AMIs and doing the capacity planning before I can even write a single line of business logic that will actually do what the client wants to do with the app. That was normal ten years ago. When I first started with AWS, there was no DynamoDB, we had to do a lot of work just to get a simple DB at a time to scale to what we need. There was no RDS at the time either. So at one point, we had to set up a cluster of at least two instances to get them all connected, and then configure My Sequel. For the key value store stuff, we were using simple DB, which was something like 50 writes per second on a petition. There was no easy way for you to scale that quickly.
Ten years ago, Memcache D was the thing. There was a company called North Scale at the time that built something like Memcache D, but with the persistence. So we used that for loading data really quickly, then we had a persistent layer to save into a simple DB at a rate that a simple DB can handle, and there was some stuff we had to put into My Sequel, so we had to put a caching layer in front of that. That stuff took so much time. The project would be delivered in a few months time with us working very hard as a team, and that would be considered fast. The things I don't have to worry about anymore are just amazing. It's great. It's made my job so much easier. If I'm looking at this situation as an engineer or as someone who wants to tinker it's terrible, because all the fun has gone out of discovering how to make everything work. That excitement is gone. But what's replaced it is getting things done so I can just relax. As I'm getting older, I appreciate that a lot more. Those things were exciting when I was younger, but now I like my sleep.

Q: I'm starting to see a lot more people talking about no code, low code, and AI cogeneration, to help developers move faster and build things quicker. Do you think this is the natural progression of software? What are we headed towards?
Yan Cui: I have seen some of those AI examples and demos for that are really, really amazing. But at the same time a lot of it is just about semantics and being able to work out what things tend to go together, so I guess it's not genuine creativity. That's something that we haven't quite figured out how to do; I haven't seen demos that are clever enough to portray genuine creativity or problem solving. But I think we're definitely getting closer, which is pretty exciting. I look forward to the day when you can just tell a computer to make something and it's not just not a trivial thing. Who knows, maybe that is going to come in the next 5/10 years, because we are definitely hitting exponential growth when it comes to AI and the applications of AI. I'm super excited about what's coming in that particular space. In terms of no code, low code solutions, it really depends on what specific solution you're talking about. I think most of them are really useful for a particular niche, but at the same time, it's not all that different from a lot of the managed services you can use. I haven't used one that makes me think it's good enough to say 99% of the problems that I need to solve a lot. All of these things are still either too opinionated for more general use or too niche in terms of the problems they are designed to solve.

Q: In this no code, low code, are people trying to develop things just to make them clickable, and does this create things that are too generic when we think about the actual business value?
Yan Cui: I think it really depends on what it is you're trying to build and if you can do certain things but the moment you need to deviate from the design it becomes quite difficult. You can see the same thing happening now with tools like Amplify. I was just talking to a customer who wants to use Amplify because it allows them to focus on the bit that differentiates the user experience that they offer, which is on a front end, which is great. That's what Amplify is great for, but at the moment, they need to do something a bit out of the box of what Amplify can deliver, and that becomes quite difficult. And to the point where they're now thinking it's more effort and cost to try to make it work with Amplify than it is to just rip it out and replace it with something that gives them a bit more control. I think you're going to hit that with a lot of tools that we have right now that solve problems in a specific opinionated way, which works well if you're able to fit inside it. So limitations at the moment are the need to do more customization, because it becomes a fight against the tool or the framework. So far what I've seen that fits into the bracket of no or low code is like that; at some point there is hidden limitation, which is going to be more effort to work around than to just rip out and replace with something more customizable. But again, a lot of people are very happy with tools like Amplify because they don't have to customise it. It's just a case of when you need to customise things and get beyond what Amplify can deliver.

Tools and Applications
Q: A lot of tutorials focus on the Hello World type of application. With Amplify you can build something very fast with a low amount of inputs, there's a lot of different tools that allow you to do that. What types of things do people hit as it grows, and how do they go about making decisions to switch off of tools like this?
Yan Cui: Writing the code and deploying it like you do with a Hello World is the easy part, but that's less than 5% of what you end up doing. There are other things you still have to think about in terms of security, making sure that you've got things rules to protect you against SQL injection, in terms of observability, how you debug things and test things... No-one's gonna bother to include those in the Hello World example, but those are the things that everyone has to deal with all the time when you are working in a real world project, and you need to teach people how to do this. You can't just abandon them at Hello World and let them figure out how to do all the rest themselves. So that's the inspiration for why I did what I did in the production ready service, because I didn't find anything at the time that was focused on that. The less glamorous part of it is, what you don't see after your build is a flashy demo. But that's what we spend 95% of our time doing as developers. That's why the production ready service came about.
As for a problem that people run into using tools like Amplify, I've heard a couple of second-hand stories from customers or people that I know that have used Amplify in production, and then they've run into problems. Things that seemed to come up quite a lot are again, when you need to customise things beyond what it's designed to allow you to do. The other thing that I often hear that made people switch back to something that offers more control is, say, you're making changes to your model, everything is great from the Amplify perspective, but you're still bound by limitations that the confirmation has. If you need to introduce a new model that ends up adding to global secondary index, it's not going to do that. This particular problem hit a lot of people and forced them to basically delete a stack and start from scratch. When you're building a proof of concept or a demo app, that's fine, no-one's gonna really complain. But you can't do that to a product you have to run 24/7, with a real customer who's paid money to keep this thing running right. So that's another thing that hits a lot of people. The good news is this particular problem, for global secondary index, is kind of fixed. The Amplify team have recently announced a thing, whereby when the amplifier generates a change that's going to require adding to a global secondary index, it is actually going to do them in the two steps. So instead of doing one cosmetic update it's going to two, which is a really clever way to work around this without you having to think about it consciously.
You can still hit other problems that fall into the same category of cloud formation changes that you are not consciously thinking about, because it's generated by the tool. It reminds me a bit about some open source projects that give you a Docker image, you can just run this thing in your environment, you can just host it and run it yourself. It sounds great on paper until you realise that you don't own the code, but you own the uptime, which means any problems that you run into along the line of deployment configuration, or just something happening in production, that is still you to actually fix it. And because you don't own the code, that fixing bit is going to be extra difficult, because you don't understand how things work under the hood.
I think that teams are more likely to use tools like Amplify because they don't understand how a lot of the stuff works. That is probably the most dangerous thing you can do to yourself, because you're automating things that you don't understand. If it breaks you're going to be really hard pressed to fix things in the most stressful situation possible, because customers are complaining that your application is down or it's not working, and because you don't understand how things work under the hood, it's harder for you to actually understand what the real problem is and how to fix it. So if you understand how things work under the hood and you're just looking for ways to automate as much as you can, that's better.

Cloud Development
Q: Can we ever escape cloud formation? What does that look like for the future of cloud development?
Yan Cui: I don't think cloud formation is necessarily a bad thing that we need to get away from. I mean, the CDK still compiles to cloud formation, so you are still gonna be bound by all the same limitations that cloud formation has. As a cloud developer, you really should spend the time to familiarise yourself with some of these basic building blocks, like cloud formation, like SAM. They're going to be your goodies. If you want to be using benefits on AWS, you really need to understand how AWS works. I don't think there's any shortcut to that. You could use terraform, but that just gives you a different set of trade-offs. Cloud formation gives you rollbacks when it's probably for deployment. Terraform doesn't, it just leaves you hanging in the middle of wherever you end up in your deployment pipeline. To me, that's not a good thing. Most of the time I don't want my application to be left hanging in some halfway house. Terraform has got other problems as well.
So right now you've got the two main options of either cloud formation or terraform as the basic building blocks, and the source of truth that builds on top of that for syntax or automation so that you don't have to write thousands of lines of code by hand. I think it's great in that it also offers visualisation on top of all that, which makes it a lot more approachable in terms of visualising what you have in your stack. As you're working with that and configuring them, I think that's a good thing to have, but I don't think confirmation itself is a problem that we need to necessarily move away from. I do think there's a lot of things they need to improve on. One really big one is just endless releasing of things. So without support, there are mixed messages. On one hand, they keep talking about infrastructures code. On the other hand, they keep releasing services or features that don't have a cloud switching support. And the documentation or tutorials just keep telling you to click buttons in the console, which is not what you're telling people to do. So I think confirmation needs improving, but it's not something that we really need to get away from, per se.

Q: Is there a line where this goes too far and you really don't know what you don't know?
Yan Cui: So there's this quote that's often attributed to Einstein. I don't know if it's actually true, but he says something like, make something complex simple, but not any simpler. There's also a really famous talk from Rick Hickey, who is the creator of the closure language. He talks about simple made easy, and the distinction between simple and easy. Oftentimes we mistake one for the other, or we use them interchangeably. So I think there's this notion that we should make something easy, like we just should just hide cloud formation altogether. But then you have this problem of making something too simple. You're not just making something simpler, but you're also taking away possibilities or flexibilities that you need to have in order to solve more complex problems. You're taking away not just the complexity, but also the capabilities of what one can do with the tool. Whereas simple should be reducing the complexity, but without taking away any of the things that you can do. Simple simplicity is great. But oftentimes, we end up with ease instead. And I think that's the wrong thing to optimise. Because, again, we are optimising for Hello World examples, but not the real world messy problems that people have to deal with.

Q: You have this automated tool that abstracts all these things in the background the more layers that you add there. What about ten or fifteen years from now? What happens if these apps that we're building today actually last longer than a couple of years? Are there ways that we can prepare for our current systems to become legacy?
Yan Cui: It's always easy to look back with hindsight, but trying to predict what may become legacy in the future, that's much harder, because we just don't know what's gonna unfold. When talking about tools that you can depend on for the long haul, I think anything from AWS you can pretty much guarantee is going to be around. I guess it depends on the vendor. You can't really trust Google. They are just going to queue your service that you use in production tomorrow with no remorse. Certainly when it comes to AWS, I do trust that whatever is there today should still be available in ten years time. As for tools like the server framework and terraform, the fact that it's not just entirely a commercial product base, it's also widely supported by the community and widely dependent upon the party community, also gives me a lot of faith. The fact that most of the updates now to the server framework is done by external contributors, that's probably what gives me more confidence about the long term. When I look at a tool that I want to use for the long haul, how much can I depend on this tool being around in 10 years time, I look at if it's coming from vendors, if it's a service, then notice the vendors that have got a good track record of carrying things around for the long haul. Is this tool supported by a sizable community that depends on it, but also contribute towards it?Do they have a say in the outlook of the tool, or is it gonna be entirely dictated by one company? Those are the sort of things I look at when I think about, okay, are these things going to be a problem for me in 5/10 years time?

Ryan Jones: “It's interesting to think that the stuff we're doing today will be legacy, and time will pass, and we'll look back on all this stuff much, much older.”

2020
Q: 2020 was the first virtual Re-Invent. What were your thoughts on the format? Do you think conferences like this are gonna happen in the future?
Yan Cui: I thought the format was okay. I did enjoy some of the talks. The fact that you can watch on demand is also useful. But what remained for me the last couple years has been the networking side of things, which you can't really replicate online. I know they tried it, but it still doesn't replicate the physical networking opportunity that we had at ReInvent, which for me was the most important thing of going to Vegas every year. It's meeting up with people that I talk to all the time online, but never seen in person. It's just great to hang out with people that you know virtually, and finally meet them face to face to build those connections and friendships. These ReInvents definitely don't have that, but given the world that we live in today, that's unavoidable. I hope that there will be better solutions soon in terms of recreating that. I've seen a lot of different attempts, but nothing has quite come close to it. As I get more and more engaged in the market via Oculus Quest 2, and some of the stuff I'm seeing with VR chat, that just seems like a quiet good community, even though that frame rate in VR chat rooms for lots of people, that's probably not great, either. But I do think we're gonna see virtual conferences as a thing going forward.

Ryan Jones: “As software developers, I think sometimes we think everything can be solved with code or something. And we can recreate, you know, this type of experience online in some form. There's got to be a solution, but if there is a solution, we haven't seen it yet.”

Q: Which announcements spoke to you the most, and which ones should people pay attention to?
Yan Cui: One of the biggest announcements for serverless was Aurora serverless v2. Obviously, that's the most eye catching stuff, like container images support. I think there are some specific use cases where that's useful, but it's not something that everyone should be jumping to right away. There's a lot more complexity involved when you need to bundle the messier container images and there's additional stuffing to get involved. There's also the support millisecond billing. But I think the Aurora serverless v2 is probably the biggest one for me, as far as the serverless announcement goes, because it does offer some really interesting technical advancements over Aurora serverless v1. I think for me, that's the big one.

I remember when the meltdown inspector came out, I spent a week just patching all the different API's and container images I had for the company I was working for at the time. We didn't even think about lambda until I saw a tweet from Chris Munns, that all the infrastructure that's running fargate and lambda has been patched for both Spectre and Meltdown. And that just hit me as one of the benefits of using the platforms that are managed services like lambda. You don't always have to worry about security. And the same goes in terms of performance. If anyone was actually paying attention to lambda performance over time, they will have seen a gradual improvement over the last couple of years, as the lambda team has introduced a number of different changes to improve the overall cost performance for the functions over time. With intensive costs, as well as the fact that now you don't get built in the 100 millisecond blocks, things are cheaper. So now using managed services like that, you're gonna find your application becomes more secure, more resilient, and cheaper and faster over time without you having to do anything. In the past I have to invest into infrastructure to make things go a tiny bit faster, a tiny bit cheaper. Now I just wait for the vendor to do that work for me.

Conclusion
Q: How can people get in touch with you? Do you have anything that you want to promote?
Yan Cui: Theburningmonk.com is the first place you should go to check out my stuff. I've got so much content around serverless on my blog already, and you can also follow me on twitter at the burning monk or LinkedIn as well. Check out AppSync Masterclass at appsyncmasterclass.com or my upcoming workshops at productionreadyserverless.com. If you want to hear me and Ryan chat about how Ryan's been doing stuff with serverless and how he's going, go to realworldserverless.com and listen to our episodes for my guest last week.

Ryan Jones: “Thanks again for being on the podcast Yan. It was great to have you on the show. This has been “Talking Serverless podcast” with Ryan Jones. Please leave us a review on iTunes, Spotify or Google podcasts if you found this valuable. And of course, join us next time as we sit down with another fantastic guest.”