Ep 23: Ken Collins Principal Engineer at CustomInk

Once again, welcome to the Talking Serverless Podcast. I am your host, Ryan Jones, and I am joined today by Mr. Ken Collins. As a long-time blogger and open-source software advocate, Ken Collins is an AWS serverless hero and principal engineer at Custom Inc. Moreover, he is an organizer for the Ruby user group in Norfolk, Virginia. Ken has been working in the software industry for more than two decades.


Background

Ryan Jones Q: How have you been transitioned into where you are now?

Ken Collins: I think what interests me the most is that I've always been a little bit technical and completely self-taught. Even though I had a high school degree, I had to get summer school to get my diploma.

Thanks to the Internet and many articles where people will take the time out to explain things. I have taken a lot of my time to try to figure out what that is and learn everything that I can. I have changed careers about three to four times in my life. I used to be a graphic artist, and I was in middle management. I used to be an account executive at an agency and was a marketing director at an e-commerce company. I guess I've only been programming for the past 12 years, maybe 11. But Custom Inc has been in business for 20 years.

I think about 12 years ago; I got my job as my first-time software engineer. It only took me about a year or so to get heavily involved with open source. At that time, my largest open-source project was writing the low-level c bindings for Ruby to talk to SQL Server and enhance it. Hence, I worked at a company that was doing SQL Server on rails. I think we had adopted rails at that company, right? Eventually, when rails 1.26 came around, they dropped the native SQL Server work integrated into rails and focused on MySQL, Postgres, and SQL Lite. So, I wanted to have an excellent job, be happy, and continue to program in this fantastic language. I took it upon myself to pick up the adapter project and try to own it to have fun at work. I solve that problem by solving it for me, so I cannot think about it.


Benefits and Principles


Ryan Jones Q: What does it feel like for developers when you work in a company that releases stuff to the open-source community? How does it benefit you as well as the company?

Ken Collins: Yeah, I think it's rewarding for me. I approach problems differently. I don't ask for permission; I do. So, at big companies, I can sometimes work to your advantage. I don't think there's ever a point at Custom Inc where I had to go to a manager like a manager and say I would like to write open source. It feels to me like it's pretty clear on the boundaries where specific solutions lie. I usually step forward with open-source work and then sort of backtrack into it.

Even if you know you're writing software, in some cases depends upon where you said, trying to open source first, right? In some cases, try asking for permission or just doing it right. I found a lot of the people that I feel fortunate to work with. They are outstanding mentors to me, like managers and VPS. I present things to them when I'm working on them. It'll be open source. With these sorts of implicit things like security or other people's contribution, any company would recognize that they're doing open-source software or having good content on tech wallets that might help us recruiting, making it a win-win situation for everybody.

Ryan Jones Q: Do you have very standardized principles that you follow on a day-to-day basis?

Ken Collins: In my software engineering career, there will be a time where I learned some computer science terms, just the basic ones, since I don't have that sort of classical education. The way that I approach and solve problems makes sense to me. I'm motivated by this weird sort of internal monologue that I don't know how to express well. The SQL Server adapter and the work that I did there, I did all that to solve the business problems. I took specific strategies when I wrote that, like, how I didn't want the adapter to be coupled to certain ODBC connections or other things like that. I solved all the problems and native t sequel, the native language of SQL Server, rather than doing underlying things, which made the adapter more portable and stuff. We have learned terms for that lictors attractions later on.

I don't know if it's how I approach problems or the fact that I don't have a computer science degree. If I had a degree, it would probably be some form of Liberal Arts or approach these problems a little bit differently. Like giving back to the community, everything I've learned is from a blog post, or somebody is taking the time to do something. It seems natural to want to go through and do that same thing as well. It is basically the best way for me to remember something. Also, if I go through and I do these articles, and I do teach people things, then what they're going to do is find holes where I haven't done things right, or they're going to teach me something in return. So, I feel like it's the type of 70s type of weird Montessori school that I went to when I was younger, where I think I don't even know what grades were until maybe six or seventh grade. I do things that might sound weird.

Moreover, I'm not the smartest person in the world. Also, I feel like my capacity always sort of wraps my head around, maybe a small percentage of what other people can force me like, think of things in more simple terms. I'm a huge fan of node JavaScript. I love front-end frameworks. I'm risk-averse and exhausted from the JavaScript frameworks, which is why I love Ruby so much. I can wrap my head around these sorts of things like react or the simpler counterparts like stimulus dot j: s and whatnot.

It feels like I would rather spend a lot of quiet time figuring out what a small core problem is and engineering the simplest way to adapt it right up. I think that all engineers are doing product stuff, and you're dealing with the business and asking the business, do you need the software? Maybe the best software you need is no software and just really critical of asking questions like that. So, I've recently learned to have massive, no code movement. Also, I think Amazon has a recent product called the honeycomb.

Story of Custom Inc

Ryan Jones Q: How did the story unfold with Custom Inc and with serverless?

Ken Collins: So, I think the best way to describe it is like if I had been in software, about 11 or 12 years now, this journey would have started about at year 10. I think it is the amount of time you need to get good at something. About two years ago, I kept out of my career of like; I can do Rails apps well. I've done front-end JavaScript frameworks. I've done various amounts of gems.
What is the next point? When I got my principal engineer title, I was like, where would I go with my career? What would it look like? What would my contributions to the various fire teams that custom make look like? I finally decided that with a little nudge from one of our directors, that learning the cloud is where we wanted to be. Custom Inc had a recent success a couple of years ago, where we finally got everything that we owned, lifted, and shifted into the cloud, but we had no more physical servers. There was this desire now to go all-in on the cloud; what were the force multipliers, that if we adopted the cloud from the get-go, rather than lifted and shifted our architecture that we might think about?
We knew we had some successes from some of our other engineers with lambda, where we hyper optimized some of our workloads with lambda? Let us make DevOps and do cloud-native adoption better. That's when I started to learn lambda, and that's my primary purpose of the cloud. The Landy product is about version 1.5 of that adoption process for me. I first wrote node lambdas. I did more of the repeat process for CustomInc of hyper optimizing some of our workloads and extracting some of our models into these sorts of really well-tuned microservices.
One of our engineers came to me and said that these are neat services. But they all look different. They all look like when you go from one to the other. You have to wrap your head around the domain. You have to think about how this person or person approached writing their plain old Ruby or JavaScript objects? How do they need to do around it? Then he had asked, how could you make more structure for these, these lambda projects? I don't think he anticipated I would throw rails into it.

Ryan Jones Q: When you threw rails into it, how was it taken? How does Custom Inc write their lambda functions now?

Ken Collins: Is it purely Ruby with Lambie, or is it a mix of both? For us, most of the Arlanda adoption is still about tuning our platform. We have been in business for 20 years; we had one big Java back end and one big rail front end. One of the cool things about the customer that I like about them is that after 20 years in business, you're afforded these architectural key lines that are well understood from business success. We're not architecting things at this point in our life, where we're getting up to whiteboards and drawing things out and saying. It is well understood that we need to pull this service out, or this workload for, like our image architecture for handling how clipart our fonts or our designs are rendered. That is mostly where Orlando's work has been falling on where we take the image architecture, whether you are in our design lab and resizing fonts around or our graphic art images, uploads. That is all, for the most part, more so later handled through a lambda.

Growth and Obstacles

Ryan Jones Q: Did you hit any obstacles when going through the process?

Ken Collins: Lambda is still very hard if you're new. You know how to approach it. We've gone all-in with Sam, and a few of our first projects were the serverless framework. But right about this time, when I started learning lambda, the same framework was out. I just fell in love with it as it solved many of the problems that I like. I found it very hard to learn serverless because I felt like I had to learn a serverless language, translating to cloud formation. I felt like I was learning two languages at once and not even have come in from what infrastructure as code was.
Hence, we standardized on Sam; we standardized on if it was a microservice, cookie-cutter approach. We wrote cookie cutters for Python and Ruby. To any engineer who was getting started, they could hit these cookie cutters, who have an optional configuration to set up API gateway or not. Basically, lambda projects had all of our business conventions worldwide, like using multiple AWS accounts rather than one AWS account. There were conventions in place to deploy different accounts or organizations, from development to staging and production, making it easy. You could easily say, "I can clone this repo with a Santa knit function; get my lambda."
Now I set up locally and deployed into a development account with literally a couple of terminal commands, and a few minutes later, that's that healthy adoption out. The Landy framework is meant to tell that story of Hey; how can we take these existing Rails applications and lift and shift them to lambda? We can get that commodity out there and then start thinking about how do we do cloud-native.

Ryan Jones Q: How did that develop from the initial stage of not standardizing things to organically structuring things?

Ken Collins: Luckily, we had a customer, and we have over a couple of 100 applications. Even before, we adopted servers, which would explode that number because if you started breaking out microservices, what you might call some kind of proper, like architectures that might have things smaller from the get-go. I'm not saying that all that microservices shorter from the get goes proper; it depends.

But we had these things in place. It was a blog article done by GitHub a long time ago that called scripts to rule them all. It was saying that no matter which project you move to, you would be able to do these sort of standardized script conventions for bootstrapping, setting up, and running the project's test. Even though most of our projects were rails, some of them had Elasticsearch, some of them had different resources associated with them. We had all that in place. The cookie-cutter quickly adapted to that mentality; it means you had to do it from a different perspective. Like you had to write Ben scripts around Sam.

But I think coming from a company with some sort of standards was independent of the implementation that helped our adoption. We didn't have a lot. It was challenging to say, Hey, we want to adopt Sam. When people were writing serverless frameworks, with a serverless framework, or anything, write to them. About three to four years ago, lambda was the very wild west. It didn't matter how you got that out, how you integrate it, either with Jenkins, or Travis or GitHub actions or anything like that, as long as it was sort of like, Hey, we got the computer out there, it's doing the function.

So, I spent almost four months just reading all the documentation, evaluating everything, going through this, writing the process out, and looking for those things that say, Okay, if we're doing it this way, what do we get out of? The biggest one that I didn't understand initially was the whole infrastructure concept as code and where those boundaries lie in the project like a bucket and a DynamoDB table with HTTP API, and lambda that's like four. But knowing that you could do that all within the confines of one repo was just a game-changer for us. You no longer had to go to the web ops team or the infrastructure team and open up JIRA tickets. We are still going through that transformation, where we now realize that engineering has an outsized capability. We are only still sort of learning on standing up our infrastructure.


Team of Developers

Ryan Jones Q: How have you seen the team's development change, and what are you trying to make it better in the future?

Ken Collins: We have been in the act of training, for about four months to maybe six months. We've had enough pieces around to where we want to architect things or how we may want to solve different problems. If you're learning to drive, even though I sat in the back of my parent's car, and by the time it came for me to drive, I knew how to go places; I knew where the ball was, I knew where things were, because I look out the window, and I paid attention.
But taking yourself there is a different thing. So, we've had to actively define some infrastructure and architecture goals, pass those off to various engineering teams, get them actually to work on those in isolation, and have them work through the problems. They have the projects like if you follow Basecamp, then have the freedom to sort of work with it in there. It has been exciting watching different engineers, how they learned. I'm getting better at being a senior engineer; it realizes how different people like to be trained. Some people want to have sort of very collaborative training and stuff. But I think the biggest thing we've been doing at CustomInc is recognizing that we need to shift. We have done training, allowing people to opt into it, but shifting to some infrastructure-based projects. We will have to pay those dividends for a cloud-native adoption and setting those up for the engineers to work on.

Ryan Jones Q: Are you kind of allowing your developers to go through that process switching between the terminal and all that stuff?

Ken Collins: I wrote an article that was a Sam getting started. It was put on our Dev Tools account; I can do a show note link forward. It was as if you were going from zero, and all you had was an AWS account. You were looking to learn lambda, using Ruby as a microservice. So I wrote that whole process as just a little workshop. Many of the engineers don't know about these decisions. You can show them an article that says, hey, here's how you get HelloWorld in. But if they're like, well, I need to listen to events for a bucket?

You probably have hundreds and hundreds of buckets. How do you listen to those events when you can't create the bucket from the infrastructure resource? That is learning the boundaries like click ops lies and the amount of stuff you have to recognize if you went through this journey of about a year. You are like, Okay, I know what Sam is, I know what all these things are. I understand how GitHub actions work with ci CD. You go to engineers back in Travis back in the day; they just haven't had to solve many of these problems. So, you have to give them the room to sort of like re-exploring that. If you provide them with enough time and the right sort of the issues to solve, the mission of like, Hey, this is where we want to get us there, then they're going to learn stuff that you don't know. And that, to me, is exciting. Because, as I said, I'm not smart enough to know everything. When people know more than me, I can learn from them. That's the type of project we've been setting up.

Future

Ryan Jones Q: How would you approach building an application like the Greenfield application from scratch and 2020?

Ken Collins: I would use the tools most familiar to me, and that is rails. The Landy gem that I wrote commoditize is an API gateway like Apache or Nginx in the cloud. Hence, one of the things that I learned early on is that API gateway, and the way that rails work, is Express. What are some of the ones with Python-like flask to translate this event into another event? So that means, once I did that work of translating the API gateway event into what was called a rack event per rail, you can get a Rails app in there. Now the problem is, I can write Rails apps very quickly.
In fact, we have a couple of projects that Custom Inc, where I've just gone in and done internal tooling for the team. That is a basic Rails app. The one app that I did a few months ago was like Photoshop in the cloud, where I hooked the rails up to DynamoDB and s3. I created a system for our creative teams to develop layered content, images, and personalized images. So, they can upload Photoshop layers and PNG files; they can spec out a replacement of images for personalization. For the social teams or the marketing teams, they can now self-service inside using this application.

It's basically a Rails with turbolinks and a bunch of live hips image processing to sort of compress images down in real-time. I will do that over and over again. There are so many internal tools, and it is called intrapreneurship. If you're in a giant company, you probably need a lot of internal tooling. With rails connected to DynamodB, now we can connect a Rails app to RDS to the newly released RDS proxy. This makes the database connections work for those that still may be working in from like a classic form, not forcing everybody to go down in our DB if they're not ready for it.

Ryan Jones Q: With the software development, would it be a no-code industry now and then after we hit that level, you know, what would come next according to you?

Ken Collins: I co-opted a full-stack service. I want people to say, hey, I want to do full-stack serverless. If you start as a landi QuickStart Guide, you will have a Rails app ready for JavaScript development, whether with TypeScript or with its web, Packer integrated into it. You throw DynamoDB in there or an RDS proxy, and you've got your full-stack literally. You can take the existing frameworks that you don't have to invent new frameworks for serverless.

I talked about JavaScript frameworks. But what if the commoditization of serverless and all the complexity is below the waterline? To me that that's pretty much here. I don't know if we've recognized it right like that. So, you probably need somebody to set that up; I can have my Rails app and Heroku right now. I think we're slowly going to see more of that and realize that the tools we have are the amplified stuff that is just incredibly powerful right through a lot built-in natively to AWS. We're going to see that the promise of lambda, the commoditization of that compute, and API gateway is the commoditization of Apache or Nginx. I don't have to think about it, and I can deliver business value almost immediately. I think that realization is common.

Ryan Jones Q: You recently wrote an article called monolithic ideas for many file systems, following the release of AWS lambda file systems that looks like so what does this release mean and what does it open the door to?

Ken Collins: We're going to see more news like this repeatedly, where you get these releases changed the game on like, what's available to do in AWS lambda. For us, efms unlocked two big critical applications that we could sort of move over to it. I spent some time and did some postulating on what are some bad ideas that people could do. But for us, I think we're going to do some excellent uses for it. We have one font rendering service where we needed about 200 font files to be shared between the services. Moreover, we've always wanted to port that over to lambda.

You have the storage that can be attached and synchronized, and you don't have to worry about the temp storage like we were going to do a solution before where the funds may come in from an s3 bucket or something like that. I think it's been on the list like people have been asking for a feature like that for years. There is bound to be more stuff like that's going to come out that will radically shift precisely what you think.

When Chris Mullins talked about the VPC configs, it used to be that people would discount land and say, Well, I can't do this. I needed to speak to a VPC, or perhaps it didn't need RDS like a database before the proxy was out; that would cut that argument off. Once the VPC cold starts were solved, it opened up many things for us, and the elastic file system was a massive win for us. I can't wait to see what people do with it, and I'd like to see some more content on there. I have heard big machine learning things. But I'd like to see what people are thinking about it from the legacy of current frameworks, how you might use that, and more.

Ryan Jones Q: What would you tell those companies when you ask them to go the serverless route, or did you ask them to go an alternative way?

Ken Collins: Our success story has identified points that we can find the expensive stuff as you're scaling your infrastructure. It could be a single rails controller action, like one that slowly can be distilled down to one function, take that thing out, and move it over to lambda. And get familiar with the fact that you can sort of hyper-optimize for one critical path in your infrastructure. That is the sort of whet your appetite, and then do the training, do the things in place that take that cycle and iterate on it. But then also recognize that lambda can do the full stack for you to sort of put these larger frameworks in. So, there's a lift and shift story there.

Keep watching out for the news and look for when you can take x service and put it into lambda. I think there's an article about the lift and shift shot clock, by forest. The goal is not to say, Hey, can I take and lift and shift everything that I've done? It is easy to end lambda, but you want to learn about the new things you know about lambda. So how would you like to evolve a service that maybe you lift and shift at work? You would want to rewrite the service to take advantage of the infrastructure that's there for us. A lot of the ones were like observability and instrumentation.

Once you're in lambda, a cloud watch is there. Cloud watch insights embedded metrics. So we are using tools like New Relic insights and pushing things off. Now we're looking to use Cloud watch and its observability and instrumentation tools. So hyper optimized, dump the big stuff in there, and then look for the ways of either rewriting these big things or what are the ways that when you would approach a new project, that you might approach it from a cloud-native or serverless first approach? That's a lot of what we're doing.

Conclusion


Ryan Jones Q: Where can people follow you and where you are promoting your stuff?

Ken Collins: You can follow me on Twitter. There is the customer technology blog, a technology customer calm; the Landy product site is at landing custom, a tech COMM. I can include all these links out there, but I am an AWS service hero. I would be happy to talk about anything or answer any questions if anybody had anything.

Ryan Jones: Perfect. To those listening, this has been the Talking Circles podcast with Ryan Jones. If you like our show and want to learn more, check out talking serverless.io. Please leave us a review on iTunes, Spotify, or Google Podcasts, and join us next time as we sit down with another fantastic Serverless guest!