How to Handle a Web Infrastructure Migration Without Losing Sleep Over It

Often overlooked, the web service infrastructure of an application quite literally keeps the lights on. But what happens when we need it to change?

Infrastructure is a strange beast when it comes to large-scale software projects. As if ensuring that an application runs smoothly and as bug-free as possible wasn’t enough, we also have to ensure that the application’s hardware has appropriate bandwidth, storage, memory, and compute power allocated. Small amounts of care and consideration early on pay dividends down the line when the infrastructure needs of an application change.

Much like the coding principles and patterns we follow at Envato for a consistent codebase, we also strive for consistency in defining the infrastructure for our applications. This consistency is important as it allows engineers to move across projects and debug unfamiliar applications more easily. 

This infrastructure can cover all manner of applications, and at Envato, one such application is the internal service we use for managing customer-facing information banners on our Market sites; you have probably seen them when we’ve had a sale on! In this blog post, I will run through how we migrated the infrastructure for the appropriately-named Banner Manager from Heroku to Shuttle, our in-house infrastructure & application deployment toolkit.

The Current Infrastructure Setup & Why We Want It To Change

The infrastructure for the Banner Manager application is currently managed by Heroku, a ‘platform-as-a-service’ that allows developers to only worry about the code for their application and manages all associated infrastructure needs, from provisioning and deploying to scaling. Even the URL and domain name are taken care of to make the application is accessible on the cloud. 

Now, I know what you’re thinking. Having all of the complexities of web service infrastructure managed by a third party seems like a pretty sweet deal, why even migrate off Heroku anyway? There are a few important reasons.

One reason is cost. Platform-as-a-service services aren’t free, and our monthly Heroku bill represents the complexities we’re offloading to a third party. Directly managing our infrastructure in-house means that the only price fluctuations we need to worry about are the costs of the AWS resources themselves. The second factor is consistency. As the number of applications Envato manages grows, the fewer outliers (in terms of how their infrastructure is managed) we’d like to have. Ideally, every single Envato application would be running on Shuttle, and this migration is one step closer to realizing that.

Software Infrastructure? Like, The Cloud?

“There is no cloud, it’s just someone else’s computer,” or so the saying goes. While it is a bit more nuanced than that, the bottom line is software applications ‘on the cloud’ do indeed run on some kind of physical hardware somewhere in the world, whether that is someone’s personal computer running a Plex server for their living room or a network of Amazon Web Services (AWS) data centers serving up an online marketplace for digital assets for users around the world.

The process of managing this web service infrastructure as a developer has undergone many changes over the years. Initially, developers would have to reach out – often in a literal phone call – to a cloud computing business like Rackspace every time a change was required. While this eventually evolved to filling out online forms for these changes (a bit better…) and representing your infrastructure resource in a JSON file that’s submitted to AWS (getting there…), the current industry standard for managing web service infrastructure is considered to be having a personalized layer on top of the AWS CDK.

At Envato, when we spin up a new application, we don’t know exactly what it will do, but we have a pretty good idea of its infrastructure requirements: the same as the last application. For example, we know that all of our applications’ database servers will use MySQL or PostgreSQL instead of SQL Server, and all of our application servers will be running Linux instead of Windows. For this reason, we developed Shuttle, which acts as this layer to enforce the Envato infrastructure standards and best practices as guided by internally-vetted  sensible defaults – much like code review enforces the coding standards.

The Migration 

The end goal of this migration was to decommission the existing Heroku application and use Shuttle to create a stack that contains all of the AWS resources that Heroku was spinning up to run the Banner Manager application. Think of a stack here as a logical partition containing relative resources. For example, one stack could handle network-related resources, and another would contain resources for an app’s asset management. A stack could even contain other stacks and provide details for them to function together. Even though all these stacks are separate, Shuttle takes care of deployment with a single action.

Taking Stock

The Banner Manager application is fairly straightforward enough as far as being similar to other Envato applications: a Ruby on Rails web app with a database. But we also wanted our new application stack’s database to contain all of the data from the existing application, so selecting a flavor and version of database software that was compatible with restoring all of the data from the existing database would be crucial.  

Some domain knowledge was also useful in determining all the resources that were required. Fortunately, I had a bit of experience developing features for the Banner Manager to know that some kind of object store to hold banner assets would be required, along with some kind of construct to handle a daily scheduled task that publishes assets to this store.

Managing Downtime

One primary consideration during this process was managing downtime. The best way of accomplishing this was actually the reverse order: we’d get the Shuttle ‘version’ of the application spun up first so that it was running in parallel to the Heroku ‘version’ and then, once we were sure that the Shuttle application is behaving as expected, redirected the traffic from the old version to the new.

Once we determined the new version met the requirements, we could proceed with decommissioning the existing application. This method results in less downtime and minimizes disruption to the application’s users.

Deploy, Check, Repeat

Developing in parallel meant we were able to iterate and test independently of production, and the iterative process afforded us the flexibility to deal with hurdles as we went.

To do this, we built the Shuttle deploy workflow into our existing CI/CD processes so that there would be: 

1. Minimal friction getting a code change deployed, and
2. Clear accountability as far as what effect each change to our shuttle configuration has on the running application (remember what I said about version control?)

From there, we simply addressed whatever issue was present – whether that is a missing credential, misconfigured resource, or a missing resource entirely – and deployed again, repeating this process until we end up deploying a version of the app that matched exactly the version that was already running with Heroku.

By the end of this month-long process, our attention to detail had paid off, and we could mark this down as another successful migration.

Conclusion

Much like a house is only as stable as its foundations, a software application greatly benefits from modern practices and technologies applied to its infrastructure. When looking at the ecosystem of all the Envato applications at scale, the importance of stable foundations is even more pronounced. By migrating the Envato Banner Manager application from a third-party platform-as-a-service to an in-house toolkit, we facilitate Heroku-like IaaS functionality on AWS which gives us the benefit of both worlds and move another step closer to consistency in web service infrastructure management across all Envato applications.