Recently, I was upgrading the core functionality of a Shopify app that we built for a client. The nature of this app required the majority of the changes to be tested in the production environment. While this wasn’t ideal, sometimes it’s just the way the cookie crumbles. Naturally, this introduced a certain amount of anxiety because releasing work into an active environment could have real-world consequences for our client and their stakeholders if handled improperly.
Releasing code is rarely as straightforward as hitting a “deploy” button. For example, does the release require that database migrations are run? Do you need to modify existing data? Are there tasks or background jobs that need to be kicked off? Are there changes to infrastructure that need to be made to support the new feature? Attempting to keep all of the relevant details straight in your mind can feel at best chaotic and at worst an impossibility.
With so much to sort through, I decided to consult the broader Gnar team. Together, we discussed our collective experience of releasing code and came up with a solid starting point for how to organize and prepare for the deployment. We focused on simplicity, documentation, and testability. The group also discussed gotchas and tooling that have proven useful in the past (see the “A Few Extra Pro Tips” section for details).
The Best Offense Is a Good Defense
The ultimate goal of the playbook is to make it as simple as possible to perform the release. The document could be in any format (i.e. text, spreadsheet) so long as it is easy to organize and follow. It should list out exactly what needs to happen (and in what order) ahead of the release. This will force us to think through the process and prepare us for a variety of potential outcomes.
While it’s unreasonable to assume that we can know everything at the outset or plan for every potential scenario, it is reasonable to perform due diligence and generate a comprehensive contingency plan. Having that plan handy come release time will (hopefully) help to inspire confidence in both you and your application’s stakeholders.
Before diving in, it’s important to call out that this article does not represent a “one size fits all” approach to planning a release. Every application is different. It is essential to consider a variety of factors to determine the release strategy that is right for your application and your team, such as:
- feature complexity
- application usage and activity
- development team capacity
- infrastructure
- quality assurance
In any event, I believe that there are useful takeaways in this playbook that could be applied to planning any release. The following section is intended to be a representation of a single step and should be duplicated for each step in the release workflow.
Description
The description of the step should be clear, actionable, and ordered.
Party Responsible for Execution
You might not be the person responsible for executing a given step. It could be another developer, a member of the infrastructure team, or the QA team. Outlining who is responsible and more importantly, communicating that responsibility will help to ensure that everyone is on the same page.
Associated Executables
This may not apply to every case, but there may be code or commands that need to be run in support of the step. Laying out the executables ahead of time will allow you to easily copy and paste the values into your terminal and can help to prevent spur-of-the-moment errors.
Testing Plan
When applicable, we should describe how we know a certain step was either successful or unsuccessful. This is especially important in multi-step releases. Validating the outcome of the step will inspire confidence that we are ready and able to execute subsequent steps.
Associated Risks
Each step likely carries associated risks. While it’s not feasible or necessary to list every possible risk (i.e. “The internet explodes”), laying out risks that are in our control and mitigable can help to prepare us for hangups. For example, what happens if a process we run exceeds our memory quota or a data migration doesn’t go as planned? There will likely be some overlap between this section and the Risk Mitigation Plan, the Testing Plan, and the Rollback Plan for the step.
Risk Mitigation Plan
After the risks for a given step have been outlined, ask yourself if there is anything you can do to mitigate them. In the section above, we outlined two risks: what happens if a process we run exceeds our memory quota or a data migration doesn’t go as planned? For the first risk, it may be helpful to spend time extra understanding the application infrastructure and how to scale it on the fly to support the new feature. For the second risk, we may want to create backups of the data in the event that a migration fails.
Rollback Plan
Sometimes bad 💩 happens and we need to revert our work. But no need to panic - you have a list of predefined steps and executables outlined to make it go as smoothly as possible! It is important to mention that order often matters here, especially when it comes to database migrations, so pay close attention and provide as much detail as possible to make sure you understand when to execute the rollback plan for a given step.
Notes
Take careful notes during the release process. Some steps may not require detailed notes but having a dedicated space to jot down any important observations can be useful, especially if you need to roll back and re-release at a later time.
A Few Extra Pro Tips
In addition to the playbook itself, there are a number of tips that might be helpful to explore before you jump into your release.
Consider the Optimal Date & Time to Launch
We can use data and/or monitoring tools to determine when the application is most active. If there is a lull in activity on a certain day/time, it may be worthwhile to release during that time to minimize the impact on users.
Lean on Your Team
Not everyone has the privilege of working on a team, but if you do, lean on yours! Ask teammates to review your playbook, bounce ideas off of them, or invite them to join you for a “mob” release session. Fun fact: many of the wonderful ideas found in this playbook came from asking my team at The Gnar for help with planning a complex release!
Clearly Communicate the Plan and Schedule
This is a biggie. Creating a playbook is surely helpful for the folks performing the release but what about the other relevant parties? The playbook should be reviewed by the broader team (should you have one) as well as the client/stakeholders. When reviewing it with product folks, it doesn’t need to be super technical but it is important they understand the timing and impact of the release. When I performed the release I mentioned in the introduction I set up a 30-minute review session with our client to cover the plan and their role in it, and to answer any questions. Afterward, I set up a calendar invitation so everyone was aware of exactly when the release was taking place.
Set Up Logging & Monitoring
A well-tuned logging and monitoring system is invaluable. It can help you to identify when things are humming along as expected and more importantly when they aren’t. I strongly recommend having at least some form of logging set up before your release. If you aren’t familiar with logging/APM, there are a number of popular tools out there like Papertrail and New Relic that would be worth a look. In addition to the tooling, having actual log statements in complex areas of the code can be particularly helpful as they can indicate that a certain step has failed or succeeded.
Leverage Git
Git is another important tool in your belt when it comes to managing releases. This is especially important if you need to roll back your work. Think about how rolling back a release impacts your main branch. What will your strategy be if you need to deploy a clean representation of the main branch after a rollback? There are a bunch of different strategies for leveraging Git but some aspects of Git that I found useful were tags (to mark important places in history), creating copies of branches (archives) and storing them on the remote, and the revert command to remove and document the removal of changes from history.
Final Thoughts
Each application is different and the release strategies for them are similarly unique. While your mileage may vary with this playbook as a whole, I believe that it is always beneficial to plan out your course of action and think through how to execute it. Once the plan exists, it becomes a source of truth that can be shared with team members and stakeholders to communicate the timeline, level of participation from individuals, and expectations. Hopefully, this guide and the tips within will bolster confidence during your next release. Best of luck!