Join the DZone community and get the full member experience.
Join For Free
Deploying software into production is one of the most crucial activities in the software development lifecycle. It's a moment of both excitement and risk -- excitement because new features and fixes are being released to users and risk because any misstep can lead to downtime, bugs, or poor user experiences. In this blog, I will walk through the best practices for ensuring safe and smooth production deployments, tailored primarily for experienced software engineers.
In this article, we'll dive into strategies for mitigating deployment risks, optimizing team efficiency, increasing deployment frequency, and improving the overall software delivery process.
Traditionally, many teams assign the on-call engineer or service engineering teams to handle production deployments. While this approach ensures accountability, it brings several issues to the surface:
If the on-call engineer/service engineer is busy resolving critical issues, deployments can be delayed, often resulting in the deployment of several days/weeks' worth of changes at once.
The on-call engineer may not be familiar with the details of each change, requiring pull request (PR) authors to validate their changes post-deployment. This distracts developers from their ongoing tasks where they wrapped up a task a few days ago but then suddenly need to drop everything and validate an old change.
If multiple PRs are deployed simultaneously, a problem with one PR might force a rollback of all changes, including those unrelated to the faulty code.
These issues highlight the need for an optimized deployment process that minimizes risk and maximizes efficiency.
To address these challenges, here are some of the recommended best practices for safe production deployments.
One of the most effective strategies is to shift the responsibility for deployments from the on-call engineer/ a separate team to the PR author. This practice ensures that:
This change also reduces the risk of deploying multiple PRs together, as each PR is deployed individually, simplifying rollback scenarios if an issue arises. Based on the context/service, such deployments per PR can also be automated where the change instantly goes to production after a merge, and the author is only responsible for the final validation.
Even if a PR has been tested in canary or staging environments, it is crucial to perform the final validation on the actual artifact being deployed to production. Chances are that the change itself works as expected on its own, but another conflicting feature was merged right after/before the primary change breaks functionality. Doing the final validation on the exact artifact that will be deployed to production ensures that any new changes in the main branch, which might interact with your code, do not introduce unforeseen issues.
For significant changes, end-to-end (E2E) testing becomes vital. While component testing is important, it's equally critical to test how the changes affect the entire system. Additionally, leveraging buddy testing -- where another team member reviews and tests your changes -- can catch blind spots.
A practical approach is to assign QA buddies on a per-person or per-sprint basis to streamline this process and ensure thorough validation.
After deployment, sanity validation is always performed in the production environment. This includes:
Deployments should not happen during non-business hours, before extended weekends, or late evenings, as this increases the chances of issues going unnoticed until on-call engineers are unavailable. If there's an urgent need to deploy at these times, ensure:
When issues arise post-deployment, the first course of action should be to roll back to the previous stable state. Even if this means rolling back other deployed changes, it is often safer than applying a quick fix in production, especially without thorough QA validation.
If a rollback isn't possible due to irreversible changes (e.g., schema updates), make sure any emergency fixes undergo proper QA and staging validation.
Feature flags are your best friend when deploying large or risky changes. This approach allows you to:
Before any deployment, ask: can this be safely rolled back? Consider potential risks, such as changes to the database schema or cache structure. If rollback isn't feasible, carefully plan how to mitigate potential failures, such as by using feature flags or additional testing in staging environments. Also, consider the option to auto-rollback if certain success metrics are not met. This can be combined with auto-canaries, where a change initially only goes to a certain percentage of the main audience and is rolled out to the full production set only if the success metrics from the canary look good.
Deploying software to production is a high-stakes operation, but adopting these best practices can significantly reduce the risk of failure and increase your team's deployment confidence. These strategies -- deploying per PR, leveraging buddy testing, using feature flags, and always planning for rollback -- empower engineers to move fast without breaking things.