In this guide, you will learn how I tend to deploy a website. Why listen to me? I've been deploying websites for large enterprise level organizations for over 15 years. I've worked with some of the biggest companies in the UK. I've worked within CMS vendors where we helped clients ship code on a near-daily basis. I've worked in digital agencies, where we worked on multiple projects. I've also worked client-side, where the cost of getting a release wrong could be up to 50 thousand pounds an hour, so mistakes were costly.
In that time, I've learned a few basic rules of how to increase the chances that your release will go smoothly that I will share below. Stressful releases mean late nights, angry team members and going home at the end of the night venting why your job is so shit.
Before starting any deployment process to production, you need to know exactly what you're doing. Trying to figure it out on the day is a recipe for disaster. Trust me.
Other professional professions have known this fact for years. Take flying a plane. In order to become a pilot, you need hundreds of hours of practice flying. Before they fly they go through pre-flight checklists. Hundreds of hours are put into creating these checklist pilots need to follow. Thousands of plane crashes have been researched and the mistakes have been included in the checklists. In aviation, they learn from their mistakes as the stakes are so high.
On each flight you take, there will be a co-pilot to ensure that all the preparation steps are followed. When we ship software though, no one in your company is likely to enforce this on you. When you don't have a clear picture of what you need to do, you significantly increase the chances that things will go wrong.
THE CHANGE HAS TO WORK!!!!!!!!
Before releasing anything that you deploy it needs to work. I can't count the number of times that small last-minute changes that 'won't break anything' is the cause the whole deployment fails and needs to be reverted.
There's a common occurrence in all companies to ship code quickly, untested, rather than ensure it works. If the code you're shipping is wrong, no matter how good your deployment process the end result will still be a failure, no matter how hard you try, or how good your deployment process is.
Document All your Release Steps!
You get paid by a company to act like a professional. You wouldn't be too impressed if you went into life-threatening surgery and the doctor turned up last minute, with no clue what was wrong with you, or what he needed to do in order to fix you.
The same principles are true for releasing software. If you are a fan of 'winging it' however it's not a scalable approach, especially when you work in a team and you have people relying on you. You may find documenting stuff is boring, but it needs to be done. That's why you get paid the big bucks.
When I started my career I thought I knew it all, I made very few mistakes when shipping code - however - I still made some unneeded ones once in a blue moon. The bigger issue is that people on the team didn't know what was going on, which meant they couldn't help me as much. This meant that I created a situation where I was the only person who really knew what was going on. As people weren't helping me, I felt like I was all alone in the process. This meant I tried to do more and more on my own. This caused friction between everyone and it caused me stress. The funny part... as I was the cause of the issue.
These counterintuitive things often occur in releases when people in the team are unsure of what they need to do during a release. To get around these issues, document it. In times of crisis, explaining things to people addresses stress. All these issues can be pre-thought about, documented so they do not become issues later on.
Another annoying thing about the small stuff. After you have done 100 releases of the same website, the steps that guaranteed you success, don't seem too important... the releases usually go smoothly, so you skip them. Then something goes wrong.
If you don't believe how important it is to follow the steps that guarantee your success in any professional venture, I recommend reading 'The Checklist Manifesto'. The book documents numerous case studies why checklists are essential in order for complex processes to go smoothly.
Get the right people
The next thing you need to ensure is that everyone is on the same page. Have you got access to all the right people - people with the right permissions? Have you got a tester? Do you need an infrastructure person to flick any switches?
Before you release any code, you need to ensure the right people know. If you don't do this, you'll likely get in trouble with someone. In these instances, if you let everyone who cares know you might be doing something that could affect them, you've done your part. How you let people know will vary depending on where you work, usual candidates are:
- Email - Ensure you use distribution groups, rather than individual email addresses. Helps avoid issues when people leave jobs etc...
- Customer service
The best approach when it comes to communication.. the more the better until you get told differently. If your change can break someone else's system, as long as they know about it beforehand, no one can blame you. If you break another team's system and you didn't tell them, they then waste time trying to figure out what went wrong, people tend to get mad. In short... it's your fault.
Ideally, your deployment process should be automatic and triggered by a single click of a button. If you manually need to copy any files from one place to another place, your process is wrong. A general rule of thumb with the companies I've worked with is that the companies who manual deploy code, tend to have more bugs and spend issue deploying and the developers spend more time fire-fighting. Usually, the thought process is they don't have the time to do a proper job. The companies who invest in automation usually have a little more overhead at the beginning, but, this is a massive saving long-term as bug fixing time is massively reduced on an on-going basis.
You may not be able to move to a CiCd pipeline today, however, if you have to do a release, scream and shout about it. If the release goes wrong because you mess up a manual step in the process, it's not your fault. It's only your fault when you give up trying to move the team towards automation. Any manual steps should be automated. Any issues made in a manual step is a process issue, not a personal issue.
In a manual process, the pre-prep stage is vital. You need to have all the release steps documented and agreed.
I can't say the amount of times after shipping some new code, a tester has flagged there's an issue. After a period of time, the issue magically solves itself... mainly due to a cache resolving. After deploying make sure you kill whatever cache is needed. Testers kill their browser cache, you clean your output cache, Redis cache, CDN cache, WAF cache, mongo cache.. if you don't understand how your website is cached. Stop and figure it out.
Whenever you release you need to test. Ideally, in a perfect world... all testing is done via automation and your tests are of that high quality, so no bugs ever make it to production. In reality, most projects I've worked on have good test coverage but still need some manual checks. Get your tester to go through the site. They should have some test scripts that cover the most important journey on the website. Journeys like someone buying something etc... if they have script ensure everyone in the team has access to them.
The better the automated testing, the quicker these steps will be
This is pretty much reversing the steps in the communications step. Anyone you told at the start of your deployment you were shipping, needs to be informed when the process has ended. it doesn't matter if it's a successful deployment, or, you needed to rollback. As long as you have told people, your responsibility is over.
If you follow these steps I can guarantee you will have less stress when deploying than you do now. Over the years, I've easily done over 750 live releases. I've worked with large enterprise-level clients, I've worked within some of the UK's biggest digital agencies, I've worked as a freelancer on smaller projects for companies around the world and there is definitely a common theme.
The teams who follow the basics religiously have more successful releases than the companies that wing it. Most of the steps I've listed will sound like common sense, but still, most companies tend to forget to follow them and then act surprised when things fall over.
Having a living document that lists every step in your release process will instantly give you smoother releases. if you're thinking that there are too many steps to list, then you know your process is wrong.