During my career, I developed certain opinions. Some of them left a particularly deep mark, reinforced by years of experience. I tried to articulate what they have in common. The general idea is that the only thing that matters is the code in the store. Only production is important, the rest is a technical duty.
This point of view probably arose due to long years of work between the departments of operation and product development. I am confident that these departments should strive for optimizations that allow the code to be brought to the production as soon as possible, as well as respond to incidents in the production.
This idea, as well as all the implied actions associated with it, may seem counterintuitive or controversial, so I want to reveal it a little deeper. Below is a list of practices and principles that I consider correct based on my opinion that the only thing that matters is the code in the production.
Engineers must use their code
Developers are the main specialists in the code that they write, and they should be responsible for its operation in production. In this context, "operation" means the breakdown, instrumentation and monitoring of the code, as well as assistance in resolving incidents related to or affecting the code. The responsibility for code operation is stimulating - it encourages developers to write observable and easy-to-debug code, and also connects them with what is really important to the customer. It encourages them to be interested in how their code behaves in production. It is important that engineers serve the code on call, this creates a positive feedback cycle and makes it easier to understand whether their attempts to write ready-for-production code pay off. I heard that someone is complaining about the prospect of on-call service, so I will ask only one question: if you do not serve the code, then who will do it?
If you do not already serve code on the call, but you want to do this and you can influence this decision, you can do the following: Set up PagerDuty (or similar service) schedules for each group of engineers responsible for specific services or parts of code. The high-quality schedule contains 6-8 engineers. There are many variations, but usually weekly shifts are used when you are a spare week on a call and a main week. Setting up alerts is a separate topic that probably deserves a whole post, but in general it is worth focusing on what affects customers (see Symptom-based alerting); Remember that you are ultimately responsible for responding to alerts, so you can change them.
If you can avoid creating something, then it is worth doing
Code is the most expensive way to solve a problem that does not concern the main area of your business. For most small or medium-sized companies, there are unsustainable or better hosting solutions that solve a wide range of standard problems. I mean such things as a hosting of git-repositories (Github, Gitlab, Bitbucket, etc), tools for observability (observability) of systems (Honeycomb, Lightstep, etc.), cloud-based databases (Amazon of RDS, Confluent Kafka, etc.), the systems of alert (PagerDuty, OpsGenie, etc.) and also whole bunch of other standard technologies. This also applies to infrastructure - if possible, do not deploy your own Kubernetes clusters (note: do you even need Kubernetes?), do not deploy your own load balancers if you can use Amazon ELB or ALB.
Unfortunately, the syndrome of rejection of someone else's development is very common and some companies burn very much on this. I saw teams spending time and money reinventing components with better and more proven alternatives on the market. The same teams almost always spend years later fighting the technical debt accumulated as a result. If you are on such a team, and you have the desire and ability to influence changes, then start rolling back such decisions one at a time. Migrate your databases to the cloud, migrate the feature flagging system to the SaaS tool. Continue the process until the only software you support is the software that benefits your customers. You will greatly benefit from this.
Simplify the deploy process
Deploys should be a frequent and boring process. Engineers should be able to perform deploys with minimal manual intervention. The success of the deploy should be easy to see. (To do this, the code needs to be supplemented with observability, as I mentioned above). In addition, if something goes wrong, then the deple should easily roll back. Frequent deples imply small volumes, and small deples are usually easier, faster and safer to perform.
In many commands, there are schedules for banning deploys (they can be called code freeze) or they apply deples policies such as "No Deples on Fridays." Due to such periods of downtime, changes can accumulate, which increases the risk of accidents.
If you work in a team that is afraid of deploys, then spend part of development time improving the pipeline of deploys to eliminate this fear. In my previous team, we were able to reduce the time of deploys from three hours to 30 minutes, which significantly increased the confidence of the team in the process of deploys. The natural side effect of this was that the developers began to perform deploys much more often and stopped waiting for the accumulation of changes sufficient to create a "release" (this word was synonymous with deploy).
The system is best understood by those who work with it
This applies to any part of the socio-technical systems in which we all participate. In the case of software systems, the level of operational risks is best understood by engineers performing daily depots and serving the most critical services on call. There is a sad trend - managers usually overestimate the degree of progress of teams in some transitions, for example, when moving to the cloud, DevOps, etc. The higher in the management chain, the greater this error in estimates. Engineers who perform depots and call when something breaks know where all the skeletons are buried and what you need to invest more labor in. Therefore they have to be the main interested persons responsible for placing of priorities of technical work.
Another manifestation of this principle applies to teams working with the platform or providing technical services. If you are responsible for creating a common component that is used in your organization (for example, messaging, CI/CD infrastructure, shared libraries or services), then you are waiting for an unpleasant discovery: people who use the result of your work, in many cases, know more about it than you. They indirectly understand how it serves consumers and know what obstacles and problems need to be overcome to make the system work. Listen to them to understand how to improve the UX of your services and tools.
QA measures reduce quality
Many teams have a manual QA step that is performed before the deeds. I assume the idea is that someone should run automated or manual tests to verify that the changeset is ready for release. It looks soothing - you have a person (or collective) "verifying" release before release, but it falls victim to a lot of false assumptions and creates discrepancies that do more harm than good.
First of all, if you need some manual work before you release the depot, it's a hindrance—if you're deflating easily and often making minor changes, no QA department will be able to handle the test of each case, and will inevitably interfere with the deflation. It's bad. If you have manual tests, automate them and build them into your CI pipeline (if they really have value).
Second, QA departments often lack context and time. As a result, they can start testing "impact" instead of "tasks." For example, I've seen examples of how QA departments spend time testing that something happens in a database when doing UI actions. What happens when a developer refactoring this UI component and changing its underlying data model? Functionality will continue to work, but tests will fail. Since two departments are involved in this, it will take coordination and time to fix it. I also saw how QA departments prevented the deeds from passing tests after adding caching to the CDN layer — a TTL of 5 seconds news feeds can even be invisible to the user, but can spoil QA tests, causing unnecessary conflicts between QA developers and engineers.
Fortunately, this problem is easy to solve. Instead of a special QA department working on manual and automated control tests performed in a fictional QA environment, it is necessary to direct this team's efforts to continuous testing in the production. Instead of being a gateway for the depositions, QA can continuously check the operation of the product. In addition, QA departments are well suited for chaos-engineering projects that intentionally inject malfunctions into the product. QA engineers can also work to increase the reliability of the CI/CD pipeline, so that the drops no longer become a nightmare.
When possible, always strive for boring technologies
Systems are inherently unpredictable, and when things start to collapse, you always need to have insurance in the form of a lot of accumulated knowledge. In addition, there are mandatory routine operations (deeds, database migrations, etc.), so it is convenient to use widely used and well-tested tools for such things. When I think about this principle, I often think about databases. MySQL has many intricate features, but it is widely used and in most cases it is worth following.
Very few organizations have the resources to debug unique problems. We don't need unique problems, especially when performing routine tasks like saving bytes to disk, choosing a new cluster leader, garbage collection, requesting temporary data, and so on. The emergence of unique problems will kill a small and even medium-sized team. It takes out all your creativity energy, which is best used to create benefits for customers who want to pay for your software. Spend your innovation counters wisely!
If you use boring technology, you can rely on a large community of users. You can swear as you like, but there are very few PHP problems that someone else hasn't encountered before you. Today this is probably true for the widely used versions of Ruby on Rails. I often say that I seek to be in the third wave of technology development. The first wave is an organization at the forefront of progress. The second wave is people who are willing to take a certain risk. Let these two groups go ahead, let them face all the big problems, and then step on your own, benefiting from all their hard-earned experience.
Simplicity always wins
There's little to say, but we all write YAML and JSON instead of XML, and we use HTTP instead of CORBA, RMI, DCOM, XPCOM, etc. Really? Similarly, I would better debug problems in the LAMP stack instead of micro-service architecture.
Microservice brief: Like many technological trends, they are positioned as a panacea. Don't get me wrong: When properly designed, micro-services can solve some specific problems, as well as many solutions to complex problems, while requiring certain compromises. If you want to move in this direction, I have an opinion on how to do it, but I also think that it should be avoided as long as possible.
Everything always breaks
It is impossible to avoid failures, and even undesirable. Accept the fact that failures are inevitable, and focus on how to respond to them. This means that investment will be required in an ever-improving incident response process. There are no ready-made systems suitable for any company or team, but you should have a good idea of what to do when something begins to break, and have established mechanisms to learn from these situations and improve your processes. Invest in incident analysis. This is a huge area with many valuable tools and resources to maximize the efficiency of investments in case of incidents (or when they do not occur!).