The DevOps Journey, Part One

When we start working with a new client, we are often asked, “What is DevOps and why do we need it?”

So today we’re going to help everyone understand why DevOps is such a crucial function in any software business. We’ll start by taking you through a hypothetical example since it’s much easier to explain this way rather than giving you a technical definition of DevOps that might make your eyes glaze over.

A Hypothetical Example

1. Easy Times at Acme Corp

In this example, our hero’s name is Sam. Sam just accepted a new job at Acme Corporation, where he’ll be making a mobile application to help track and capture roadrunners. Acme has a limited budget and is new to software development, so they hire Sam as the sole developer. The scope of the application calls for some front-end mobile development, as well as a database and API solution.

Sam was responsible for designing the system, passing it to Pepe in Business Analysis to define and refine requirements, working with Tweety in Quality Assurance (QA), setting up a remote repository to manage the code, and then implementing features on his local machine.

He also set up a workflow for himself in which he’d build a feature, then once it was done, he would run a self-test, and if it passed muster, he would set up a database on AWS RDS, and an EC2 instance for the API and front-end app. Finally, he packaged the API build, front-end build manifest, and deployed to the production AWS RDS database. After that, he did a quick test to make sure everything worked as expected. If all was well, then he’d move on to the next feature until the app was ready for release.

Sam didn’t have a ton of experience building commercial software but he was confident that this process would suffice given the company’s needs. He told Acme’s CEO (Mr. Coyote) that he’d have the app ready to ship by Christmas.

2. Things Get More Complicated

A few months later, Mr. Coyote came to Sam and told him that he thought the app looked good so far. However, he heard from his buddies that AI was the future of technology, so he wanted Sam to add an AI roadrunner tracker to the app, and he still wanted it delivered by Christmas. This was quite a substantial increase in scope; however, Mr. Coyote authorized Sam to hire two more developers in order to make it happen.

Before Mr. Coyote could change his mind, Sam quickly hired Sylvester and Elmer to the development team. With three developers, they could accelerate progress by working in parallel on three features simultaneously.

This made Sam nervous. He was doing well as a sole developer, but he had concerns about scaling the team. He knew his current workflow would be risky with three developers. But he didn’t want to appear incompetent in front of his boss so he “hoped for the best.” He believed his concerns could be resolved day by day with the team's support.

Sam wasn’t wrong to be nervous. Predictably, the new team quickly faced some obstacles:

1. There were code commit conflicts. There were no rules or documentation on how to commit code, and sometimes Sam forgot to push his latest changes to the develop branch. As a result, Sylvester and Elmer missed Sam's updates, which led to bugs and confusion.

2. Code style was inconsistent. The three developers brought three different styles in naming conventions for variables and functions. This inconsistency slowed down code reviews and complicated collaboration.

3. Sam was still manually deploying and implementing features of Elmer and Sylvester, which meant his focus on development was frequently interrupted. He wondered what would happen if he were off for a few days; would his absence block the team's progress?

4. Elmer, a junior developer, chose an insecure library for visualizing charts. Fortunately, Sylvester spotted it as a security vulnerability during a code review, but he couldn’t help but think, "I got lucky this time, but what about next time? Relying only on manual code reviews might not be enough.”

5. One evening, after a routine release to production, the QA team quickly found a critical bug: user accounts were being automatically logged out after five minutes. This was a terrifying scenario! The team quickly jumped into the production environment to debug and discovered they had missed an important step: adjusting the session timeout when committing code to the main branch before the release. They patched the bug immediately. This time, QA caught it, but what if a critical issue is discovered by customers next time?

These challenges started to negatively affect the team’s work environment. The team was unhappy and felt a lack of trust, since a simple human error could lead to a serious production issue. If they can't prevent such issues in the near future, the app is likely to fail, and that will look bad on everyone’s CV.

3. Learning From Their Mistakes

Sam regrouped the team to figure out what went wrong. It was a long meeting, covering detailed points:

• Many risks arise when deployment is handled manually– even if there is only a single developer.

• Missing steps in the deployment process can easily lead to critical issues.

• Performing manual self-tests for every deployment doesn’t always work, and it can lead to production issues.

• The current Git workflow does not work well for this small team; they need to review, document, and test it together.

• There are glaring security issues.

• What if someone accidentally commits secret keys to the private remote repository? Those secrets could be hacked relatively easily, causing chaos for Acme Corp.

• Choosing any library or component without caution can lead to critical security vulnerabilities.

• Deploying completed features directly from the test environment to Production may be risky. They need a Staging environment (a production mirror). However, they need to keep in mind that deploying to Test, Staging, and Production will take a lot of time and carry significant risks if done manually.

• Inconsistent code styling leads to confusion and slows development time

• And there were more minor issues that we’ll leave off for the sake of brevity.

So where does the Acme Corp development go from here?

4. Adopting A DevOps Culture

The team tried Googling solutions but ultimately decided it was time to hire Marvin, the DevOps contractor, to come in and save the day. Marvin worked closely with the team to identify wasteful practices and to refine their plans to address these issues in their development process. With a combination of self-research, self-learning, and expert guidance, the team was ready to create a solid plan and move forward with a clearer path.

Sam and Marvin agreed that the most important step was to foster a DevOps culture across the team. They knew that without a strong mindset and awareness, it would be challenging to implement DevOps practices into their development process.

To build this culture, Sam hosted dedicated meetings to raise awareness of DevOps principles & practices, encourage the team to share lessons learned, and identify wasteful practices. Together, the team discussed the importance of collaborative problem-solving and continuous improvement.

First and foremost, every team member was assigned an area of DevOps responsibility so that ownership and accountability resided with a single individual. The following job assignments were pinned to a special Slack channel:

Implementing DevOps practices would take time; it couldn't be done in a day, a week, or even a month in some special cases, but the current progress couldn't pause; it needed to continue. Sam brought this challenge to the team, and they agreed to put in extra effort to make it work. As planned, they committed to a 140% effort for a month.

Story to be continued in Part 2…