How did DevOps come about?

New Thinking 25 April 2016 Peter-John Lightfoot

In a time long before the advent of computers and digital information – and indeed throughout the rise and settle of two industrial revolutions – manufacturing continued as mankind's favourite brainchild. So many brilliant minds worked at it, for so long, that by the time software development came along, the processes and guiding principles behind them were delivering such tremendous value that no-one thought to question their relevance. Like a cookie cutter, Software Development fell in line as yet another manufacturing minion.

Except it wasn’t.

Software Development was intrinsically different. Requirements were fluid to the point that they were obsolete before the features could be delivered. Business wanted more and the industry promised to deliver.

The early contributors to Agile were trying to come up with ways and means of dealing with such changing requirements, but – while they made giant strides in the people and process arenas (with help from some of the stalwarts from the manufacturing industry such as Kanban and Lean Principles) – they were severely hampered by technological limitations (or more accurately by their adopters’ lack of confidence in the technologies that were available at the time).

Even so, the Software Development industry continued to inspect and adapt – and little by little it got to fixing itself. In the space from where Agile left with Lean to inform the rest of the world of a new way of thinking, Development and Operations gritted their teeth and fought through to bring the industry into its own. From this battleground, DevOps emerged as a movement that promotes the tools, processes and culture that can bring Software Development to levels of efficiency comparable to that of Manufacturing, and fulfil the promises that the industry made decades ago.

One might therefore – quite accurately – recognise it simply as the present state of the same evolutionary track that gave us Agile… but the easiest way to gain a deeper understanding of DevOps is to take in its story…

How did DevOps come about?

The story of DevOps is the story of Agile Software Development, continued.

Some time ago, there were some software developers who were unhappy with the way things were going. The engineers were mostly unhappy because they invariably had to work ridiculous hours to meet the ridiculous expectations of their leaders. The leaders were unhappy because, while the software engineers were heroes of their time and able to accomplish tremendous feats of engineering, they were – for the most part – not so able to see the tremendous commercial potential of their creations and required constant guidance and direction. The unhappy software developers tried to change things and some were more successful at it than others.

Little pockets of heroic effort, scattered throughout the industry, started coming up with small ways to improve the situation. Even before the 1980’s, we see the emergence of methodologies and practices with cool names like 'Adaptive Software Development Process', 'Evolutionary Project Management' and 'Competitive Engineering'. In the depths of the industry, largely unseen by mainstream business, these pockets of influence grew and lighted others.

During the 1990’s, some more methods with cool names emerged – 'Unified Process', 'Dynamic Systems Development Method', 'Scrum', 'Crystal Clear', 'Extreme Programming', 'Adaptive Software Development', 'Rapid Application Development' and 'Feature-Driven Development'. At the same time, a polarisation occurred that pitched these emergent and evolving 'lightweight' software development methods against the perceived 'heavyweight' traditional methods – which naturally only served to provide the rebel pockets with identity, unity, and purpose. So much so that – even though at that time Agile was still not a 'Thing' – these methods are often, in hindsight, collectively referred to as 'agile methods'.

In February 2001, 17 of the rebel leaders came together and declared war on the traditional software development methods by writing the Agile Manifesto into existence.

And then – of course – Agile became a Thing.

Like the stem cell that divides to become an exact copy of itself, but then also differentiates to specialise in ways different from the primary cell, so did these rebel cells leave two separate, but entwined paths of evolution behind. On the one path, Agile Software Development had brought new life into the trenches of the industry, where a more sustainable pace and improved interaction between individuals meant that software engineers could continue ever more successfully to find ways of improving the software development process. And on the sibling path, continued focus on improving the efficiency and efficacy of people and processes inevitably and inexorably drove its success ever deeper into other parts of business as this Agile Thing.

"Agile is a movement that was brought about to break down the walls between Business and Development in order to work more efficiently and be more effective".

And so started a decade of confusion. Agile Software Development was, in effect, usurped by Agile. And Agile was a Thing. More importantly, it was better loved by those in business who held the purse strings – and so it was better packaged, better conveyed and, ultimately, better adopted.

So what of this other Thing called 'Agile Software Development'? Well, sadly, that didn’t really get to be a Thing at all. It remained in the trenches, where – through heroic effort – small pockets of Agile software developers, unhappy with the way things were going, kept making it better… because the one thing that Agile lacked was a sound technological footing… and in a disruptive digital world, that’s kind of important.

And because the new rebels also thought so, we saw other things with cool names like 'Continuous Integration', 'Test-Driven Development', 'Continuous Delivery', 'Behaviour Driven Development', 'Test Automation', 'Release Automation', 'Software as a Service', 'Platform as a Service', 'Infrastructure as a Service', 'Continuous Deployment' and 'Hypothesis-Driven Development'.

The other thing that Agile lacked was a satisfying answer to the mounting tensions between Agile Development and IT Operations. Where the former wanted to change and disrupt, the latter wanted to maintain and stabilise. Agile drove development to deliver frequent small increments, and operations tried to batch those as best they could in order to limit the opportunity for outage to specific controlled points.

Then, one day at the Agile 2008 conference, a discussion arose between Andrew Clay Shafer and Patrick Debois about Agile Infrastructure. Shortly after, John Allspaw and Paul Hammond delivered a typical 'thrown over the wall' routine at the 2009 Velocity conference to illustrate the Development vs. Operations notion. Later that same year, a series of events rally around the topic. The events were called 'DevOpsdays'. Debois spawned the first of these in Belgium because these were techy kind of people who called a developer (or developers, or development) 'dev', and operations 'ops' and – well – they were held on specific 'days'. During these events, the term 'DevOps' (pronounced with weird capitalisation) was made popular. It loosely translates to “things that are of varying degrees of importance to both development and operations”.

Subsequent to those events – and largely unseen by mainstream business – many of the notions that have come before were integrated into something that could coherently and succinctly convey the enormous breadth of content needed to answer the original question of how software development can efficiently and effectively deliver the value that business needs – without leaders getting upset, software developers being unhappy or IT operations staff burning out. Some are more successful at this than others.

Describing the answer is still a work in progress, but even so, now – finally – DevOps is a Thing…

"DevOps is a movement that was brought about to break down the walls between Agile Development and IT Operations in order to work more efficiently and be more effective".

This blog post was originally published on Peter-John's blog site where you can find other articles about DevOps and software delivery. [Image credit]