Purposeful automation may sound like just another buzzword. And maybe it is a buzzword, but it’s one that actually matters.
In this post, I discuss what purposeful automation means, what it doesn’t mean, and why it’s a valuable concept for DevOps teams.
Automation Then and Now
There was a time when "automation" meant the use of a machine's mechanical capabilities to perform repetitive tasks, typically with an operator in part-time attendance to monitor (but not control) the process.
These days, of course, the term is as (or more) likely to refer to the use of machine intelligence to perform complex tasks which may involve sophisticated decision-making, often with an equally automated monitoring system set to notify a human operator only in the case of an error or other condition requiring intervention. Automated tasks are still largely repetitive and time- or labor-intensive. For the most part, what has improved are the capabilities of the machines carrying out the tasks, along with our ability to model complex tasks and convert them into machine instructions.
Why Purposeful?
Why, then, do we need to be purposeful about automation? Why can't we just automate everything that can be automated, and concentrate our non-automated effort strictly on those tasks which are currently resistant to automation?
The short answer is that while we may soon be able to do just that, we aren't quite there yet. The longer answer is that given the technology and resources currently available for automation and our current level of understanding of both human- and machine-based processes, it is both more efficient and more cost-effective to apply automation at those points where it will have the greatest positive impact on the process in question, rather than across the board, or all tasks for which automated tools are available.
What does this mean?
Pivotal Points and Bottlenecks
In any process, there are points that are pivotal—actual or potential bottlenecks, labor- and personnel-intensive tasks, tasks which are complex (but fundamentally algorithmic) and time-consuming, or that can only be performed intermittently. There are other tasks which function more like a smooth stretch of pipeline, and still others which are so complex that they appear to be resistant to any kind of simplification or even algorithmic modeling.
Defining Purposeful Automation
Purposeful automation is the automation of specific tasks and processes (within the context and scope of the overall processes) that will lend themselves most effectively to automation—and if they are automated, will provide the greatest improvements to the process (in terms of speed, efficiency, cost, and so on).
In other words, don't just automate everything, and don't simply apply readily available automation techniques because they are available. Automate where it will make the biggest positive difference, and where it will cause the fewest problems.
Blazing Barcodes
To see how this works, consider a very basic example (from the relatively recent history of retail shopping:
The introduction of barcode scanning automated a pivotal element of retail sales. It focused on key bottleneck processes (checkout and inventory), made them more efficient by at least one or two orders of magnitude, and didn't get in the way of the overall supermarket shopping process. The effect was to increase the speed and volume of sales, stabilize inventory, and cut costs.
From the beginning, all of the key players in the development of barcode technology understood its function and value in both retail sales and the supply chain; it was highly focused, and in many ways, could serve as a textbook definition of purposeful automation.
From Retail to DevOps
What does purposeful automation mean for DevOps? After all, DevOps relies heavily on automation as it is, and at least in theory, it should be fully automated.
In practice, of course, the automation of a DevOps continuous delivery pipeline is always less than 100%, and much of the time, the actual level of automation may be surprisingly low. Why?There are three basic reasons:
Some tasks are genuinely difficult to automate (particularly creative/decision-making tasks involving design, coding, and content development).
It seemed at one point easier to leave some tasks unautomated, because (for example), the effort required for automation was greater than the effort required to do them manually.
The individual tasks in a process are automated, but the level of integration between them is low, so the automation itself is less efficient than it could be.
Room For Improvement
This leaves plenty of room for purposeful automation, and plenty of value to be gained from it. Even difficult-to-automate creative tasks can benefit from the kind of purposeful automation which moves easily automated/non-creative elements of the process to the background, leaving designers, programmers, and content providers to concentrate on the hands-on/creative parts of the process.
Inefficiency as Legacy
Both currently unautomated tasks and automated-but-poorly-integrated processes are major potential targets for purposeful automation, since they represent points where inefficiencies and bottlenecks can develop over time without being noticed.
This is particularly true of testing, which often lags behind other elements of the continuous delivery process when it comes to automation. Major increases in speed and overall efficiency can be achieved by shifting a testing system which combines legacy manual operations with poorly integrated manual tests to a fully automated online testing service.
A Purposeful Automation Checklist for DevOps
Here's a quick checklist for determining which elements of your continuous delivery process may benefit from purposeful automation:
Identify all unautomated tasks and subtasks, listing the reasons why they are not automated. Don't judge the reasons, just list them. "Haven't gotten around to it" is as valid as "intrinsically difficult to automate." In each case, ask yourself how easy it would be to automate the task with current automation tools.
Identify points where integration between automated tasks is crude, basic, or extremely simple. These may be OK as-is, but it's important to know where they are. Ask yourself how easy it would be to fully integrate each series of tasks into a fully automated process.
Identify automated processes for which the automation itself is crude or very basic (for example, a log parsing tool that produces difficult-to-read text output). The question that you should ask is, "How could we automate this process to get the greatest value out of it?"
Look Upstream and Downstream
For all of these items, you should also ask yourself what the upstream and downstream effects will be of increasing the level of automation or integration. In some ways, this may be the most important question you can ask, since bottlenecks and logjams aren't always apparent at the point where the problem originates.
An increase in efficiency for one process may place greater demand on upstream processes, or put greater pressure on downstream processes. This may in turn expose inefficiencies in the affected processes, or in the very least suggest ways in which they could be changed to improve the overall flow of the system. In some cases, such as testing, full automation may remove or loosen up logjams throughout your continuous delivery chain.
Identify Key Points, Automate, Repeat
This checklist will allow you to identify the points in your continuous delivery process where purposeful automation is likely to have the greatest effect. Typically, this means applying increased/improved automation not only at those points, but also at any upstream and downstream points which are affected.
Note that this is an iterative process—If you do it now, you will do it later as well, not one or two times, but repeatedly. Your continuous delivery process is no more static than the application that it delivers, and as your process and the tools that you use change, the points at which purposeful automation will make a difference also change.
Ultimately, perhaps, purposeful automation in the context of DevOps is a logical corollary of the recognition that infrastructure is code, and that like application code itself, it is subject to continual monitoring, testing, review, and updating. It is the application of DevOps principles to the DevOps process—continuous delivery of the continuous delivery infrastructure itself.
Michael Churchman is a Fixate IO Contributor. He started as a scriptwriter, editor, and producer during the anything-goes early years of the game industry. He spent much of the ‘90s in the high-pressure bundled software industry, where the move from waterfall to faster release was well under way, and near-continuous release cycles and automated deployment were already de facto standards. During that time he developed a semi-automated system for managing localization in over fifteen languages. For the past ten years, he has been involved in the analysis of software development processes and related engineering management issues. He is a regular Fixate.io contributor.
