When Process Becomes Ritual

There’s an old joke about turkeys. A college student celebrates his first Thanksgiving away from home. He prepares a turkey for his dorm mates just as his mother did. The turkey turns out excellent but someone asks, “Why did you cut off the drumsticks?” “That’s the way mom always did it,” replies the student. That night he calls his mom to ask about it. She informs, “Well, that’s the only way it would fit into the oven.”
The lesson here is that a process has become ritualized. The original rationale has been forgotten but we continue a process because that’s the way it has always been done.
This poses an interesting problem for IT folks. On the one hand we must be wary that our processes do not devolve into rituals. On the other, we are often asked to standardize our products and our processes and, in a sense, are forced to ritualize our procedures (“Just do it. Don’t ask why.”) Or to be less facetious, “Here, run this Python script before you start the application.”
The reason for the latter approach can be complex. Perhaps we are turning over a process to a first tier resource or an end user. They may not grasp or care about the Why but can perhaps understand the How. And the thing is, if we pay heed to industry best-practice, we *will* obscure the Why. We will reduce the appearance of complexity of our processes by hiding it behind orchestration and clever front-ends.
And that’s where a danger lies. To be able to understand the Why of many processes, one must often be intimately familiar with the technical aspects of the problem and down that path lies madness. It’s like trying to measure the coastline (2). Indeed, it is a problem of complexity because the closer we inspect a process, the more details we’ll need to grasp. When trying to build a website we cannot necessarily worry about how the underlying LVMs are assembled at the LUN level.
In fact, just such a thing occurred the other day. We were tasked with developing a backup/restore process for a set of cloud-based systems. On the front of it the process is straightforward: Each night, run a script that snapshots the underlying volumes. This approach works beautifully on our on-premise environment. From the OS support standpoint, it also met our requirements.
Alas, inspecting the coastline revealed some other problems. For one, cloud snapshots across multiple volumes are not guaranteed to run synchronously. The script may submit the snapshot job each volume within a second or two but that second can wreak all sorts of messy havoc with Logical Volumes spread across multiple Physical Volumes. I.e., exactly what we had in the cloud environment.
The solution turns out to rely on the Operating System to make these snapshots at the LVM layer. Oh no, weren’t folks saying that the OS is irrelevant in the cloud?!
And this of course leads to a larger question: Should we be doing system-level backups in the cloud? That sounds like another ritual to me.
My point here is not to say that we throw out our processes in the name of change. Rather, we must be diligent (and hopefully rigorous) in how we decide what processes and standards move forward as the IT environment changes.
Footnotes:
Measuring the coastline refers to the Coastline Paradox. The better one measures, the longer is the coastline. Infinitely long, in fact. (http://en.wikipedia.org/wiki/Coastline_paradox)

Leave a Reply

Your email address will not be published. Required fields are marked *