Regulating data systems | #14
Ashby's Law of Requisite Variety and its implications on the control of Data Engineering teams
2023-09-14: Greetings readers. Fall has arrived in Santa Fe and with it, a seemingly late burst of much desired monsoon season. With the welcome rain our 275 gallon rain barrel is replenishing, fall harvest raised beds, and array of native plants are showing life. There is a rare fog settling over the city during my bike ride to the office space I share. Change and turning of the seasons is in the air. Tonight I’ll be speaking at Meow Wolf for Santa Fe NFT’s Paradigm Shift event on the topic of complexity thinking and its inspirations for business, art, and life. Come check it out if you’re in town.
In the field of cybernetics there’s a law about how control systems must be implemented in order to be effective - the Law of Requisite Variety from W. Ross Ashby. In active information systems, designed to respond to disturbances with an appropriate response to will bring the outcome back to an acceptable range, Ashby proposed that there must be as many “regulators” as there are “disturbances” hence there must be requisitite variety of controls. So what is a disturbance? Well that depends on the system. Anything that threatens the stability or predictability of outcomes.
For my work in data engineering this cybernetic concept becomes a very real and important aspect of the job. When we’re maintaining billions of data points from dozens of external sources and processes to transform them into predictable format the system is constantly under threat from disturbances - servers going down, data is corrupt, bugs in code, changes in requirements. The list goes on. How to keep all of this under control?
As a manager I’ve learned a few best practices; some are technical controls and some require human intuition. First, technically, use unit tests to decompose a complicated system down into individual unit functions with expected inputs and outputs. When a process breaks the root cause can be identified down to the specific function and controls (error handling) can be introduced. In systems design, reducing the number of components to the minimum while still meeting performance, cost, and other business requirements is another good practice.
On the human side, it’s all about developing context for the information we maintain. Knowing right from wrong when it comes to bits requires “data intuition” and that’s a sense good data engineers develop over time working within their domain. I take more traditional systems view of information in that I focus on the bottom up - how was the information generated in the first place? And what was its journey to my terminal?
This last part about intuition, it really comes down to approaching problem solving with a wide open mind, and having curiosity for life and learning. The bigger mind we develop, the more variety we can manage. In the words of W. Ross Ashby: “only variety can destroy variety.”