The IT Pendulum
Are IT trends a pendulum, swinging back and forth forever?
For a long time, I’ve had a theory, but it’s only recently that I’ve started to explore it further. It goes something like this:
How we approach IT delivery swings back and forth, from one extreme to the opposite, over time, much like a pendulum
I’ve an inkling that it applies to a lot of IT delivery mega-trends. For example, the swing from distributed siloes of data, to centralisation, to distributed with a virtual layer.
As part of my increasing interest in developing this theory, I did the obvious thing. I googled it. I found no reference to other opinion on this in technology at all. Interesting. What I did find however, took me on an online exploration through subjects including psychiatry, Chinese religion, politics, philosophy, ancient Greek and Roman.
I’m now fairly confident that this effect is a real thing, and that it has roots in some of our natural human behaviours. I also think that by understanding it, we can attempt to take learnings from it, and improve the way in which we adjust our approach to delivery.
Foundations of the theory
This thinking has formed over some time, through a number of things I’ve seen posted in articles, on LinkedIn, seen in action in teams I’ve worked with or organisations I’ve worked for. As technologists (and humans) we love to disagree and surface opposing opinion. To see this in action, simply attend any architecture meeting!
I have regularly seen actions or opinions that support switching from one approach to another, at the opposite end of the spectrum. The guilty include (but are not limited to) engineering “trend-setters” like Netflix or Facebook.
It manifests at all scales:, from organisational unit to tiny engineering team. Yo-yo from in-house IT to outsourced and back to in-house. Mono-repo to multi to mono. Monolith to microservice and back again. I don’t remain innocent in this either, actions noted include my own.
None of these examples will surprise anyone. They’ve been commonplace for some time; however, this behaviour happens so regularly that I assigned a name to it in my head, and even started referencing it in conversation. And I found myself referencing it quite regularly.
Some research highlights
As I started to do some research into this area, I came up blank on references related to IT. However, I did find plenty of references to something called the Pendulum Effect, and plenty of discussion about dualism which led me to a principle called Enantiodromia. A little reading on these subjects and there seemed to be an interesting relationship to the behaviour I’ve described.
The Pendulum Effect is a law in Physics. Discovered by Galileo, it describes the regular swinging motion of a pendulum caused by the action of gravity and acquired momentum. More interestingly, the pendulum effect is present in cultural trends too; politics, social conscience, fashion, etc. There is plenty of discussion about this out there, and it’s clear to see in the recent political trends. So, it’s not unreasonable for it be present in IT trends.
I also found enantiodromia to be an interesting analogue. A principle of Carl Jung, the founder of analytical psychology, it is defined as:
“the emergence of the unconscious opposite in the course of time. This characteristic phenomenon practically always occurs when an extreme, one-sided tendency dominates conscious life; in time an equally powerful counterposition is built up which first inhibits the conscious performance and subsequently breaks through the conscious control”
- Psychological Types, Carl Jung, 1921
Obviously, this was not created with IT delivery in mind. However, I’m pretty sure that ‘extreme, one-sided tendency that dominates conscious life’ could be applied to, for example, agile or microservice evangelists!
A slightly less obvious example
The examples I’ve provided so far are fairly obvious. They are some of the most visible IT delivery mega-trends. But there are plenty less obvious ones. A good chunk of my background is in data and analytics, and the mega-trends we see here exhibit this behaviour too. Taking a specific example:
Decentralised database silos, centralising all the data, back to decentralising again
Many organisations have numerous databases. Staff access these separately to run operational systems, reporting and analysis. It becomes necessary to use something to bridge the data where it’s not in the same database. The obvious first choice is Excel spreadsheets. Over time this causes a headache of poorly managed excel files with disparate, diverging copies of data.
So, it’s decided that they will implement a centralised option. A warehouse, a lake, whatever. Going to one place for your data is attractive. Done correctly it will solves many of the frustrations. Data is moved and transformed out of existing databases, standardised and modelled. Organisation structure, roles and responsibilities change to mirror this. Reports and analysis are generated from this new single source. Unfortunately, after a time it’s found to be really difficult, significant effort, complex technology, hard to operationalise and the data is often not there when those who should act on it need it.
A lot of staff find ways around use of the new system and still rely on going to the source of data. Analytics teams find ways to effectively use these source systems so as not to wait for the data to be centralised. The warehouse becomes another database and you are back to decentralised silos. To bridge the gap, this time perhaps we use data virtualisation, federated query capability or even implement data mesh.
This particular example is not uncommon. I’ve seen it multiple times and industry research tends to support it too. Of course, I’m not suggesting that in every case the pendulum swings back like this, but in plenty of situations it does, or it will eventually do so. I suspect that sometimes it may take many years for a swing to materialise.
I’m also not suggesting that, because of this effect, you shouldn’t seek out alternative approaches. Rather you should be aware of this effect and understand how you might learn from it.
A good question at this point, and one I asked myself. What does this actually mean and how might I benefit from being aware of it?
Honestly, I don’t have the answer to this yet, though I do have observations that I’ll share in a minute. It’s only recently that I’ve started to consider what this means, rather than just its existence. My gut tells me that you can’t stop this effect, nor really influence it. Seeing as it acts upon mega-trends at an industry level it is always going to be very hard to have an impact upon unless you are Spotify or Netflix, for example. Though maybe if those organisations understand this effect, or their influence on it, then they could help out the many others that follow their movements?
Reacting to frustration
The first of my observations is related to how we, humans, react to frustration. In general, when I get to a certain level of frustration with something, I want to get away from it. And if I can’t, then this frustration builds and builds until it’s occupying too much of my mind. At some point I’m going to be looking for significantly different options. Ones that will avoid this specific frustration, and free up my mind to concentrate on progress.
The easiest way out is perhaps to choose something totally opposing. Something that is very unlikely to repeat those particular frustrations. I’ve weighted them heavily in the decision-making process because of how pressing they are upon me. Of course, the new option will also have its own benefits, some of which could be very appealing. “All these pro’s none of my frustrations — get in my belly!”. Call me emotional, but this is clearly the effect of emotions on decision-making.
It feels a little akin to enantiodromia…
“in time an equally powerful counterposition is built up which first inhibits the conscious performance and subsequently breaks through the conscious control”
It also feels like frustration is equivalent to the action of gravity on a pendulum. Much as gravity acts to slow and reverse the momentum of a pendulum, over time frustration acts in greater amounts, and at reaching a critical point, it reverses momentum.
The community effect
I’ll admit, inability to control my emotional response is unlikely to be the cause of a mega-trend pendulum action in itself. However, I suspect that a combination of this behaviour and the in-your-face availability of “we did it this way and it rocked our world” or “X vs Y: why X is so much better than Y” articles may.
The whole thing acts a bit like an information cascade, with an accelerating number of engineering teams jumping on the bandwagon of any mega-trend. When enough are frustrated with the path they have chosen to walk, they seek out the opposite end of the spectrum and the pendulum swings.
Throughout this process, we document our findings, why X was better than Y. We oppose each other’s opinion in public forums and try to convince people that they are missing out. Much of this information is publicly available for others to find, providing evidence for their desired change in direction. It seems that our natural behaviours mirror the back and forth nature of these trends, setting the path for them.
What might you take away from this?
Perhaps the most useful thing I take away from this is on the macro-scale. Mega-trends are an important part of my job, but they are a spectrum, from one end to the other. This has a bearing on the advice I might give at any time, the predictions I might make, and how we might target services to the clients and markets we work in.
By way of example, take low code platforms, clearly a trend with significant building momentum. Some markets who have been early adopters of low code platforms are making a little noise about frustrations. While I believe that low code is here to stay, I also believe there will be numerous “swings” (we’ve actually already had one; 4th Generation Languages anyone?). Organisations will eventually wish to redevelop some systems previously delivered through low-code platforms. Knowing this, we support or even target such redevelopment opportunities. Alternatively, we can try to give the pendulum an extra push, meaning a longer lead-time before the action of frustration becomes critical again.
If you’ve made it here, hopefully you’ll have built up some of your own views on this subject. Please share as I’d be keen to hear them, as well as any examples you may have of this effect, or even the absence of it.