The shadowy hierarchy
I remain curious about how I can make better or wiser decisions. I am sharing this as part of my journey as I unpack my own boundaries and models that prevent me from making better decisions.
Context
I have personally, and will always, dislike and distrust “traditional” hierarchy, probably because I perceived that the “power” wielded on me would never be available to me. I was always on the outside; it is the joy of neuro-diversity that you become aware at an early age that to fit in the system and structure, you have to align to it, which for me, had no natural alignment. You either fight to fit in, fight the system or create your own. For many fitting in is natural, for me it never happened, and I stupidly opted for creating my own. I rebelled against the system and structures as I could only see hierarchy as a method of control to something I did not align to - telling me to do things that made no sense. Write with your right hand as a lefty.
I am not alone; from Machiavelli to Second lieutenant Boris Drubetskoy in Tolstoy's War and Peace, many have realised that there are two structures of power. There is an obvious hierarchy of control, reporting and subordination, and there is a second unsaid and unwritten one of lobby and access. The obvious one is easy to rebel. The lobby and access inner ring is where true power rests; it is the one to try and join. However, it is invitation-only, and many will sacrifice more than they realise in terms of ethics, morals and beliefs to be at the table and remain part of it. When thinking about data, bias and decision making, the incentive to join and incentives when in the club is a critically important driver. This unsaid shadowy hierarchy creates outcomes we did not plan for as some will jeopardise judgment and trust to be rewarded. I would not align to this one either, and it was self-evident that you can only become a partner in a big firm when you have a big mortgage, depending on a big salary. The significant debt and dependence on salary provide leverage. The humour in “Yes Minister”, the 1980’s BBC political satire sitcom, exposed much of this.
Why does this matter? We increasingly talk to the equality and diversity (race, gender and neuro) agendas at the board and in the senior leadership team, but it is increasingly evident that there is an inner circle, which means we are not making smarter or better decisions. The targets for transparency and equality are just that, a target. (says the white over 50 male!)
I appreciate that it is difficult to separate the two dependent and co-joined aspects of the hierarchy. One aspect represents power and control, which is realised in terms of budget, size, scope, authority, and political dominance. The second is how and where important decisions sit, but those who lobby may be outside of the insights that data now provides for complex decision making.
The separate but parallel structures have worked for a long time as the consequences of long term decisions and intergenerational outcomes were not rewarded or measured. ESG, climate and the brutality of our assault on our environment, coupled with data, means this is all changing, but those who gave everything to join the inner circle have an incentive to maintain the status quo for as long as possible.
What ghosts are in the system that means decisions we make, with the best intentions, do not create the outcomes we desire
Take a moment to reflect. Are there separate hierarchies in your experience, or are they the same thing? It matters today more than ever before as we now have to make long term decisions against the backdrop of our systems of short term incentives and rewards. The inner ring becomes blind to a structure that always gives the feel of power as their short term decisions are highly rewarded through immediate incentives, which means being in the club is valuable for the few.
This viewpoint is looking at these two hierarchies (formal and informal) but positioning where either can be in the shadow of the other. Why am I writing this because we appear to find short term incentive-driven decisions easy but struggle to make long term judgment? I do not doubt the integrity of leaders who want to make better long term decisions using data and be better ancestors but get frustrated that it does not work. There are likely ghosts from the old decision making and lobby hierarchy in our current heuristics that create outcomes that were not planned.
Note: The term “Shadow Hierarchy” is used in management textbooks to describe the difference between a formal published and public structure and the one that actually has power and decision making. Googling “The Shadow of Hierarchy” will lead you to the 2008 paper from Adrianne Heuritier and Dirk Lehmkuhl, which was part of an EU funded project looking at “new models of governance.”
Our past is not easy to face up to.
There appears to be a much longer story that starts when decisions, power and control were all united. Over the past 5000 years, we have become increasingly short term focused and separated the decision-making process for the long term from power and control based on the science of management incentive and short term goals. There is no doubt that the structure of economics, along with other biases, come to play. However, we have created data technology on top of complex relational dependencies that means signals that should become noise through the layers of analysis, in fact, become significant distracting “interruptions” at the board. Whilst in the old system, noise and signals were hidden by the informal lobby that prevented more thoughtful long term decision making based on data, today they (signals and noise) create paralysis at all levels in structures as everyone knows everything.
Should the head of a religious movement be leading planning for the next 100 years or focussing on if the budget is spent according to the plan this quarter? Should our political leaders face the daily news headlines and respond or ensure we are equipped to face a global pandemic? Should the head of state be allowed to focus on sustainability, climate and global concerns or defending their grandchildren’s choices? What makes noise and news dominate interruptions. In a joined power and decision hierarchy structure a long time ago, a few individuals could make those decisions and choices, and lobby worked. Today our hierarchies, analogous to decision making, have become paralysed, confused and ineffective as the volume of data and signals mixed with noise has risen to a level where our ability to know what is the right thing to do is broken. Currently, we have not transitioned to something that works. Indeed lobby has also failed as it has become increasingly linked to short term reward and incentives.
The figure below captures this concept on a two-axis chart of scale and impact. The bottom right being an idea, it has no scale and little impact. The top right is big government, global businesses, global NGO’s and charities, global religion and large scale movements. On the journey from the idea to scale, we either transition from our ability to make long term decisions to focus on a quarter by quarter reporting justifying the delta between actual and plan *or* hold onto inter-generational consequences. Risk, funding, capital and markets have a significant impact on the loss of that long term thinking, as the rewards for the players become aligned to short term incentives. Whilst the long term hierarchies become corrupted, the shadow hierarchy of lobby gives way to a different incentive and power game. Impact and scale create the same problems irrespective of the organisation; short-termism can be recognised and rewarded.
Joseph, as in the Bible story and multicoloured dream coat fame aka the Tim Rice and Andrew Lloyd Webber musical. It is one of the earlier written examples of man’s capability for longer-term planning. It was a 14-year cycle, so not long term; however, 7 years of abundance followed by 7 of famine. Grow, harvest, store and distribute later. 4,000 years on and a 14-year strategic planning cycle looks massive but still short compared to the famed 100 years plus China and Japanese plans. I may have rose-tinted glasses that we were once better at long-range forecasting, but this is a good piece from three world-leading experts “is humanity, in fact, unable to successfully plan for the long-term future?” In the context of our currently limiting systems - yes. We have to break the system. I smell revolution.
Complexity of relationships
No doubt, a small part of the issue with our inability to long-range plan is the management of the complexity in relationships. The web of collaborative relationships that we need to consider only ever gets more strained, detailed, involved, dependant, and unbalanced, as each party align to their incentives and rewards. Critically, the unbalanced nature of the relationship means it is increasingly difficult to predict outcomes and reactions. (search: emergent complex systems). Our commercial framing is explored in the figure below.
The shareholder has a dependency on the board to make a decision that supports their funding of the capital. In contrast, the directors are accountable and responsible for their decisions, including unlimited liability. Everyone now has a voice that can affect choices and actions. The Director/ regulator/ ecosystem axis is dominated by who has the accountability and responsibilities, which are different for each stakeholder in an ecosystem and are often driving in different directions because of Balanced ScoreCards (BSC), KPI’s, incentives and being in the club. OKR (objective and key results) are no better, and the difference between built vs operate is a false one for the long term. Building the wrong thing can be rewarded very well in OKR land.
Continuing around the model, there is a remarkably fluid relationship between the executive teams’ and the board, where the board depends on the exec team. Still, the exec team is accountable to the board. (The level of fluid varies by nation and company law.) The relationships between and with the customer are particularly misunderstood, but ultimately the law says the directors are held accountable. Each of the roles in the chain of relationships from shareholders to the executive team required individuals capable of dealing with complex judgement. However, this assumes that the first part of this viewpoint is null. Let’s expand.
We face three connected issues but not mutually exclusive because they are a team or part of an ecosystem.
Some individuals in positions of influence and power are part of the club or want to be in the club and therefore have incentives to be in the club and stay in the club. Their judgement and actions are aligned with being in the club incentives.
Some individuals have reached influence and power but cannot grasp the new mental models and skills for complex judgment. Their judgement and actions are aligned with their experience.
Some individuals can understand the complexity and seek to explain and justify decisions and actions.
Taking this thinking and plotting this on skills/ ability vs decision environment.
The diagram above shows a decreasing number of skilled individuals who can cope with complexity. This is because the training system is about effectiveness and efficiency and not about determining if you are doing the right thing (efficacy). Many arrive at senior roles and find they have to shift their mindsets; some do, and some don’t, but they both now have decision capacity.
For example, we know that systems that create inequality, insecurity, and unsustainable practices are not easily transformed. Think of our government and economy. We have a system where 95% of the world live meal by meal, day by day, week by week or month by month. An additional 4.99% can survive for 6 to 8 months on the available cash and funds. Less than 0.01% (80 Million) of the world’s population can plan for more than a year. When you have those few in power and lobby, will they ever need to vote for change? Worth following and reading Umair Haque; I love this essay “How the Economy is Designed to Keep You Poor and Powerless.”
What do “ghosts in the system” look like?
Imagine you are at the fairground, and there is one of the stalls where you get to throw something to win a prize. In the UK, we have the Coconut Shy. You pay to get three balls or bags, stand behind the line and throw them one at a time at your coconut of choice. Knock a coconut off, and you win it. A simple game of skill (apparently). However, when there is a ghost in the system, it is not so simple. You line up your ball on coconut number one (it is the biggest one) and throw it with all your skill. As the ball approaches, the coconut moves and your ball sails past. You line up ball 2, aiming again at the largest coconut (the biggest prize); this time, you miss, but coconut number 4 wobbles. Your last chance on this budget. Lining up coconut number one for the last time, you hit it, but it does not fall off, but coconut number 6 does, the smallest one. The ghosts win. Your decisions and outcome was coconut number 1; you got number 6. It was not your lack of skill; coconut number 1 is glued on. Different motivations and rewards.
The ghosts make signals and noise.
This viewpoint started from hierarchy, and I want to return to thinking about the two hierarchies, one of decision and one of power and unpack the issues that the abundance of data has created for us. The diagram below sits with the idea that there is a natural order of decision making and power. The movement from the bottom to the peak is a move in the decision time horizon. Overlaid on this model is VUCA (volatile, uncertain, complex and ambiguous) situations. VUCA came from preparing/ training the solder who would have to face a situation where they had to decide.
The hierarchy is worth a paragraph to unpack, as the context of VUCA is situational. Front line workers are trained to be reactive. They know how to act based not on scenario but based on the situation (police, army, emergency, fire, medical, call centre, customer-facing representatives). How they react is one of the rules, heuristics and repeated numerous times. As the NIKE brand says - “just do it”, which is what you can do when you have trained to do the same thing for 10,000 hours - you don’t have to think it is a reaction. Above this line is the management who ensure the environment the workers have has the lowest possible risk. Above them is management, who think about “how do we adapt to new situations and threats?” Above them is the strategy layer, which considers “what skills do we need for the next five years, where are the gaps, and how do we access the skills.” For most companies, this takes us to the CEO. However, in public operations, there are two more layers. The transformational one is thinking about the infrastructure for the next twenty years and, finally, the policymakers. The policy leadership should be thinking 50 years hence and considering what policy we will need to form and how. Even at this simple layering, we can see that global leaders from presidents, prime ministers, and heads of state struggle to plan for 5 years yet are tasked with 50. We are not 10 x better; we have created a system 10 times worse.
What we should be witnessing is that one layers signal is another layers noise. Each layer takes all the signal from below that becomes their noise but detects new signals that they work to. An upward flow - not downward instruction. As the figure below shows, each layer has to apply different skills to find the signals needed to do their role. Interestingly, the lowest layers have the most operations discretion as they are exposed to the highest operational risks. Most diversity is welcome, but not from those who cannot do the same thing every time. Innovation may not be your friend in the depths. The strategy layer has peak decision discretion. Innovation is critical, and so is all diversity is critical. At the pinnacle is policy discretion, where is the least personal risk and whilst diversity is essential but so is adherence to a north star and single vision - so less diversity might help. Diversity is about situational improvement in decision making for better outcomes. Ouch.
However, this is not what we are witnessing right now - the above is theoretical rubbish. What we are feeling is summed up in the diagram below. There is are signals from the bottom, creating signals at the top, every layer adding more signals and noise. VUCA has gone from situational layers to the entire organisation, where everyone is reacting to everything. The ghosts of old processes and previous decisions are no longer limited to a layer, but everyone owns everything and has to react and understand. To repeat a previous line. Today our hierarchies, analogous to decision making, have become paralysed, confused and ineffective as the volume of data and signals mixed with noise has risen to a level where our ability to know what is the right thing to do is broken. Currently, we have not transitioned to something that works.
The observation took time, but the question for us all is, do we believe it, and if so, what can we do? Do you believe it should be reflected on within the context of being in the club and how much you have already compromised on? Our Millenials are not in the club and will not compromise, ask them.
Our stability has vanished, and our tools have broken.
Even to a casual observer, we live in turbulent times, which is seen through an increase in VUCA. The difficultly facing boards who carry responsibility for their decisions cannot be overstated. We have to deal with the ghosts of the past, the voice of the present and the spirits of the future. The diagram below brings out some of the tensions and conflicts being faced as we struggle to determine what we are optimising for. The two-axis are communication (said and unsaid) and status (know and unknown). The unsaid is that which is not written or spoken and assumed, often to avoid more conflict.
We have to deal with the ghosts of the past, the voice of the present and the spirits of the future.
Top right (said and known) This is the day to day operational aspects of the board and senior leadership team. There are two camps (supporter and action owner) at the board with 10 people involved in decision making. The focus is on management, KPI’s BSC and reporting. There are known status with data, and everyone is able and capable of engagement in the topics. We are married to this quadrant as it is easy and pays the remuneration, and we are incentivised to focus on it. This quadrant has traditionally depended on stability, but with VUCA disrupting that and the volume to signal and noise - we have to spend all our time here as that is all we have time for.
The bottom right is said and unknown. In this situation, everyone has a view about the unknown resulting in 10 camps (personal opinions and experience) with the 10 people. However, communication tends to be frank and honest. We go to this quadrant every now and again but quickly withdraw to safer grounds.
The known and unsaid, top left. The unsaid here are the assumptions that we all make about everyone else in the room and their viewpoints. This time there is are 100 camps as we all assume about what everyone else thinks about everyone as there is no communication. The principal reasons not to communicate is conflict, dominance and leadership style. We are trapped by the debt we have (mortgage, credit, school fees, lifestyle, divorce) and need the salary; therefore, there is a degree of control. We try to avoid this quadrant but every now and then, we end up there because of other stress, pressures and the need for a diversion or win a political game.
The bottom left, the unknown and unsaid. The ecosystem has a voice, the partners of the directors have a voice, everyone has a voice, and many, due to previous flights and allegiances, are in several camps at the same time. The crux is that we are human and bring our differences, but it is very messy because it is unknown and unsaid. But this quadrant represents the volatility, uncertainty, complexity and ambiguity we are currently facing, and we don’t have the tools to deal with it. We have a preference based on skills, experience and incentives to focus on the top right.
Our processes and methods enable only certain decisions
One critical aspect of being in a leadership capacity is to question and determine how our processes and methods guide and frame certain decisions, which means we have to unpack legacy and find the ghosts in the system.
Legacy within this framing is threefold. Decisions. Decisions. Decisions. These are:
Previous incentives and power games created decisions that created processes, methods and rules; they are now ghosts in the systems. These decisions were taken so long ago that no one knows why, how or when it was decided. It is the way we do it; it is our IP, our brand.
Decisions that created “information and technology debt” included embedded and baked-in systems, hidden and no-longer supported code, and automation based on tools and data biased when created.
Decisions that created noise in the hierarchy to lose or filter signals that someone did not want to hear. It was the creation of layers, reports, practices, structural regulation and unchallenged assumptions.
Unpacking legacy questions will take time. It is worth asking questions about legacy when you are new to a company and then verifying them over time as we become blind to the tools that mould us.
I am focused on this because I want to determine how I can make smarter/ better decisions with data. For that, I need a data decision framework. Therefore I tend to ask what one thing do we as a leadership team want from our data? The response varies but include:
Evidence-based, actionable insights
What should we automate?
How do we know we are doing the right thing?
Where are there efficiencies to be gained?
What do customers really want?
How to manipulate customers to increase margin and revenues?
Where are risks that we cannot see?
What is being hidden that we cannot see?
If you look at this list in the context of what tools and decisions already frame the response, are these what we are looking for data to answer or are we looking to data to affirm/ justify what we have already decided. A different response that no one will say “to justify what we are already doing!” This fits into the know/ unsaid in the previous matrix. The top left, the one we avoid opening up.
Data has bias because of previous decisions. Or we can write, “the ghost of previous decisions will guide our current decisions”. Importantly, our data, which we trust, is only the representation of the past, which means our tools fail.
Therefore, as a leadership team, we have to find non-data tools to check what decisions from the past are biasing the current data, processes and tools. We cannot usefully answer the question we have set ourselves “What is the one thing that we, as a team and organisation what our data to drive, deliver or provide? Without understanding the situation.
The CTO knows that they have to build a new platform when the bug list, new feature development and maintenance costs are bigger and will take more time than developing a new platform — this is the technology debt question. The CIO or newly created #CDO role has to understand what is your information debt. The CTO will struggle as there is no clear path from policy to code. Similarly, the CIO/CDO struggles with no clear path from policy (what one thing) to better data for the decisions we require. The data leadership team inherit and are now accountable for previous ghosts and decisions, which constrain what is now possible as the biased tool has created what we have. The costs of collecting, labelling, holding, sorting and creating training data continually increase, creating a more significant gap and misalignment in values and expectations from data.
“We become what we behold. We shape our tools, and then our tools shape us” is often mistakenly attributed to Marshall McLuhan and called McLuhan Law. The quote was actually written by Father John Culkin, SJ, a Professor of Communication at Fordham University in New York and friend of McLuhan. Such is the problem with data.
As we have separate decisions, power and now data, perhaps we should reflect on these questions as a leadership team.
What do we want to become, and what tools and data will help us?
What tools do we use, and what will they enable us to become?
What is the minimum viable data set required to give the best value?
Do our tools and data trap us?
Is the work of data to “detect and measure” or to “enables change and transform?”