A Discussion Paper

Written by Viv Read | November 2020

‘If a measure becomes a target, then it ceases to be a measure.’
Charles Goodhart. “Goodhart’s Law”

‘I believe a practice has developed where the drive to meet numerical targets has overshadowed the interest of children despite evidence that they may be at risk.’
John Taylor, Acting Victorian Ombudsman 2011

Performance, measurement and accountability are high priorities on any organisations agenda. The unintended consequences of using targets and KPI’s as measures in complex systems are becoming well recognised.

For example, the 2010 Productivity Commission report into the Not For Profit Sector demonstrates that focus on the use of statistical information and consistent outputs as a way of measuring the impacts and activity of various community services organisations as inadequate.

Likewise the 2011 report of the Australian National Audit Office (ANOA) identifies significant flaws in the delivery and evaluation of government services over the last two decades. However an approach that will address the flaws identified has not yet emerged.

What would a new approach look like?

Quantitative and qualitative measures have been regarded as separate processes using very different approaches. Increasingly however, organisations providing services need more than the numbers to demonstrate the value they provide. Numbers obtained this way do not provide answers to the question “Why”?

Based on our own work, and conversations with clients, the desirable characteristics of an effective evaluation framework that we identified were:

• It would be able need to address the whole system, and identify relationships between the various key elements.
• It would assess the current state and provide insights that gave options for improvement.
• It could be readily customised to meet the specific contexts of different organisations, while having sufficiently commonality to enable comparisons.
• It must be cost effective, dynamic and provide feedback quickly.
• It should provide both quantitative and qualitative measures.
• The insights would be the basis for action , improvement and feedback for both service delivery and policy.
• It is scalable – appropriate for small and large systems?

What do we use?

Most human systems are complex; and simple linear relationships between various elements do not exist. The ‘One size fits all’ approach to various policies, systems and approaches has been found to be wanting. Likewise, imitating what is seen to work in another country, another organisation of even another part of the same organisation rarely works – context is critical.

The many management tools and processes that are available to address organisational issues are based on the assumption that there is a definable relationship of cause and effect between identified elements that can inform strategy and interventions. In current complex and turbulent environments, it is increasingly obvious that these approaches do not serve us well when dealing with complex interactions. Traditional evaluation methods, that rely on hypotheses about cause and effect relationships remain appropriate for some parts of a system – but not all.

At the heart of the Complexability approach is the Cynefin framework; an open systems model that enables the identification of the different elements of a system that need to be monitored. Understanding the distinction between ordered and unordered elements of a system leads to the application for different methods for diagnosis and intervention depending on the nature of the issue.

The Ordered Domains

Ordered systems are based on the belief that it is just a matter of time and resources before the relationships between cause and effect could be discovered in any situation. The relationship may be obvious (Simple) or require research and analysis (Complicated), but it is able to be identified and mapped. These include Force Field Analysis, SWOT for strategy. Or, like LEAN, 6 Sigma and Business Process Reengineering, they map a system’s inputs, throughputs and outputs as a way of identifying system improvement.

Measuring, monitoring and evaluating issues that are in the ordered domains relate to outputs, standards and targets.

There are a wide range of existing tools and methods designed to determine what numerical information to capture, and how to report it. The quantitative data is usually subject to analysis and interpretation by experts. These approaches are founded on assumptions of order, and cause and effect.

The Unordered Domains

The primary feature of unorder is that we do not know the nature of relationships other than in retrospect. There are far too many elements and possibilities to develop and test hypotheses. Therefore the approach is to make sufficient sense of the current state where multiple interventions ( or probes) can be implemented, monitored and quite quickly either stopped or enhanced – depending on what happens. Unordered domains required tools and methods that measure and monitor impact and outcomes.

Qualitative research has primarily relied on expert-led discussions and interviews, such as focus groups. It is significantly more expensive than collecting quantitative data, and the results can be susceptible to researcher bias.

Narrative research is now used in a variety of settings, given that narrative allows complex information to be shared. However, most of the processes for analysis still depend on the researcher’s hypotheses, conceptual frameworks and beliefs.

Sensemaker™

Is software that has been developed by Cognitive Edge specific ally for issues that are in the complex domain. It can capture large amounts of narratives quickly and efficiently, and provide participants with the opportunity to attribute their own meaning to their experiences, rather than experts attributing that meaning.

The software analytics provide quantitative measures of qualitative date, ( impacts and outcomes) and insights that enable action and intervention should this be required.

Contact Viv Read:

Complexability Pty Ltd
ABN 13143 164 078
www.complexability.com.au
5 Leslie Court, King Scrub QLD 4521
(61) 414294339, (617) 34252267
Viv.Read@complexability.com.au