Using Narrative to Evaluate Knowledge Programs/Activities

A few posts ago I wrote briefly about Measuring the Impact of Knowledge Management and a few things I'd heard at the APQC conference in May. I had a short but provocative conversation with Kirby Wright about this challenge, and after speaking quite succinctly about the futility of using traditional measures and biased surveys to measure KM impact in complex environments, he suggested using Sensemaker and the various methods from Cognitive Edge / Dave Snowden's work for narrative elicitation, capture and analysis to uncover the real impact of KM work.

A very interesting idea. After all, do we not look very carefully at comments fields in traditional surveys for richer information. Do we not wish more people would fill out the comments field more? Are we careful not to act on one comment alone, but look for trends across multiple comments - and sometimes have difficulty doing that?

I wonder how one would go about convincing decision makers to embrace the foundation principles and try a pilot?

Measuring the Impact of Knowledge Management

Prior to the launch of this year’s AQPC KM conference, I attended the pre-conference session titled Measuring the Impact of Knowledge Management, led by Cindy Hubert. Quite a “juicy” session, but if anyone was looking for a magic bullet, they were disappointed. As in typical organizational performance management, which the session aligned well with, measuring KM impact is equally challenging and highly contextual to the business objectives and outcomes of the organization. As well, the conversations held during the process of designing and implementing measurement are as valuable as the final measurement and associated learning itself.

APQC has done a good job in exploring performance management in the context of KM implementation and through Cindy’s presentation provided:

• Tips – such as “tie new KM measures to already accepted process measures and metrics,” align measures with stakeholder needs, “start thinking about measurement from day 1, KM performance management is a “thinking problem” not a “data problem, select few vital measures, align expectations with realistic outcomes, don’t establish metrics without a measurement process
• Thinking models and frameworks – APQC’s Measurement framework, which leads to the positioning of KM activities in business process to impact business outputs and outcomes, the APQC KM Maturity Model, and most interestingly, a Value Path Analysis thinking model.
• Tools – measurement alignment tool, examples of measures for KM programs, some case studies, and APQC’s Measurement Information Worksheet.

As well, Cindy had a few interesting observations from APQC’s work and research such as:

• Best practice organizations do not include content management costs in measurement of KM effectiveness – a heavy up front investment takes a long time to prove value
• Top five objectives for KM Programs across best practice partners are productivity, quality of products and services, increase profitability and cash flow, avoid redundancy and reinvention, and improve customer service.

The one thing I wish we would have had more time to discuss is the real tough part – attribution, and establishing causal links between KM activity measures and proxy measures to business outcomes when there are many other initiatives and activities underway that feed into those outcomes. And one final observation – through much of the language Cindy was using, I got the impression that “knowledge is an object” was an underlying assumption for much of what she presented. Unfortunately there was not enough time to explore this a bit to see if I was mistaken, and to look at some specific cases where the impact of tacit knowledge was being measured, which to me, would be as challenging as measuring the impact of learning and development.