(The following posting is taken from an email response to a set of questions, shown below as the first “comment.”)
Yes, the upper right quadrant "How well did we do it" category can contain measures about customers. Consider the measures % of classes delivered on time and % of customers who say classes were delivered on time. They are almost identical measures. Customer data can often tell us how well services were delivered (How well did we do it?). The fundamental question to ask when thinking about whether a customer measure is upper or lower right quadrant is this: Could we do well on this measure and customers might still not be better off? For example, the % of customers who think the class is delivered on time is 95%. But the class itself is terrible and no one learns anything.
This same principle is illustrated by how we categorize the completion rate for a service or training course. The mere completion of a service or training course usually tells us little or nothing about whether people got anything positive out of it. So % of clients completing service or training is usually an upper right quadrant measure. But there are exceptions to this rule. If the training has an established reputation for giving its students new skills, then completion can be a lower right quadrant measure. Life saving courses, where earning a certificate really means something, is a good example.
This principle can be carried further to the general question of how to classify different measures of customer satisfaction. Customer satisfaction measures fall into two broad categories: Did we treat you well? and Did we help you with your problems? Customers can say they were treated well, that they liked the counselor, thought the building was comfortable, accessible etc. but still not be helped with their problems. So "Did we treat you well" is an upper right measure and "Did we help you with your problems" is lower right.
The deepest underlying principle here is that measures are classified not on the basis of some intrinsic characteristic of the measure, but rather on the basis of what question the measure helps to answer. If measures help answer How well did we do it? they go in the upper right quadrant. If measures help answer Is anyone better off? they in the lower right. If they could conceivably answer both questions, then pick the stronger answer, or as a last resort put it in both places. This same principle applies to the difference between population indicators and performance measures. Sometimes a given measure can play both roles, so that sometimes it serves as a performance measure, and sometimes as an indicator.
I would encourage you to read Chapter 4 (especially pages 65 to 79 and 99) of "Trying Hard Is Not Good Enough" where these matters are discussed. There can sometimes be legitimate differences of opinion about how to categorize a measure. When this happens, just put is somewhere and move on. Because all the measures in both upper right and lower right quadrants will be considered in Step 4 of the 5 step process for identifying the most important performance measures (See Appendix G).
Finally, the reference to "circumstance" is specifically about the client or customer's circumstance. Are they in stable housing?. Do they have a living wage job? It is not about a more general definition of circumstances. For this reason it always goes in the lower right quadrant?
Mark
Thursday, September 10, 2009
Wednesday, September 9, 2009
Creating Results Culture Change
People talk about culture change the way they talk about magic. We know this wonderful transformation is needed and will somehow happen, but we're not sure how.
Let's take a shot at a definition of organizational culture. Organizational culture is the range of accepted norms of for what people in an organization think and how they act. An organization might have a dominant culture and numerous subcultures. Subcultures might vary by program, profession, or organizational role. Organizational culture also includes a story line or mythology about why the norms exist the way they do.
Culture change then is any significant variation in or deviation from organizational culture or subculture. Culture change can be positive or negative. Sustained culture change is any culture change that survives some defined degree of turnover in key positions. It is worth noting that culture change is not a smooth transition. There are holdouts. Some parts of the organization go ahead of others. Some people never get it.
It is arguable that there are three elements needed for culture change to take place. This is a kind of hypothesis that might be tested with regard to creating a results culture.
1. Leadership: No culture change happens without leadership. It may be possible to talk about the evolution of organizational culture without leadership direction. But any form of deliberate change requires leadership.
2. Vision and small steps: Culture change requires the odd combination of vision and small steps. First, a vision of what the organization culture should be, that is a picture of destination. Deliberate or directed change also requires a well defined series of small steps that can begin the process and move it forward. Massive overhauls are often not practical and usually don't work. So culture change requires a pathway of small steps that lead to bigger change. One guide to these small steps is the RBA Self Assessment Questionnaire new version (available on resultsaccountability.com). Start with one supervisor identifying and using performance measures. Then two supervisors, and so forth. After the accumulation of enough small steps, it is possible to reach a tipping point where change happens much more rapidly. It is worth noting here that among the seemingly small steps to take is the change in forms and formats for strategic plans, budgets, RFP's and contracts. Forms are the skeleton of any organizations culture. They can live for decades. And if you can change them the change can last for decades.
3. Finally, feedback. Any change process requires a feedback loop to see if you are making progress. Two types of feedback are needed. Information is needed on the extent of implementation. One can use the RBA Self Assessment Questionnaire to calculate a score on the extent of implementation. The second type of feedback is on whether curves are turning on key measures.
Let's take a shot at a definition of organizational culture. Organizational culture is the range of accepted norms of for what people in an organization think and how they act. An organization might have a dominant culture and numerous subcultures. Subcultures might vary by program, profession, or organizational role. Organizational culture also includes a story line or mythology about why the norms exist the way they do.
Culture change then is any significant variation in or deviation from organizational culture or subculture. Culture change can be positive or negative. Sustained culture change is any culture change that survives some defined degree of turnover in key positions. It is worth noting that culture change is not a smooth transition. There are holdouts. Some parts of the organization go ahead of others. Some people never get it.
It is arguable that there are three elements needed for culture change to take place. This is a kind of hypothesis that might be tested with regard to creating a results culture.
1. Leadership: No culture change happens without leadership. It may be possible to talk about the evolution of organizational culture without leadership direction. But any form of deliberate change requires leadership.
2. Vision and small steps: Culture change requires the odd combination of vision and small steps. First, a vision of what the organization culture should be, that is a picture of destination. Deliberate or directed change also requires a well defined series of small steps that can begin the process and move it forward. Massive overhauls are often not practical and usually don't work. So culture change requires a pathway of small steps that lead to bigger change. One guide to these small steps is the RBA Self Assessment Questionnaire new version (available on resultsaccountability.com). Start with one supervisor identifying and using performance measures. Then two supervisors, and so forth. After the accumulation of enough small steps, it is possible to reach a tipping point where change happens much more rapidly. It is worth noting here that among the seemingly small steps to take is the change in forms and formats for strategic plans, budgets, RFP's and contracts. Forms are the skeleton of any organizations culture. They can live for decades. And if you can change them the change can last for decades.
3. Finally, feedback. Any change process requires a feedback loop to see if you are making progress. Two types of feedback are needed. Information is needed on the extent of implementation. One can use the RBA Self Assessment Questionnaire to calculate a score on the extent of implementation. The second type of feedback is on whether curves are turning on key measures.
Friday, August 21, 2009
Measuring the success of population level strategies
The general population level progression from results to indicators to baselines, story, partners, what works and strategies, often leads people to ask next, "What performance measures will tell us if our strategies are working?" This, of course, is the wrong question. To tell whether our population level strategies are working, we must look to see if the indicator curves are turning. Indicators measure the extent to which strategies are working. Performance measures tell us if the individual components of our strategies are working.
For example, a partnership might come together to promote community safety as measured by indicators such as the crime rate or the percentage of people who feel safe. After working through the RBA/OBA process, they might settle on an initial three part strategy which includes community policing, improved lighting and a neighborhood watch program. To see if the overall strategy is working, we would look to see if the curves are turning on the selected indicators (i.e. crime rate and percentage who feel safe). As for performance, we would take each component of the strategy in turn and identify performance measures for that component. So for the neighborhood watch program, we might look at the percentage of neighbors signed up, or the crime rate for neighbors in the program compared to neighbors not in the program.
There is one other kind of performance measure relevant here, which may be the source of some of the confusion on this subject. For those managing the overall strategic planning process, there is a need to know the extent to which the strategy (or strategies) have been implemented, and how well they have been implemented. So a performance measure for the partnership managing the strategy might be the percentage of agreed action steps that are on track. Notice how this measure tells us how well the partnership is working, not how well the overall strategy is working. One could easily have a strategy that is implemented beautifully, but has no effect on the indicator baselines.
For example, a partnership might come together to promote community safety as measured by indicators such as the crime rate or the percentage of people who feel safe. After working through the RBA/OBA process, they might settle on an initial three part strategy which includes community policing, improved lighting and a neighborhood watch program. To see if the overall strategy is working, we would look to see if the curves are turning on the selected indicators (i.e. crime rate and percentage who feel safe). As for performance, we would take each component of the strategy in turn and identify performance measures for that component. So for the neighborhood watch program, we might look at the percentage of neighbors signed up, or the crime rate for neighbors in the program compared to neighbors not in the program.
There is one other kind of performance measure relevant here, which may be the source of some of the confusion on this subject. For those managing the overall strategic planning process, there is a need to know the extent to which the strategy (or strategies) have been implemented, and how well they have been implemented. So a performance measure for the partnership managing the strategy might be the percentage of agreed action steps that are on track. Notice how this measure tells us how well the partnership is working, not how well the overall strategy is working. One could easily have a strategy that is implemented beautifully, but has no effect on the indicator baselines.
Saturday, August 15, 2009
Next Generation Contracting - Key Provisions
People have been asking me for more detail on what I mean by "next generation" contacting (or commissioning as it is sometimes called in the UK). Here are some additional thoughts. I welcome your comments and suggestions.
Next generation contracts will have three essential provisions. Provision 1 (Most important performance measures) and 2 (Continuous improvement process) are sometimes used in present generation contracts, but these two with the addition of provision 3 (Performance partnership between funder and contractor) create the "next generation" funder / contractor relationship. Provision 4, having to do with simplifying and standardizing contractor reporting, is a highly desirable but optional component. Clearly there are many other components of a complete contract genome (see Dawkins) that are necessary to have a fully functioning contract, so these are necessarily parts of a larger whole.
1. Most important performance measures (Provision 1) specifies the most important 3 to 5 How well did we do it? (Upper right quadrant) and Is anyone better off? (Lower right quadrant) measures by which the contractor's performance including its effectiveness on behalf of customers can be judged. If these measures are not identified in the contract itself, the contract should specify a process for quickly agreeing on the measures, with possible reference to the RBA 5 step process in Appendix G of the RBA book.
2. Continuous improvement process (Provision 2) specifies that the contractor will use a continuous improvement process (the RBA 7 Questions are naturally recommended) to monitor and improve performance over time. The contract must specify that this process takes place (usually monthly) at the highest level of overall contractor performance (usually agency or major program level) but should also, if possible, specify that the RBA 7 questions will be used (or phased in) to every supervisory relationship.
3. Performance partnership between funder and contractor (Provision 3) specifies that the funder and contractor will meet periodically (most likely quarterly) to jointly assess performance on the 3 to 5 measures, the story behind the baselines for these measures, and what each party can do to improve performance in the coming period. These meetings should be preceded by the submission of a performance report in RBA format (See Wyoming Part II and San Mateo budget formats) from the contractor to the funder. In these meetings, the funder and contractor act as co-equal partners in helping facilitate improvement. This means that the funder will also agree to tasks that can help the contractor improve, such as providing technical assistance, working to break barriers the contractor may be experiencing in getting access to necessary resources or cooperation from other agencies, essentially anything within the funder's means that can materially support the contractor's performance improvement. Nothing in this provision lessens the funder's usual responsibility for oversight of the contractor's performance and use of funds. Nothing in this provision substitutes for the grantee's ongoing improvement process in Provision 2 nor for the contractor's responsibility to pro-actively solve its own problems without the funder's help.
4. Coordination with other funders (Provision 4) - optional - The funder agrees to actively pursue agreement with other funders who also purchase services from the contractor for the purpose of simplifying and standardizing contractor reporting requirements.
These are conceptual descriptions of the contract provisions and must, of course, be translated into actual contract language. Suggested language or examples of such language are welcome.
Next generation contracts will have three essential provisions. Provision 1 (Most important performance measures) and 2 (Continuous improvement process) are sometimes used in present generation contracts, but these two with the addition of provision 3 (Performance partnership between funder and contractor) create the "next generation" funder / contractor relationship. Provision 4, having to do with simplifying and standardizing contractor reporting, is a highly desirable but optional component. Clearly there are many other components of a complete contract genome (see Dawkins) that are necessary to have a fully functioning contract, so these are necessarily parts of a larger whole.
1. Most important performance measures (Provision 1) specifies the most important 3 to 5 How well did we do it? (Upper right quadrant) and Is anyone better off? (Lower right quadrant) measures by which the contractor's performance including its effectiveness on behalf of customers can be judged. If these measures are not identified in the contract itself, the contract should specify a process for quickly agreeing on the measures, with possible reference to the RBA 5 step process in Appendix G of the RBA book.
2. Continuous improvement process (Provision 2) specifies that the contractor will use a continuous improvement process (the RBA 7 Questions are naturally recommended) to monitor and improve performance over time. The contract must specify that this process takes place (usually monthly) at the highest level of overall contractor performance (usually agency or major program level) but should also, if possible, specify that the RBA 7 questions will be used (or phased in) to every supervisory relationship.
3. Performance partnership between funder and contractor (Provision 3) specifies that the funder and contractor will meet periodically (most likely quarterly) to jointly assess performance on the 3 to 5 measures, the story behind the baselines for these measures, and what each party can do to improve performance in the coming period. These meetings should be preceded by the submission of a performance report in RBA format (See Wyoming Part II and San Mateo budget formats) from the contractor to the funder. In these meetings, the funder and contractor act as co-equal partners in helping facilitate improvement. This means that the funder will also agree to tasks that can help the contractor improve, such as providing technical assistance, working to break barriers the contractor may be experiencing in getting access to necessary resources or cooperation from other agencies, essentially anything within the funder's means that can materially support the contractor's performance improvement. Nothing in this provision lessens the funder's usual responsibility for oversight of the contractor's performance and use of funds. Nothing in this provision substitutes for the grantee's ongoing improvement process in Provision 2 nor for the contractor's responsibility to pro-actively solve its own problems without the funder's help.
4. Coordination with other funders (Provision 4) - optional - The funder agrees to actively pursue agreement with other funders who also purchase services from the contractor for the purpose of simplifying and standardizing contractor reporting requirements.
These are conceptual descriptions of the contract provisions and must, of course, be translated into actual contract language. Suggested language or examples of such language are welcome.
In the Beginning
Dear Friends,
The RBA / OBA Blog is devoted to the worldwide community of Results-Based Accountability (RBA) and Outcomes-Based Accountability (OBA) practitioners. It's a place for me to share notes on a daily, weekly or monthly basis that might be of interest. Your comments on these entries are most welcome. And if you prefer more private communication you can contact me at xfpsi@aol.com. If you're new to RBA/OBA you can find out more at http://www.resultsaccountability.com/ or http://www.raguide.org/ or http://www.resultsleadership.org/.
All the best. Mark Friedman 8/15/09
The RBA / OBA Blog is devoted to the worldwide community of Results-Based Accountability (RBA) and Outcomes-Based Accountability (OBA) practitioners. It's a place for me to share notes on a daily, weekly or monthly basis that might be of interest. Your comments on these entries are most welcome. And if you prefer more private communication you can contact me at xfpsi@aol.com. If you're new to RBA/OBA you can find out more at http://www.resultsaccountability.com/ or http://www.raguide.org/ or http://www.resultsleadership.org/.
All the best. Mark Friedman 8/15/09
Subscribe to:
Posts (Atom)