Questions & Answers from Wales Master Class

Mark Friedman RBA Ideas Leave a Comment

Delegates were asked to discuss in groups one question which they would like to ask Mark Friedman. The following questions will be sent to Mark who has agreed to respond:

  1. Is understanding [RBA principles] more important than rigid fidelity to the RBA process?

RBA is specifically designed so that it can be broadly and flexibly applied.  I would always argue for understanding principles over rigid fidelity to RBA or any other model. Having said that, there are well thought out reasons for the conventions offered in RBA. For example, the difference between population and performance accountability reflects real differences in the nature of efforts to improve quality of life as opposed to delivering services. It is not a distinction you could set aside and still say you were using RBA. Other parts of RBA, such as the Population and Performance talk to action 7 questions are central to the thinking process of RBA/OBA, and are best preserved in their original form, with commentary or supplemental material shown in parentheses. Most other material can be supplemented, adjusted, reordered or rephrased as circumstances dictate. A famous music teacher was once quoted as saying that you should learn the rules of harmony so you can decide when not to use them.

  1. Are sub-populations okay? How many layers can you have between services at the bottom and high level? Does it matter?

See the discussions of subpopulations on pages 39 and 99 of “Trying Hard Is Not Good Enough.” One of the principle distinctions between population accountability and performance accountability is the service/customer relationship at the heart of all matters of service performance. Any time a group is defined in terms of its receipt of a particular service, it is a customer population and therefore part of performance accountability.    //   Within performance accountability, there are higher/lower or larger/smaller customer populations. For example, the customer population of a particular service is smaller than the customer population of the whole agency which is smaller than the customer population of the service system to which the agency “belongs.”  //     In population accountability, populations and subpopulations can be defined in almost any way you like. In every case this definition must involve a geographic area. So for example, we could consider the well-being of the population of all people in Wales, all children in Wales, all children in Cardiff, all children 0 to 5 in Cardiff, etc. RBA can be used for whole populations or subpopulations depending on the circumstances. //  One place where subpopulation considerations are also important in RBA is in the “Story behind the baseline.” If we are trying to understand the rate of teenage pregnancy, or school failure, it may be useful to examine how these problems are concentrated in certain areas or certain subpopulations so that we can tailor our efforts to greatest effect.  //  Another interesting part of the population / subpopulation discussion is the fact that “target populations” are part of population accountability. They represent the potential group from which actual customers are drawn, who then become part of the customer population.  //  The conclusion from all this is that there are not  strict rules about how many levels there are in population and performance accountability. It’s as many levels as are needed.

  1. Can we have a hierarchy of outcomes? [e.g. by sub-population] 

There is a long history to confusion about “outcomes.” The first matter of confusion has to do with language itself. What is meant by an “outcome” or a “result.” I believe that the language discipline in RBA provides a template for how to reach agreement on some form of common language. In it’s most basic form, you only need agreement on how to label six ideas in order for people across government and the voluntary sector to talk to each other in a disciplined way. Those six ideas are presented on the Definitions slide of the RBA 101 presentation and pages 19 and 20 of the book. The words I most often use to label these ideas in Wales are (Results, Indicators, Performance Measures, How much did we do? How well did we do it? Is anyone better off?)  //  Having said that, you can use RBA as a diagnostic tool to see if people have gotten ideas and words mixed up. It is not uncommon for the word “outcome” to be attached to several different ideas. Or for a single idea to have several different words associated with it. For example, in some of the original work on ECM in England, they used a two tier outcome structure at the population level, but they called the two level of outcomes by different names (outcomes and aims). When you look at the content of these two differently named things it turns out they are all quality of life statements and therefore they are all population “results” or “outcomes.” //  In this case ECM was proposing a two tier result or outcome structure at the top. Two and sometimes even three tier systems are not uncommon. But on page 53-54 of the book I show how you can often simplify the structure by changing the third tier into indicators. //  It is likely that the sorting out you are thinking about has to do with first making clear the distinction between population and performance accountability and then finding a disciplined way of using language to label important ideas like those six discussed above.

  1. Could MF do a critique of Welsh Performance Management Framework as he did for English?

Again, I am not familiar with the Welsh Performance Management Framework. Why don’t you send it to me to look at. //  It is often important to get the auspices right under which a written review is produced. It is always best if the review is requested by the relevant authority. Where this can not be done, it is possible to create an analysis that others can use to influence decisions in the right direction. //  Still another alternative is for a resident RBA expert in Wales to analyse this framework and draft something for others to look at. I am very committed to people building their own capacity to do this work.  //  I have to be careful about promising written comments on any complex document. Perhaps, after looking at this, the questioner’s purposes could be met with a conference or web call.

  1. What similarities and differences are there between performance frameworks (generally) and RBA accountability?

This is a complex question, but here is a simple answer. Most other frameworks (1) do not clearly distinguish between population and performance accountability, (2) often have a large and confused set of performance measures, in contrast to the three plain English ones used in RBA (How much did we do? How well did we do it? Is anyone better off?), and (3) many other frameworks are unwieldy, paper-intensive processes that are not actually useful to the people who have to fill out the forms, in contrast to the RBA 7 talk to action questions.  // All people who have invented these kind of frameworks, including me, are essentially trying to do the same thing, that is provide tools to help make the world better. So comparing frameworks is a matter of judging which of these approaches works best for you. You might want to look at page 135 of “Trying Hard Is Not Good Enough” where I discuss “Comparing Frameworks,” and suggest that you judge alternatives by how well they measure up against some simple criteria: “simple, common sense, plain language, minimum paper and useful.”  //  There are, of course, other differences between RBA and traditional frameworks, including the use of baselines to judge progress, the use of targets only when they are fair and useful, and the emphasis on no-cost and low-cost ideas. But let me leave it at that for now.

  1. How credible is use of self-reporting tools on soft outcomes against more scientific tools?

This is a question about the reliability of data. There are basically two ways to know something in the social sciences. You can observe it (e.g. a person got a job or didn’t, was re-admitted to hospital within 30 days or wasn’t) or you can get some form of opinion from the customer, worker or third party. (e.g. the young person thought the mentoring program helped him or her do better in school, the older person thought the in-home attendant was kind, caring and helpful, the teacher thought the child was making progress on social skills, the parent reported better relations with his or her child etc. ) We sometimes refer to reported opinion as “soft” data. And there are known problems associated with interpreting customer satisfaction data. But this does not mean that it is not important or does not have a place. My recommendation is always to use “soft” data in conjunction with data based on observable facts.

  1. How do you turn the curve when there is a lack of available data, or data takes time to come through? What do you do in the meantime?

This answer may sound a little odd. But imagine the most important measure you could conceivably have. Now you have two choices. You can create a “pseudo-baseline” as we do in the Turn the Curve exercise, where we estimate (1) where we are now on this measure, (2) what we think the history has been. Has it been getting better or worse or about the same?, and (3) where do we think it’s headed in the future if we don’t do something more or different than what we’re doing now.  This working version of a baseline can be used to drive the basic discussions in getting from talk to action in RBA, namely, What is the story behind the baseline? Who are the partners with a role to play in doing better? What works to do better? and What do we propose to actually do?   If for some reason you don’t want to create a pseudo or working baseline, then you can simply reach a consensus on whether things are getting better or worse. In this case the direction of the baseline is used to drive the turn the curve discussion. There is a third somewhat more complex variation of this presented on pages 128 and 129 of the RBA book.  //  The most extreme variation would set data aside altogether and ask What is the purpose of the service? How are we doing and why? Partners…..What works….Action Plan. If this helps, then use it. But this should make you curious about how to know with greater certainty the answer to the “How are we doing?” question. This inevitably requires some form of data. So create a Data Development Agenda and begin gathering the data you need to manage and improve the service. Workers in the organization will generally be more agreeable about gathering data if they can see from the beginning that it is needed to answer a fundamental question about the effectiveness of their work. Be careful not to gather more data than you need. Many data systems place unnecessary burdens on workers and actually detract from service delivery. Data systems should be useful to the line workers who generate the data. You might want to look into case management software like that of RLG’s partner Social Solutions. // The principle here is NOT to wait for data or any other precondition to get started. Get started with what you have. Make informed guesses about what you don’t have. Be deliberate about making the process better and more complete over time.

Mark FriedmanQuestions & Answers from Wales Master Class

Leave a Reply

Your email address will not be published. Required fields are marked *