The Matter of Evidence

Mark Friedman Publications, RBA Resources Leave a Comment

A Short Treatise on the Rules of Evidence in Budget Court

In order to show that there are benefits of investing in prevention, and in particular, that there are cost savings, we need evidence. What kind of evidence do we need? What will pass muster? What will help us make a convincing case? What will allow us to close a deal for reinvesting savings in improved family and children’s services? What do our partners “need to see” to be convinced that savings are real and available for reinvestment?

 This is, at the bottom line, a budget question. And it leads us to frame the discussion in terms of what passes for evidence” in the budget processes where we must compete for investment and reinvestment money?

Rules of Evidence: How do we know? How do we convince those in power?

In the courtroom the rules of evidence are written down and pondered over. Lawyers specialize in “evidence”. There are Supreme Court decisions. Budget processes are a lot like judicial processes. Decisions are made which affect the lives and livelihood of many people. But the process of making those decisions is sometimes hard to fathom. While it is not always true, budget decisions are also made on the basis of evidence. And these rules are not written down anywhere. They must be intuited from the ways in which budget decision makers decide. And of course they change from one court to another, and from one budget session to another.

These facts are of course maddening to those who are not part of the budget priesthood, and yet who must interact with, and in many cased depend upon, budget work and budget decisions. How can we begin to understand these processes, and be more successful in negotiating them? One place to start is with the unwritten rules of evidence in the budget process.

All budget people are from Missouri. “Show me.” is the most basic dictum in the budget world. But show me what? That is the question. Let’s take an example an show how to answer this question. We take as our example the complex and challenging work of proving that a particular strategy/program/action has saved money and is therefore a good place to “invest” more money. The reason for this choice has more to do with the fact that these questions are now being raised in relation to family and children’s services and how to get ahead of the curve with investments in prevention. This work is taking place in many parts of the country. But, in truth, if we can answer the rules of evidence question here, we will have gone a long way to shining light on the rules that apply in other situations.

We put forward the proposition that there are three types of evidence which can be brought to bear on “proving” this point.

But first, a word about “proof.” “Prove it to me,” is the old playground challenge, which in this case must be left there. The budget world sometimes uses research and the work of researchers to “prove” one thing or another through some application of the scientific method. But the budget world is not bound by scientific precepts. It is a curious mixture of fact and art, politics and persuasion, discipline and pure hunch. There is no such thing as “proof” in the budget world. The best we get is strong, and often circumstantial, evidence that something is true. Forget proof. You are not even going for true beyond a reasonable doubt. Budget people never stop doubting just about everything except themselves. You are trying to convince the jury that something is most likely true and may continue to be true in the future.

And a word about combining evidence In this business, the more evidence the better. And when one piece of evidence corroborates another piece of evidence, that’s even better. People often make the mistake of relying on one piece of evidence as conclusive. As we discuss types of evidence, we argue that the best case is to have all three types of evidence, mutually reinforcing.

And that leads us to the three types of evidence: group evidence, systemic evidence and simulation evidence:

Type 1: Group Evidence

This type of evidence shows that a particular effect works for a particular group or population. Such groups are often samples of a larger population. For example, we can show that a particular program keeps children out of foster care, or improves reading scores. Group evidence often comes from a pilot or demonstration program and the evaluation findings for that program.

Types of Groups: There are many types of groups. But, three in particular are relevant to our discussion of reinvestment. Pilots, samples and cohortsPilots are generally structured around a whole population in a small geographic area. Samples are just that, samples of larger populations. Cohorts are a very particular type of subpopulation which have particular relevance to budgeting. Many programs operate in such a way that people enter and exit the program and what happens in the middle costs money. It is possible to think about the people entering the program in the way we think about classes moving through the education system. If we can show that a particular change in the program affects one class and its associated costs in a positive way, it helps answer two difficult questions in budgeting: what is the effect for the whole program; and how could we estimate the benefits of phasing in the new approach. Cohorts are natural ways to phase in changes of policy and attendant cost effects.

Before we go on to systmic evidence, a word about anecdotes. Anecdotal evidence is actually a form of group evidence. It says, “Here’s how this worked for one person… And maybe it’s working the same way for others like him or her.” Anecdotes are sometimes dismissed as, well, anecdotal. But they can be powerful stories that connect to real world experience and that work to corroborate other evidence. Anecdotal evidence sometimes make the difference between winning or losing support.

Type 2: Systemic Evidence

Budget people live at the system level. By system level we mean the level at which whole programs cost money. What is the total cost of the Medicaid program, the welfare program, the foster care program? Budget directors are concerned with where the bottom line total cost (and more importantly the general fund share) of these programs is headed. Changes which don’t show up on this radar screen don’t matter. The second type of evidence is the evidence that some action or change had an effect at the systemic level. We have lots of examples of this. If we raise welfare benefits by $5.00 per month we see an immediate and predictable increase in monthly cost. If we expand eligibility for a program we see a certain, but much less predictable, increase in cost.

Containing the risk of such increases is a major part of the ways in which budgets and programs are structured. It leads to three important types of program budgets, which in turn have a lot to do with how we see and use systemic evidence:

  • Uncapped entitlements: programs which by law spend as much money as required regardless of appropriations. There are a few of these left in the federal structure. They are known as “entitlements”
  • Weak cap, quasi-entitlements: These are programs that are capped in law but not in practice. They are programs which tend to go up based on need, because it is the practice of the body politic to respond to such need. The most important example here is prisons. But this also applies to many programs which are “caseload driven” such as foster care, investigation of child abuse and neglect, etc.
  • Fully capped non-entitlements: This is the most common type of program. The cap is established in law or in the budget appropriation process. And the cap is usually fully honored in practice.

These types of funding are important because many investment approaches involve spending a discrete amount on capped programs (like respite care or family support) in order to drive down cost in entitlement and quasi entitlement programs. In other words, the most important place to demonstrate systemic savings is in programs which can not be directly controlled with a cap. For fiscal people, this is part of the purpose of investment.

Baselines are essential to showing systemic evidence. A baseline includes an historical component, showing where we’ve been, and a forecast component showing where we’re headed if we stay on our current course. The most powerful form of systemic evidence takes the form of a departure from the baseline. If we can show that caseload or cost growth at the systemic level slowed and then began to decline relative to the baseline we have evidence that “something” is causing this to happen. If we can show that there is a plausible connection (and timely relationship) between a set of investments, and a consequent turn (for the better) away from the baseline, then we have circumstantial evidence of contribution if not full causality. This is usually the best you can do. (See the Strategy Map and other Finance Project papers for a fuller discussion of baseline concepts.)

Type 3: Simulation Evidence

One last challenge remains. If we have group evidence and we have systemic evidence, what do we need to make decisions about using this evidence to budget for new investments in the future? How in other words do we know what it will look like when we do program X for the whole population, not just the population in the little school on the east side? This is not a trivial problem. We don’t just multiply by the difference in population. We need to consider a wide range of factors including:

  • the differences between the test population and the whole population,
  • the choices about “which version” of the program to implement
  • the questions of economy of scale
  • the ripple effects of program changes on other programs
  • the phase in schedule and approach
  • the changing demographics of the population

It is, in other words, a whole lot more complicated to take an action, or implement a program or policy change at the system level than at the small scale level. (This is of course why pilots are relatively easy to do.) How do we get our minds around this problem in a useful way that allows us to consider our choices and make the best decision? This is where the third kind of evidence comes in: simulation evidence. This is a very specialized type of evidence, which is now making its appearance for the first time in courts of law. Computers are used to recreate accident scenes and simulate the appearance of the accident from different points of view, or to answer “what if” questions. Budget folks have been doing this for a long time. Budget analysis is often a kind of simulation evidentiary process: asking and answering questions about “scenarios.” A more powerful way to do this can be found in various types of computer simulations. There three types of computer simulations which can be used:

  • Cell-based modeling: Such models (including some very sophisticated social and economic models developed by such organizations as Mathematica and Brookings) involve the creation of a large and complex data base consisting of client or family “cells” which are given characteristics of the population to be studied and are programmed with variables which specify the ways in which each cell will respond to incentives or stimulus (for example, the wage levels at which work effort will increase or decrease). Simulations of policy which use these kinds of models develop total cost effects from the summation of the thousands of individual actions of the component cells.
  • Flow modeling: A second approach to modeling involves the depiction of system flow, and the movement of people (or materials) through the system. Such models allow the user to take stock of the system at any point in time, to vary the way in which parts of the system function, or to consider how the system responds to changes in volume (e.g. different sized cohorts) as these move through the system. These kinds of models are used for everything from analysis of social programs to industrial design and analysis.
  • Matrix modeling: This is the kind of modeling most commonly used in budget processes and is usually created with spreadsheet software. These models often take the form of a computerized “fiscal note,” which can be used to test the different effects which arise from different levels of program performance or from different characteristics of program components or clients.

This last form of simulation is the one most relevant to the matter of reinvestment. With this kind of simulation, it is possible to model the costs and benefits of taking a demonstration program to scale. In effect, the simulation allows you to look at what would happen if the group and systemic evidence played out on a larger scale.

The Evidentiary Challenge of Reinvestment

The most expensive programs we operate are remediation programs. We spend enormous sums trying to fix the problems of children and families after they develop. These programs are often those in the first two funding categories: uncapped or quasi uncapped. The principle challenge is to show what kinds of changes in programming or other actions can save money in these program costs. This is big money in state budgets. The cost of remediating problems for children and families is taking up an increasing share of federal state and local budgets. The problem is that we have a lot of good ideas about what to do about this – invest early in children, before problems occur – it is cheaper to prevent than fix – but we have relatively little evidence of this at the systemic level. We need to begin to build all three types of evidence to show how and exactly to what extent various strategies can save and avoid costs and warrant full scale implementation.

Mark FriedmanThe Matter of Evidence

Leave a Reply

Your email address will not be published. Required fields are marked *