The second way to address this is to convert program impact\ninto growth in percentage so that A-ROI results\nare standardized by change in reference to the baseline data. In this way,\nA-ROI results are represented by percentage of growth at a certain cost. With\nthe second approach, cost-effectiveness can be compared between the three\ninvestments because 1% change is considered to be equivalent across all three\noutcome measures. In other words, 1% growth in reading achievement is\nequivalent to 1% improvement in suspension reduction and sense of belonging, or\n2% increase in reading achievement is better than 1% improvement in suspension\nreduction and sense of belonging.<\/p>\n\n\n\n
It should be noted that both methods presented here ignore baseline differences between programs, which seems more or less likely to impact program effect. That is, for two programs that serve students of different reading levels (e.g., one serves students in the lowest 20th<\/sup> percentile and the other serves students in the bottom 5th<\/sup> percentile), one SD <\/em>or 1% improvement in reading achievement would be considered equivalent using either approach just discussed. In our experiences, obtaining such an improvement for students in the 5th<\/sup> percentile is probably qualitatively different from achieving the same for students in the 20th<\/sup> percentile. However, little research exists in this area to guide appropriate adjustments for taking baseline differences into consideration. <\/p>\n\n\n\n Building on the aforementioned two methods, the third way is\nto take the cost-utility approach by assigning different weights to the standardized\nA-ROI results (either in SD <\/em>or percentage of\ngrowth). In this way, 1 SD<\/em> or 1% change is not\nconsidered equivalent across different outcome measures. Rather, 1 SD<\/em>\nor 1% increase in reading achievement is equivalent to x SD<\/em> or x<\/em>%\nchange in suspension reduction and y SD <\/em>or y<\/em>%\nimprovement in sense of belonging, with x<\/em> and y<\/em>\nbeing the weights for the two latter outcome measures, respectively. <\/p>\n\n\n\n In essence, all of these three methods involve constructing\na new scale and projecting the raw A-ROI values onto the new scale through some\nsort of linear transformation, with each approach making a different assumption\nabout the appropriateness of the transformation that would render commensurate\nresults. These assumptions are necessary and largely value-driven, since there\nlacks an empirical basis for addressing commensurability when different outcome\nmeasures are involved. Because of this, in certain fields such as public\nhealth, it has been suggested that \u201cROI should only be used for equivalent\nalternatives and not to compare interventions that are different in their\nobjectives\u201d (Brousselle,\nBenmarhnia, & Benhadj, 2016) .<\/p>\n\n\n\n To further complicate the problem, often times, investment\nitems target more than one outcome for improvement. There are statistical\nmethods that reduce dimensionality or allow comparisons with multiple outcome\nvariables, which basically involve more sophisticated transformations. However,\nthey are too complex to conduct and often produce results that are difficult to\ninterpret. A more practical approach is to employ the cost-utility method to\ncombine multiple measures of effectiveness into a single estimate of utility (Levin &\nMcEwan, 2000).\nBased on those weights, a composite A-ROI can then be derived for each\ninvestment. With that said, if possible, it would be desirable to compare A-ROI\nresults for investments that target the same rather than different outcomes. <\/p>\n\n\n\n The second issue around commensurability of A-ROI results concerns linear extrapolation. When comparing two investments with different costs and returns, we are often implicitly engaged in a linear extrapolation, either upward or downward, depending on where the reference point is set. Figure 1 shows one example of such linear extrapolations involving comparing A-ROI results for two separate investments. In the chart, investment A, represented in a diamond shape, produces 0.3 SD<\/em> growth in reading achievement for 1,000 students at a cost of $500,000 ($500 per pupil), and investment B, represented in a square shape, produces 0.5 SD<\/em> growth in reading achievement for 250 students at a cost of $250,000 ($1,000 per pupil). <\/p>\n\n\n\n When comparing the two A-ROI results and concluding that, when cost is the same, investment A has a higher return and is thus more cost-effective, we are either extrapolating the A-ROI result of investment A upward along the solid blue line or extrapolating the A-ROI result of investment B downward along the solid green line, assuming that the relationship between cost and return remains unchanged in each case and return reduces to zero when there is no investment. <\/p>\n\n\n\n However, the relationship between cost and return is most\nlikely not linear for most investments. With economies of scale, it is possible\nfor an investment to cover more students without incurring cost proportionally,\nwhich, as a result, reduces its cost per pupil level. For example, investment B\nmight be able to cover the same number of students as investment A does by only\ndoubling the total cost, as opposed quadrupling the total cost. At the same\ntime, the return probably will not be reduced by half as the result of the scaling\nup. Consequently, the actual A-ROI for investment B might follow the dotted\ngreen line as more students are covered, which leads to investment B being more\ncost-effective when it is at the same cost per pupil level of investment A. <\/p>\n\n\n\n On the flip side, we could increase the cost per pupil level\nto boost return. For example, as an incentive program, investment A offers a $5,000\nannual bonus for high-quality teachers to teach special education classes with\nan average of 10 high-need students at low-performing schools. We could double\nthe incentive to $10,000 to attract more high-quality teachers to boost student\nachievement in those schools. However, we probably would not expect the return\nto be doubled as the result of the increased cost per pupil level. Consequently,\nthe actual A-ROI for investment A might follow the dotted blue line as the\nbonus amount increases, which, again, leads to investment A being less cost-effective\nwhen it is at the same cost per pupil level of investment B.<\/p>\n\n\n\n The above discussion is based on two investments only.\nThings could quickly become complex when more investments are involved. The\nchallenge here is to decide where the reference point is and figure out how\ncost per pupil changes when an investment is scaled up or down toward the\nreference point as well as the subsequent movement of return. Unfortunately,\nresearch in this area is rather thin and does not provide much guidance on how\nadjustments should be made to cost per pupil and return, respectively, when\ncomparing A-ROI results calculated from investments of different scales. <\/p>\n\n\n\n Since the impact of scale change is potentially non-linear for both cost and program effect, comparisons of A-ROIs between programs should be conducted and interpreted with great care, depending on the purpose of the comparisons and implications for decisions. Whenever the implementation scale of a program differs considerably from the scales based on which the A-ROIs are calculated, there is a risk of the comparisons being misleading. This is because, on the new scale, each of the compared programs could potentially have a new A-ROI value, which would then result in a different winner. <\/p>\n\n\n\n Third, as shown in Figure 1 in this post <\/a>discussing validity of A-ROI, program effect could vary in the first a few years and comparing A-ROI for programs at different implementation phases could lead to misleading or even wrong conclusions. For example, assuming investments B and C in Figure 1 started at the same cost per pupil level that remained unchanged in subsequent years, comparing investment B\u2019s year 1 result (full effect yet developed) and investment C\u2019s year 2 result (full effect realized) would lead to the conclusion of investment C being more cost-effective when the opposite is true after both investments become established with stabilized program effects. <\/p>\n\n\n\n Ideally, cost-effectiveness should be compared between\ninvestments with stabilized returns and costs. In reality, however, it is\ndifficult to know for sure when an investment\u2019s A-ROI becomes stable with the\nvariation being random fluctuation. Even when stabilized A-ROI results are\nattainable, there could be a variety of reasons (e.g., political pressure) for cost-effectiveness\ncomparison between programs at different phases of implementation. It is\nimportant to help decision makers be aware that some of the A-ROI results might\nstill be in flux. <\/p>\n\n\n\n Fourth, investment decisions many times involve comparing\nA-ROI results for investments from the same context or similar contexts. For\nexample, a school district might need to decide, between two programs, which one\nto retain and which one to cut due to a budget shortfall, or the district might\nbe weighing whether to replace its own elementary reading intervention program\nwith the one implemented in a neighboring school district. At other times, decision\nmaking requires comparing A-ROI results between investments from rather\ndifferent contexts, such as one investment implemented in small a rural school\ndistrict and the other implemented in a large school system that serves\nstudents from rural, suburban, and urban areas. <\/p>\n\n\n\n It is important to point out that, when comparing A-ROI\nbetween investments from different contexts, considerations need to be given to\nnot just adjustments relating to cost of living and labor market, but also how change\nin context could affect implementation and result in a different return. In\nother words, a 0.5 SD<\/em> growth in\nreading among low achieving students in a rural school district is probably not\ndirectly comparable to the same growth among low achieving students in a\nsuburban school district. <\/p>\n\n\n\n The last issue around commensurability of A-ROI results deals\nwith how to interpret investments with similar or even identical A-ROI values\nand their implications for decision making. One major motivation for A-ROI is\nto reduce the complexity of decision making by encapsulating multiple pieces of\ninformation including program effect and cost into a single data point. While\nachieving this goal, inevitably, some nuanced but important information, which could\nbe critical sometimes, is masked in this simplified representation. <\/p>\n\n\n\n One such example involves comparison between two investments, with one producing 0.1 SD<\/em> growth in reading at a cost of $100 per student and the other producing 0.8 SD<\/em> growth in reading at a cost of $800 per student. Computationally, these two investments have an identical A-ROI. Assuming both investments serve 500 at-risk students who share similar demographics and academic performance[1]<\/a>, practically, the $350,000 difference ($50,000 vs. $400,000) in cost could have a quite different implication than the 0.7 SD<\/em> difference in return, especially for financially strapped school districts. In this case, A-ROI basically loses its unidimensional discriminating power and it is helpful to reverse the encapsulation by projecting A-ROI results back onto some of the original dimensions. One strategy is to present the A-ROI results on the cost and return dimensions at the same time. Based on certain criteria, a district might group a portfolio of investments into nine categories shown in Table 1. While cells of the same background color have similar and potentially identical A-ROI results, this presentation makes the difference in cost and return explicit respectively for those investments. <\/p>\n\n\n\n Sometimes, we might need to go back further to re-introduce even more dimensions back. For example, investments A and B are both high return and high cost with similar A-ROI values for improving reading achievement. However, they differ in the students they intend to serve, with investment A targeting Tier 3 students and investment B targeting Tier 2 students. In this case, it might help to further breakdown the information from which the A-ROI results are derived. <\/p>\n\n\n\n Table 2[2]<\/a> provides an example of unpacking A-ROI along five dimensions for two investments that both produce 0.5 SD<\/em> increase in reading at a cost of $1,250 per pupil. Despite the identical A-ROI value, it shows the two investments differ in total cost, number of students served, and the particular group of students they served. At this point, A-ROI provides little value for comparing cost-effectiveness of the two investments. Leaders will have to base their decision on something else such as whether Tier 2 or 3 students should be the focus, if a choice has to be made between the two. <\/p>\n\n\n\n It can be argued that the information shown in Table 2 should be presented and used in all investment decisions, even for those involving investments with rather different A-ROI values. This is because a case can be made in some situations that it is more important to focus on a particular group of students even when the investment does not have the highest A-ROI, and albeit important, A-ROI is one piece of information leaders consider when making budget decisions, which are always multifaceted. <\/p>\n\n\n\n In this post, we focus on issues around comparing A-ROI results for cost-effectiveness after they become available to decision makers. The intention is not to discourage people from using A-ROI for that purpose because of these both complex and complicated issues. Rather, the goal is to: 1) propose solutions when A-ROI results cannot be easily compared (e.g., different outcome measures, multiple measures, identical A-ROI values) and 2) make explicit the assumptions embedded in comparisons when A-ROI results can be compared so that proper adjustments can be made (e.g., two investments are on very different implementation scales, but we have some understanding of how program effect and cost per pupil level for one investment might move when it is scaled up or down to the reference point) and inappropriate comparisons can be avoided (e.g., two programs are on very different implementation scales, but we know little about how program effect and cost per pupil level might change for either investment when it is scaled up or down to the reference point). <\/p>\n\n\n\n![\"\"](\"http:\/\/cyclebasedbudgeting.org\/wp-content\/uploads\/2019\/06\/1237_fig1.png\")
![\"\"](\"http:\/\/cyclebasedbudgeting.org\/wp-content\/uploads\/2019\/06\/1237_tb1.png\")
![\"\"](\"http:\/\/cyclebasedbudgeting.org\/wp-content\/uploads\/2019\/06\/1237_tb2.png\")
Continue Reading:<\/h4>\n\n\n\n
\n\n\n\nNOTE<\/h3>\n\n\n\n