{"id":1216,"date":"2019-05-10T01:31:10","date_gmt":"2019-05-10T01:31:10","guid":{"rendered":"http:\/\/cyclebasedbudgeting.org\/?p=1216"},"modified":"2020-09-23T05:18:34","modified_gmt":"2020-09-23T05:18:34","slug":"four-issues-around-using-academic-return-on-investment-a-roi-to-inform-and-improve-decisions-part-ii-uncertainty","status":"publish","type":"post","link":"https:\/\/cyclebasedbudgeting.org\/?p=1216","title":{"rendered":"Five Issues around Using Academic Return on Investment (A-ROI) to Inform and Improve Decisions: Part II – Uncertainty"},"content":{"rendered":"\n

In the first post<\/a> of this series, I discussed validity of A-ROI as a measure of cost-effectiveness. In this post, I focus on uncertainty embedded in A-ROI results. <\/em><\/p>\n\n\n\n

<\/div>\n\n\n\n

In the business world, ROI is largely treated as an\naccounting measure with certainty, but the certainty only applies to the\naccounting period. That is, for a three-year investment, the ROI result is a\nboth accurate and precise representation of its profitability over those three years only. Consumers of ROI\ninformation are admonished about risks when applying it to the future or other\ncontexts. Generally, little is provided\nabout those risks other than the fine print of \u201cPast performance is no\nguarantee of future results.\u201d, which can be\nfound in almost every mutual fund prospectus or other investment\ndisclosures.<\/p>\n\n\n\n

In educational contexts, however, the A-ROI formulated by\nLevenson and traditional CER have been approached as point estimates for which\nwe don\u2019t know about their accuracy and precision with one-hundred percent\ncertainty, due to both random errors (e.g., sampling error, measurement error,\ndata entry error) and potentially systematic errors (e.g., omitted variables,\nmodel misspecification, research design flaws). This uncertainty has two direct\nimplications for the use of A-ROI in high-stakes budgetary decision making. One\nis that we cannot simply judge the cost-effectiveness of an investment by its\nA-ROI result. The other is that assessing the worthiness of multiple\ninvestments is more complex than straightforward comparisons of the A-ROI\nface-values.<\/p>\n\n\n\n

For a single point estimate of program effect, the conventional way of dealing with uncertainty associated with sampling error is to conduct null hypothesis significance testing (NHST), which involves using the 5% significance level to heed false positives and conducting power analysis to gauge the likelihood of false negatives. Uncertainty associated with systematic errors can be best addressed through rigorous research designs such as randomized control trials and quasi-experimental studies (Campbell & Stanley, 1963). There are also various sensitivity analysis techniques (Frank, 2000; Rosenbaum, 2010) to quantify potential bias and robustness of single point estimates. <\/p>\n\n\n\n

With respect to comparing multiple point estimates, while\ntempting, it is problematic to compare the statistical significance of those\npoint estimates to assess which programs are more investment-worthy because, as\nexplained by Gelman & Stern (2006), the difference between\n\u201csignificant\u201d and \u201cnot significant\u201d is not itself statistically significant.\nFor effect sizes from two or more independent studies, the uncertainty\nassociated with examining the heterogeneity of those effect sizes can be dealt\nwith by employing the method developed by Rosenthal & Rubin (1982) when estimates of variance\nare available. <\/p>\n\n\n\n

The aforementioned design and statistical methods are part of a large body of scholarly work that addresses uncertainty in scientific inquires. These various research designs and methods have undoubtedly improved our ability to reduce or quantify uncertainty in our analysis results for decision making. For practical purposes, however, they suffer from two main drawbacks. One is the complexity involved in employing these techniques, which makes their utilization rare in most school districts that don\u2019t have researchers or analysts. <\/p>\n\n\n\n

The more pressing issue, however, is concerned with\ncommunicating quantified uncertainty to practitioners effectively so that they\ncan interpret the statistical inference results properly for contextualized\ndecisions. And that requires clear, mutually understood terms (Fischhoff\n& Davis, 2014).\nGiven the widespread misunderstanding and misinterpretation of statistical\nresults among social science researchers (Mittag &\nThompson, 2000; Nelson, Rosenthal, & Rosnow, 1986), it is hard to imagine the\ntask will be any easier with practitioners. <\/p>\n\n\n\n

In spite of their predominance in social sciences, those\nsophisticated techniques are not the only weapon in our battle against\nuncertainty. Replication, which provides \u201ccritical information about the\nveracity and robustness of research findings\u201d (The National\nScience Foundation & The Institute of Education Sciences, 2018), is another powerful tool for\nresearchers.       <\/p>\n\n\n\n

In recent years, there has been an intensified interest in the so-called replication crisis (or reproducibility crisis) reported in multiple disciplines. This intensified interest has led to multiple debates and discussions, which focus on various aspects of replication based on different definitions of replication (Schmidt, 2016). For the purpose of this discussion that concerns uncertainty in A-ROI results, we adopt the replication standard proposed by Clemens (2017), which differentiates four different types of replication studies. <\/p>\n\n\n\n

Observing confusion and harm resulting from lack of a consensus standard for determining what constitutes a replication, Clemens proposed to classify replication studies as \u201cReplication<\/em>\u201d and \u201cRobustness<\/em>\u201d (See Table 1).  Based on the methods in follow-up studies versus those reported in the original, \u201cReplication<\/em>\u201d can be further classified as \u201cVerification<\/em>\u201d and \u201cReproduction<\/em>\u201d; and \u201cRobustness<\/em>\u201d can be further classified as \u201cReanalysis<\/em>\u201d and \u201cExtension<\/em>\u201d. For the rest of the discussion, replication refers to \u201cReproduction<\/em>\u201d.  <\/p>\n\n\n\n

\"\"
Table 1 Standard for classifying any study as a replication<\/em>
Note. From \u201cThe Meaning of Failed Replications: A Review and Proposal,\u201d by M. A. Clemens, 2015, Journal of Economics Surveys, Vol. 00, No. 0, p. 3. Copyright 2015 by John Wiley & Sons publications. Adapted with permission.
<\/figcaption><\/figure>\n\n\n\n

With the estimate from each replication that involves\nconducting the same analysis with a different sample from the same population,\nwe get a sampling distribution of the program effect. Based on the\ndistribution, we can use the average to estimate the program effect and\ncalculate standard error (standard deviation of the sampling distribution) to\ngauge the degree of sample-to-sample variability we can expect if the\ninvestment continues. <\/p>\n\n\n\n

As a result, NHST is no longer necessary to deal with the\nuncertainty associated with sampling error as long as there are sufficient\nreplications (Kline, 2004).  In addition, \u2018Reproduction<\/em>\u201d helps reduce uncertainty associated with measurement\nerror, coding errors, and low power (Clemens,\n2017).\nThis type of replication may also help address uncertainty associated with\nsystematic errors such as confounding since both confirmations and\ndisconfirmations can reduce uncertainty (McGrath\n& Brinberg, 1983). For example, after consistent results[1]<\/a>\nfrom three replication studies of an intervention program in a school, the\nfourth replication yields significant deviation which coincides with the\ndiscontinuation of a practice in the school. This deviation suggests the\npossibility of the original and subsequent consistent findings being\nconfounded, which necessitates further investigation. <\/p>\n\n\n\n

It is somewhat ironic that as we become more certain about the potential inaccuracy of our earlier results due to the disconfirmation, we are less certain about the effect of the program. However, at the very least, we know we cannot put much confidence in the result without further investigation. This is in contrast to the situation with a single study where we practically deem the effect \u201cscientifically proven\u201d without knowing the potential existence of biases from replications. In certain circumstances, we may even be able to identify, with additional information at hand, which scenario (See Table 2 in Part I \u2013 Validity<\/a>) the bias falls under and thus use the biased result if they are acceptable (See Table 3 in Part I \u2013 Validity<\/a>). <\/p>\n\n\n\n

By nature, an average derived from a sampling distribution\ncarries less uncertainty than a point estimate from a single study as long as\nthere are sufficient replications. The standard error can be translated into\nstatement describing percentage of times we can expect the program effect fall\nwithin the one or two standard errors of the mean. This information should be\nmore relevant, informative, and accessible for decision makers than confidence\ninterval that gives the range encompassing the true program effect with a high\nprobability (Howell,\n2012).\n<\/p>\n\n\n\n

Epistemologically, NHST and the various sensitivity and\nrobustness methods are our solutions to the problem of making inference based\non a single study. Despite repeated calls to lessen our dependence on this\ninductive logic and strengthen scientific inquiry through replications (Cohen, 1994;\nEdlund, 2016; King, 1995; Schneider, 2004), the \u201ccrisis\u201d has not\nimproved significantly. This is especially true in educational research where a\nrecent study found a replication rate of 0.13% among current top 100 education\njournals ranked by 5-year impact factor (Makel &\nPlucker, 2014).  <\/p>\n\n\n\n

There are a number of structural barriers that discourage\nreplications in academia, including editorial bias (Neuliep\n& Crandall, 1990, 1993), grant culture (Lilienfeld,\n2017),\nreputation and career advancement norms (The National\nScience Foundation & The Institute of Education Sciences, 2018), and feasibility constraints\nfor replications (Open Science\nCollaboration, 2015). These structural barriers are difficult to\novercome, which explain the lack of replications in multiple social science\nfields despite various efforts to improve the situation. <\/p>\n\n\n\n

In contrast, K-12 school systems provide friendly\nenvironments and optimal conditions for replications when it comes to\nevaluating a program\u2019s effect. In many school districts, it is not uncommon to\nsee some programs implemented year after year without change. For these\nprograms, evaluation conducted in each year that employs the same design and\nmethod can be treated as a replication (\u201cReproduction<\/em>\u201d)\nas long as the assumption that program participants in those years are from the\nsame population is not violated. <\/p>\n\n\n\n

Within a school district, that assumption seems quite  plausible when there are no boundary changes\nin school assignment, large student migration in and out of the school system,\nadjustments in program entrance and exit criteria, or significant improvement\nor deterioration of student achievement (including things that impact student\nachievement such as motivation) due to another program or district policies. <\/p>\n\n\n\n

It is important to point out that the above discussion by no means suggests that \u201cReproduction\u201d is superior to the other types of replication. However, as far as assessing the cost-effectiveness of an investment or comparing the cost-effectiveness of multiple investments in a K-12 setting is concerned, we think that, whenever doable, \u201cReproduction\u201d should be the preferred approach to gauging and communicating uncertainty in our A-ROI results. <\/p>\n\n\n\n

<\/div>\n\n\n\n

Continue Reading:<\/h4>\n\n\n\n