There have been many criticisms of econometrics' usefulness as a discipline and perceived widespread methodological shortcomings in econometric modelling practices.
Difficulties in model specification
editLike other forms of statistical analysis, badly specified econometric models may show a spurious correlation where two variables are correlated but causally unrelated. Economist Ronald Coase is widely reported to have said "if you torture the data long enough it will confess".[1] McCloskey argues that in published econometric work, economists often fail to use economic reasoning for including or excluding variables, equivocate statistical significance with substantial significance, and fail to report the power of their findings.[2]
Economic variables are not readily isolated for experimental testing, but Edward Leamer argues that there is no essential difference between econometric analysis and randomized trials or controlled trials provided the use of statistical techniques reduces the specification bias, the effects of collinearity between the variables, to the same order as the uncertainty due to the sample size.[3]
Economists are often faced with a high number of often highly collinear potential explanatory variables, leaving researcher bias to play an important role in their selection. Leamer argues that economists can mitigate this by running statistical tests with different specified models and discarding any inferences which prove to be "fragile", concluding that "professionals ... properly withhold belief until an inference can be shown to be adequately insensitive to the choice of assumptions".[4] However, as Sala-I-Martin[5] showed, it is often the case that you can specify two models suggesting contrary relation between two variables. The phenomenon was labeled 'emerging recalcitrant result' phenomenon by Robert Goldfarb.[6]
Lucas critique
editRobert Lucas criticised the use of overly simplistic econometric models of the macroeconomy to predict the implications of economic policy, arguing that the structural relationships observed in historical models break down if decision makers adjust their preferences to reflect policy changes. Lucas argued that policy conclusions drawn from contemporary large-scale macroeconometric models were invalid as economic actors would change their expectations of the future and adjust their behaviour accordingly.
Lucas argued a good macroeconometric model should incorporate microfoundations to model the effects of policy change, with equations representing economic representative agents responding to economic changes based on rational expectations of the future; implying their pattern of behaviour might be quite different if economic policy changed.
Modern complex econometric models tend to be designed with the Lucas critique and rational expectations in mind, but Robert Solow argued that some of these modern dynamic stochastic general equilibrium models were no better as the assumptions they made about economic behaviour at the micro level were "generally phony".[7]
Other mainstream critiques
editLooking primarily at macroeconomics, Lawrence Summers has criticized econometric formalism, arguing that "the empirical facts of which we are most confident and which provide the most secure basis for theory are those that require the least sophisticated statistical analysis to perceive." He looks at two highly praised macroeconometric studies (Hansen & Singleton (1982, 1983), and Bernanke (1986)), and argues that while both make brilliant use of econometric methods, both papers do not really prove anything that future theory can build on. Noting that in the natural sciences, "investigators rush to check out the validity of claims made by rival laboratories and then build on them," Summers points out that this rarely happen in economics, which to him is a result of the fact that "the results [of econometric studies] are rarely an important input to theory creation or the evolution of professional opinion more generally." To Summers:[8]
Successful empirical research has been characterized by attempts to gauge the strength of associations rather than to estimate structural parameters, verbal characterizations of how causal relations might operate rather than explicit mathematical models, and the skillful use of carefully chosen natural experiments rather than sophisticated statistical technique to achieve identification.
Austrian School critique
editThe current-day Austrian School of Economics typically rejects much of econometric modeling. The historical data used to make econometric models, they claim, represents behavior under circumstances idiosyncratic to the past; thus econometric models show correlational, not causal, relationships. Econometricians have addressed this criticism by adopting quasi-experimental methodologies. Austrian school economists remain skeptical of these corrected models, continuing in their belief that statistical methods are unsuited for the social sciences.[9]
The Austrian School holds that the counterfactual must be known for a causal relationship to be established. The changes due to the counterfactual could then be extracted from the observed changes, leaving only the changes caused by the variable. Meeting this critique is very challenging since "there is no dependable method for ascertaining the uniquely correct counterfactual" for historical data.[10] For non-historical data, the Austrian critique is met with randomized controlled trials. In randomized controlled trials, the control group acts as the counterfactual since they experience, on average, what the treatment group would have experienced had they not been treated. It is on this sound basis that parametric statistics (in the Gaussian sense) is based. Randomized controlled trials must be purposefully prepared, which historical data is not.[11] The use of randomized controlled trials is becoming more common in social science research. In the United States, for example, the Education Sciences Reform Act of 2002 made funding for education research contingent on scientific validity defined in part as "experimental designs using random assignment, when feasible."[12] In answering questions of causation, parametric statistics only addresses the Austrian critique in randomized controlled trials.
If the data is not from a randomized controlled trial, econometricians meet the Austrian critique with quasi-experimental methodologies. These methodologies attempt to extract the counterfactual post-hoc so that the use of the tools of parametric statistics is justified. Since parametric statistics depends on any observation following a Gaussian distribution, which is only guaranteed by the central limit theorem in a randomization methodology, the use of tools such as the confidence interval will be outside of their specification: the amount of selection bias will always be unknown.[13] A better approximation to a randomized controlled trial provided by a quasi-experimental method will reduce this selection bias, but these methods are not rigorous, and one cannot deduce precisely how incorrect the familiar parametric measures such as power and statistical significance will be if they are calculated on these additional assumptions. When parametric statistics are used beyond their specifications, Econometricians argue that the insight will exceed the inaccuracy while Austrians argue that the inaccuracy will exceed the insight. A historical example of this debate is the Friesh–Leontief "Pitfalls" debate, with Friesh holding the Austrian position and Leontief holding the econometric position.[14] Structural causal modeling, which attempts to formalize the limitations of quasi-experimental methods from a causality perspective, allowing experimenters to precisely quantify the risks of quasi-experimental research, is an emerging discipline originating with the work of Judea Pearl.
Critque of practice
editKennedy (1998, p 1-2) reports econometricians as being accused of using sledgehammers to crack open peanuts. That is they use a wide range of complex statistical techniques while turning a blind eye to data deficiencies and the many questionable assumptions required for the application of these techniques.[15] Kennedy quotes Stefan Valavanis's 1959 Econometrics textbook's critique of practice:
Econometric theory is like an exquisitely balanced French recipe, spelling out precisely with how many turns to mix the sauce, how many carats of spice to add, and for how many milliseconds to bake the mixture at exactly 474 degrees of temperature. But when the statistical cook turns to raw materials, he finds that hearts of cactus fruit are unavailable, so he substitutes chunks of cantaloupe; where the recipe calls for vermicelli he used shredded wheat; and he substitutes green garment die for curry, ping-pong balls for turtles eggs, and for Chalifougnac vintage 1883, a can of turpentine. (1959, p.83)[16]
See also
editNotes
edit- ^ Gordon Tullock, "A Comment on Daniel Klein's 'A Plea to Economists Who Favor Liberty'", Eastern Economic Journal, Spring 2001, note 2 (Text: "As Ronald Coase says, 'if you torture the data long enough it will confess'." Note: "I have heard him say this several times. So far as I know he has never published it.")
- ^ McCloskey, D.N. (May 1985). "The Loss Function has been mislaid: the Rhetoric of Significance Tests" (PDF). American Economic Review. 75 (2): 201–205.
- ^ Leamer, Edward (March 1983). "Let's Take the Con out of Econometrics". American Economic Review. 73 (1): 31–43. JSTOR 1803924.
- ^ Leamer, Edward (March 1983). "Let's Take the Con out of Econometrics". American Economic Review. 73 (1): 31–43. JSTOR 1803924.
- ^ Sala-i-Martin, Xavier X (November 1997). "I Just Ran Four Million Regressions". Working Paper Series. doi:10.3386/w6252.
{{cite journal}}: Cite journal requires|journal=(help) - ^ Goldfarb, Robert S. (December 1997). "Now you see it, now you don't: emerging contrary results in economics". Journal of Economic Methodology. 4 (2): 221–244. doi:10.1080/13501789700000016. ISSN 1350-178X.
- ^ Solow, R. (2010) "Building a Science of Economics for the Real World" Archived February 4, 2011, at the Wayback Machine, Prepared Statement of Robert Solow, Professor Emeritus, MIT, to the House Committee on Science and Technology, Subcommittee on Investigations and Oversight: July 20, 2010
- ^ Summers, Lawrence (June 1991). "The Scientific Illusion in Empirical Macroeconomics". Scandinavian Journal of Economics. 93 (2): 129–148. doi:10.2307/3440321. JSTOR 3440321.
- ^ Garrison, Roger - in The Meaning of Ludwig von Mises: Contributions is Economics, Sociology, Epistemology, and Political Philosophy, ed. Herbener, pp. 102-117. "Mises and His Methods"
- ^ DeMartino, George F. (2021). "The specter of irreparable ignorance: counterfactuals and causality in economics". Review of Evolutionary Political Economy. 2 (2): 253–276. doi:10.1007/s43253-020-00029-w. ISSN 2662-6136. PMC 7792558.
- ^ Angrist, Joshua; Pischke, Jörn-Steffen (15 December 2008). Mostly Harmless Econometrics. Princeton University Press. ISBN 978-1400829828.
- ^ Education Sciences Reform Act of 2002, Pub. L. 107–279; Approved Nov. 5, 2002; 116 Stat. 1941, As Amended Through P.L. 117–286, Enacted December 27, 2022 "https://www.govinfo.gov/content/pkg/COMPS-747/pdf/COMPS-747.pdf"
- ^ Harris, Anthony D.; McGregor, Jessina C.; Perencevich, Eli N.; Furuno, Jon P.; Zhu, Jingkun; Peterson, Dan E.; Finkelstein, Joseph (2006). "The Use and Interpretation of Quasi-Experimental Studies in Medical Informatics". Journal of the American Medical Informatics Association. 13 (1): 16–23. doi:10.1197/jamia.M1749. ISSN 1067-5027. PMC 1380192. PMID 16221933.
- ^ Leontief, Wassily W. (1934). "Pitfalls in the Construction of Demand and Supply Curves: A Reply". The Quarterly Journal of Economics. 48 (2): 355–361. doi:10.2307/1885615. ISSN 0033-5533. JSTOR 1885615.
- ^ Kennedy, P (1998) A Guide to Econometrics, Blackwell, 4th Edition
- ^ Valavanis, Stefan (1959) Econometrics, McGraw-Hill