Researchers Unable To Replicate Findings of Published Economics Studies (businessinsider.com) 213
An anonymous reader writes: Federal Reserve economists Andrew Chang and Phillip Li looked at 67 papers in 13 reputable academic journals. Their findings were shocking. Without the help of the authors, only a third of the results could be independently replicated. Even with the author's help, only about half, or 49%, could. Business Insider reports: "It's a pretty massive issue for economics, especially given the impact that the subject has on public policy. Li and Chang use a well-known paper by Carmen Reinhart and Ken Rogoff as an example. The study showed a significant growth drop-off once a country's national debts reached 90% of gross domestic product, but three years after being published the study was found to contain a significant Microsoft Excel error that changed the magnitude of the effect." With cancer studies and most recently psychology studies all having replication trouble, these economics papers have some company.
Are we blaming Microsoft for this? (Score:3)
Re: (Score:3, Interesting)
Good one. But I always assume in situations such as "Economics Research" that there are all sorts of incentives to cook the books so to speak. Writing what policy makers want to hear will get you hired on as a White House advisor, where you can join the economic team that uses the flawed data to implement horrible economic plans of the type that the Republicans and DLC Democrats have been shoving down our throats since Reagan. So I am not entirely sure that those were just "honest mistakes".
Re:Are we blaming Microsoft for this? (Score:5, Insightful)
in situations such as "Economics Research" that there are all sorts of incentives to cook the book
This is not unique to economics. Most scientific fields have problems with replication. Journals are strongly biased toward publishing positive results, and nobody gets tenure for negative results or replication. I believe the last Nobel Prize for a failed experiment was Albert Michelson [wikipedia.org] in 1907. There are strong incentives to cheat, or at least cut corners.
You have failure backwards (Score:5, Informative)
This is not unique to economics. Most scientific fields have problems with replication. Journals are strongly biased toward publishing positive results, and nobody gets tenure for negative results or replication.
Economics is not a scientific field and the fields which seems to have the most problems with this seem to be medical, not scientific ones and "nobody gets tenure for negative results" is simply not true because I did! Indeed it is common in particle physics where we search for evidence of new physics beyond the Standard Model and, with only one exception so far, keep coming up empty handed. As for the most recent Nobel for a "failed" experiment try the one of two days ago: this was awarded to two experiments which failed to show that the Standard Model description of neutrinos was correct.
I think your definition of "failed experiment" needs almost completely reversing. Michelson-Morley was a stunning success: it completely destroyed the luminiferous aether model for light. It was not the result that was expected but that does not make it a failure. The same applies to neutrino oscillations. Not getting a result you expect from an experiment is the thing every scientist hopes for it because means that you have learnt something new about the universe which is why these experiments often win Nobel prizes. If anything is a failed experiment it is those that just end up confirming existing theories because you were hoping you might learn something new and instead just ended up confirming what you already knew.
Re:You have failure backwards (Score:5, Interesting)
Couldn't agree more Roger. Economics is not a field of scientific inquiry. It is far more prone to fudging than biochemistry, or my field, neuroscience. Any scientist can manipulate or cherry pick the data, but in the end it will only hurt their reputation and so most experienced researchers go to great lengths to make sure their experimental results are reproducible. I am notorious in the lab for insisting on more repetition of experiments that have worked well a number of times. Nobody likes doing repetitive and tedious experiments but it is part of the job.
I do however agree that fudging in science may be increasing, but that seems more due to the pressures of scarce funding. If science were funded as well as the military is, there would be no temptation for fudging on anyone's part.
Re: (Score:2)
Did you think your comment made sense? Think again.
Re: (Score:2)
I'm pretty sure most of these 'economic research' papers were used by industry to indicate to the government that the economy will do much better if the government gives them a bunch of money.
Re: (Score:2)
Re: (Score:2)
So I am not entirely sure that those were just "honest mistakes".
We can be a bit charitable and say that once they got the results they wanted/expected, they weren't as rigorous at checking the numbers as they should have been. (By contrast, I would expect that if the mistake had disproven their theory, they *would* have found that spreadsheet error.)
I do kind of feel for those guys - that's got to be embarassing to find out that you've made your bones on a typo.
Re: (Score:2)
That sounds about right.
Well, considering that as revealed by the OOXML ISO specifications there are quite a few functions in Excel - like FLOOR() and CEILING() - whose behavior is not the defined mathematical behavior...yes, to a certain degree Microsoft is culpable since it is a reasonable expectation that these functions should work as defined by mathematics and the result is subtle in its effect. (For instance, FLOOR(-2.5) in Excel will yield a result of -2 instead of -3.)
However, I would expect that someone that deals with
Why would anyone be shocked? (Score:5, Insightful)
Economics has always been one of the least predictive of "sciences". Economists with an ideological bent make things up with no relationship to the real world and people believe them.
Re:Why would anyone be shocked? (Score:4)
Not even close.
Look at mental health/psychology for a start.
That is just a joke when it comes to scientific method.
I suspect this is more a case of follow the money than actual bad science.. politics...
Re: (Score:2)
But because the economic actors are humans, many of them with wildly unpredicta
Re: (Score:3, Interesting)
Recognition that 2500 year old methods work is not a bad thing. The difference is that there is documented empirical evidence of the positive effect gained meditation using the scientific method. More, the underlying mechanisms of meditation are beginning to be understood in terms of neural and physiological systems. I'm not actually sure what you have against this research, but I doubt you would find one serious researcher that will describe meditation as a panacea.
Re: (Score:2)
I can guarantee that the ones who think of mindfulness as a panacea don't understand it. The ones who think it's a useful tool, however, ... well, you need more than just that one piece of information.
Re: (Score:2)
Mindfulness enables you to obtain the other mental tools you need.
Re: Why would anyone be shocked? (Score:2)
Hell there are still entire fields of economics including the Austrian model (which libertarianism is based on) that utterly rejects empiricism. They refuse to accept any contrary data as disproving a theory as a result.
Re: (Score:2, Insightful)
Economics is a "science" in the same way that a pickle is "candy".
The best that can be done is to make some generalized guesses based on hazy metrics and barely-understood past events.
Re:Why would anyone be shocked? (Score:5, Insightful)
Economics is a "science" in the same way that a pickle is "candy".
The best that can be done is to make some generalized guesses based on hazy metrics and barely-understood past events.
While that may be true, that is not the problem with these studies.
TFS is misleading here in referencing the problems in other disciplines, because there the replication problems often had to do with other scientists running an empirical experiment, collecting new data, and seeing whether the same trend occurs.
In THIS study, the "replication" problems were solely due to insufficient documentation. The categories for reasons that studies could not be replicated were listed as: (1) missing public data or code, (2) incorrect public data or code, (3) missing software, or (4) proprietary data.
There were no empirical "experiments" re-tested here. All they did was try to replicate data analysis, and "replication failed" when they couldn't access the relevant datasets or analytical software.
This is still a significant problem in economics, but the failure in "scientific" methodology here was of a VERY different kind -- it just had to do with access to research tools, not new data that conflicted with previous findings.
Re:Why would anyone be shocked? (Score:5, Interesting)
I just realized my post was slightly unclear -- the vast majority of "unsuccessful replication" had to do with problems with public access to data or data analysis methods.
But there were also about 1/4 of those studies which "failed replication" which failed due to errors in the data or the analysis of that data. In any case, no new data was collected here -- but "failure" here was mostly about lack of access.
(Of course, whether full access would have allowed successful replication is another question. Regardless, they weren't actually collecting new empirical data and "running experiments" again in the usual scientific sense.)
Re: (Score:2)
Economics has always been one of the least predictive of "sciences". Economists with an ideological bent make things up with no relationship to the real world and people believe them.
To be specific, politicians and pundits with a similar ideological bent believe them, and regular people just go along with whatever their favorite politicians and pundits tell them. Dismal indeed.
How hard could it be to design real-world economic experiments that actually yield useful and reproducible results? But it seems like everyone in the field already has an agenda, and the tendency is to only submit papers that seem to back up their own personal pet theories. This can be a problem in any disciplin
Re:Why would anyone be shocked? (Score:5, Insightful)
How hard could it be to design real-world economic experiments that actually yield useful and reproducible results?
I'm guessing that most countries wouldn't want to be the subject of a real-world economic experiment...
Re:Why would anyone be shocked? (Score:5, Insightful)
I'm guessing that most countries wouldn't want to be the subject of a real-world economic experiment...
Except that most countries are the subject of such experiments ... it's just that they are called "policy."
I have one (Score:2)
I'm intimately aware of one on the US state level: Brownbackistan. It's being transformed into a Koch Brothers Randian paradise (i.e., hell on earth for the 99%).
To their credit, they have proven that by cutting taxes and gutting the state's regulatory, financial, and educational systems, you can indeed end up with far, far fewer real jobs and a demonstrably lower standard of living for those that can't
Re: (Score:3)
It would be *extremely* difficult to run controlled experiments in economics. For one thing, just try to find two identical populations to run your test on. There are *WAY* too many plausible variables.
N.B.: That doesn't mean I don't think that most economists are politicians grinding an axe. It means that the ones who are trying to do decent economics studies would have a really hard problem even if they weren't being drowned out in noise that's essentially theological.
Re: (Score:2)
"It would be *extremely* difficult to run controlled experiments in economics."
No, it is extremely simple. In fact, it is so simple that every new government tries theirs as soon as it gets in charge: they call them policy. What you can't count on is controlled *repeatable* experiments. But, you see, theories doesn't need them either.
The problem is one of willness. Problem is those governments don't really want to conduct in proper way those experiments nor learn from the outputs and, much less, that ci
Re: (Score:2)
Didn't you just contradict yourself?
If policy is an attempted experiment and governments always fail to conduct properly controlled experiments, doesn't that end up meaning that it actually is difficult to run controlled experiments in economics? Calling it a problem of will is about as much hand-waving as Keynes' animal spirits.
Generally speaking, I can see where you might be able to run very simple controlled experiments, like taking a hot dog cart to different corners in a city and see how geography aff
Re: (Score:2)
"If policy is an attempted experiment and governments always fail to conduct properly controlled experiments, doesn't that end up meaning that it actually is difficult to run controlled experiments in economics?"
No, it isn't an attempted experiment, it *is* an experiment. It's only the ones conducting it don't want to clearly state their expectancies, neither note down the results nor publish them to public scrutiny. See, you may say it's very difficult for you to leave the chair in front of your computer
Re: (Score:2)
Even the person in the hot dog stand selling them could make a difference. Some people are naturally better at sales than others.
Re:Why would anyone be shocked? (Score:5, Interesting)
Most of academic economists are non-partisan. Or if they are, they normally have pretty solid theory and numbers to back their claims. The economists who do give the study of economics a bad reputation are those who actually make very opinionated and authoritative statements and reports, but hardly publish anything in a peer reviewed academic journal, where such antics wouldn't pass. Unfortunately, there are a few well known economists who had built a solid reputation in the research circles years ago, and then moved on past producing publishable academic research into the realm of partisanship. Any economist knows who they are. There are some well known liberals and conservatives among those.
Re: (Score:2)
I lol'ed. How many of those idiots still believe in infinite growth in a finite world?
Re: (Score:2)
How many people believe in answering a troll question that doesn't make any sense? Give a reference for your straw man argument please.
Re: (Score:2)
The burden of proof lies on you. You're the one claiming that economy is a science and that at least some economists have pretty solid theories and numbers.
No matter how many PhDs and fake Nobel prizes they give themselves, they're still idiots thinking that basic physics laws don't apply to their world.
Re: (Score:2)
Every other column he writes blames the Republicans and calls them idiots.
That's not partisan, that's just empirically derived from past evidence and performance.
Advocating defaulting on obligations to pay isn't smart.
Nor is cutting taxes on the rich and increasing taxes on the poor though sales and property taxes. Which is the truth of states like Texas that claim low income rates and make it up on non-income taxes, which disproportionately affect those with the lowest incomes, furthering a cycle poverty, instead of ending it.
Re: (Score:2)
I'm shocked that anyone was trying to reproduce an economics result. I genuinely had no idea they had ever tried.
Depends on what school of economics (Score:2)
The problem with economics is that is is the study of human interaction. There is no way to predict what will actually occur. The best you can do is derive general statements from first principles. The problem with this is the conclusions you reach tell politicians to do very little and the more they try to help the more they will hurt. This obviously isn't popular with politicians so they get people from schools that like to play with data fitting and pretend they can predict what the outcome will be when
Re: (Score:2)
Re: (Score:2)
It's extremely predictive. Given a limited set of variables, basic economic theory works pretty well - hedge funds bartering for futures or Maori tribes bartering for food.
There are, of course, a lot of exceptions. Those exceptions are the study of economics.
Re:Why would anyone be shocked? (Score:4, Funny)
Economists with an ideological bent make things up with no relationship to the real world and people believe them.
It's an old, but relevant, joke:
The First Law of Economics: For every economist, there exists an equal and opposite economist.
The Second Law of Economics: They're both wrong.
Re: (Score:2)
economics isn't a science like mathematics or physics or chemistry, which contain hard empirical facts infinitely repeatable.
its a behavioral science, like sociology. and human behavior is extremely unreliable and difficult to predict/replicate with certainty.
Re: (Score:2)
Funny, my economic theories predict and explain everything pretty perfectly. Granted, I don't try to predict the stock market or the rise of new nations with economics; you wouldn't use a blowtorch to drive a screw, either.
Modern economic theories are largely stoneage garbage. I dispensed with the term "value" because I decided it didn't have a place in civilized economics; after a while, I started researching economics (because I wrote my theories in a vacuum, having never studied economics myself, and
Re: (Score:2, Funny)
Re: (Score:2, Interesting)
Econ is a heckalot more predictive than most sciences!
If it has "science" as part of the name of the field, it isn't science. If it was, it wouldn't need it.
Re: (Score:2)
It's indeed a science, just a tricky one because the subject often behaves in unexpected ways.
Take for example how supply and demand influence price. I.e. more supply means reduced price, more demand means increased price. And 90% of the time, that hold's true, but it's that 10% where it throws the models off, and the cause could be something fickle like masses of people arbitrarily decided that the product has gone out of style and they don't want it anymore no matter what price it is sold at.
A few other f
Re:Why would anyone be shocked? (Score:4)
It's indeed a science
No, it's not a science in the same way that political science is not science. Economics may borrow some scientific methods and use them to study the field but the ultimate aim is to predict what will happen not to understand why (although knowing why may help with predicting) whereas the ultimate goal of science is to understand how and why things work with the ability to predict being a good signal that we got the how and why right.
...put don't take my word for it have a look at how many university science faculties have an economics department. There may be some but I honestly can't think of any.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Take for example how supply and demand influence price. I.e. more supply means reduced price, more demand means increased price. And 90% of the time, that hold's true, but it's that 10% where it throws the models off, and the cause could be something fickle like masses of people arbitrarily decided that the product has gone out of style and they don't want it anymore no matter what price it is sold at.
I've written theories largely based on cost, and handwaved price as a market economics topic. I believe that's a valid stance.
In my economic theories, the basis of productivity improvement is labor time reduction: if you need 10,000 man-hours to produce food for 10,000 people, each one person must work 10 hours to eat. As Adam Smith observed, you can compartmentalize this: 2,500 people can work 40 hours to feed everyone, and the other 7,500 can do something else. Adam Smith's observation was flawed i
Re:Why would anyone be shocked? (Score:5, Insightful)
As someone who has a degree in this goop. I can safely say most of it is bunk. When you show me how to measure 'happiness' and 'utility' (real variables in economics) will be the day I think you are onto something.
Most economic models work *very* well in a static system *if* you can perfectly measure *everything*. Oh and when I say everything I mean it. Take for example the recent 2008 fiasco. Very few saw it coming. Why? Because they could not measure what if say 20% defaulted.
Economics only works in hand wavy generalizations. Take for example min wage. Many cities raising it. Yet they have no idea what it will really do good or bad. From a pure econ view it is probably bad. But from a humanitarian view it is probably a good thing. But neither side can really back it up with hard facts what will happen. How do you measure a job that never existed? You dont.
Re: Why would anyone be shocked? (Score:3)
Agreed. The vast majority of economics is ideology and/or theory-induced blindness. Of the fairly small number of people who predicted the GFC (I'm not including the large number of finance people who knew mortgage's were doomed, because they didn't understand the larger consequences), only one that I'm aware of was actually an economist: Steve Keen. Not coincidentally, he is also recognized as being one of the major proponents of economic education and policy reform.
Re: (Score:2)
And many of those who successfully predicted it, made faulty investment choices in response to it! Just seeing it coming was no predictor of being able to profit from it--mainly because the whole system is just too complicated! Economists are getting deep into models based on a few questionably measured variables, hoping to predict the behavior of a system with what, quadrillions of variables that we don't even understand yet?
At this point, I've come to the conclusion that all government is illegitimate
Re: (Score:3)
the thing with 2008 isn't that 20% defaulted. it's that the banks built an insurance pyramid scheme to borrow more money and one bank was left holding the bad debt.
Bank A , B , C and D each have 100 loans of which 95% are good and 5% are bad. Each bank is maxed out on loans they can legally, and fiscally move. to make more money they need to insure the loans.
So Bank A and Bank B take 50 loans each 45 good one and the 5 bad ones and get insurance from bank C. This gives Bank A and B the ability to take o
Re: (Score:2)
the thing with 2008 isn't that 20% defaulted. it's that the banks built an insurance pyramid scheme to borrow more money and one bank was left holding the bad debt.
Bank A , B , C and D each have 100 loans of which 95% are good and 5% are bad. Each bank is maxed out on loans they can legally, and fiscally move. to make more money they need to insure the loans.
So Bank A and Bank B take 50 loans each 45 good one and the 5 bad ones and get insurance from bank C. This gives Bank A and B the ability to take on 20 more loans each with at least 2 of them being bad.
Bank C goes to Bank D for insurance and gives Bank D their 5 bad loans as well as the 10 bad loans from Banks A and B. this gives Bank C ability to also take on 20 more loans.
Bank D is now holding 190 loans including 20 bad ones and their risk went up from 5% to 10%.
Wash rinse repeat. they did this until one bank had 20% plus bad loans in their portfolio. when it collapsed all the cross insurance collapsed as well. That is why 75% of the government bailout loans were repayable in 4 months. The banks just needed to cover short term costs with the pyramid to each other.
Quite an over simplification, but it also assumes that the banks knew in advance which loans where bad - which they didn't.
One of the big pit falls for the home loans was that there were (a) a lot of loans that were themselves loans for other loans (Jumbo Mortgages, Subprime lending), and (b) there were a lot of loans with variable interest rates which were then defaulted on when the Federal Reserve attempted to raise the Interest Rates; inevitably there was quite a few of 'a' in 'b', and the various ris
Re: (Score:2)
Simplification yes. Banks didn't know exactly which loans would go bad. However those insurance pools had to have loans that were low risk and loans that were high risk. Something the banks did know. If you have 100 loans of which 40 are low risk 50 are led risk and 10 are high risk. That's. 10% chance that a loan will go bad in that pool. It might not be an actual high risk loan that is bad but that is besides the point.
Re: (Score:2)
I was contracting at GMAC-RFC (the home mortgage arm of General Motors) when the crash started.
I was working on software models to predict what would happen with low-quality mortgages. There was certainly an effort to figure out which mortgages were likely to be bad and which weren't. I didn't do the verification, but it looked good to the people that did (in a statistical way; all we could do was figure out probabilities). Personally, I wondered why there were any mortgages with "stated income" and "
Re: (Score:2)
> As someone who has a degree in this goop. I can safely say most of it is bunk. When you show me how to measure 'happiness' and 'utility' (real variables in economics) will be the day I think you are onto something.
I'd be very interested where you believe the flaw lies in standard microeconomics. I'm sure you know that the theory pretty much excludes the ability to measure utility directly... so, that's not a flaw in the theory. Like you suggest it is... a flaw in your understanding maybe?
I mean, do you
Re: (Score:2)
Don't microeconomic demand curves measure utility? If I might or might not spend $100 on a widget, the widget has very approximately the same utility to me as $100. They aren't a great measure, since it's hard to figure out what a demand curve is.
We have observed minimum wage changes in the past, and their effects, so there's some guidance. One source thought it likely that a raise in the US to $15/hour would slightly reduce employment, would mean the lowest quintile would get more purchasing power, a
Re: (Score:2)
Re:Why would anyone be shocked? (Score:5, Insightful)
But at the same time the Austrian School (not that I subscribe to them) gets dismissed for not relying on studies and poorly supported models.
That's a bit of an understatement, don't ya think? It's not that they don't "rely on studies" and "models" -- they actually fundamentally deny the possibility of empirical study! To wit [wikipedia.org]:
Mises stated that praxeology could be used to deduce a priori theoretical economic truths and that deductive economic thought experiments could yield conclusions which follow irrefutably from the underlying assumptions. He claimed conclusions could not be inferred from empirical observation or statistical analysis and argued against the use of probabilities in economic models.
I'll give you that the field of economics is a mess. But the "Austrian School" denies the fundamental existence of SCIENCE. They believe that one can just "make a priori assumptions" and do "thought experiments" and they will always get the right answer. But -- if they are honest about it -- they have no way of ever proving themselves right, because they deny the ability of using empirical data to support any argument.
That may be a good methodology for a theoretical system of pure logic or for a religion, but if you deny empiricism, then there's simply no way your system could have any contact with the real world!
Re:Why would anyone be shocked? (Score:5, Insightful)
As opposed to the likes of the Chicago and Austrian school who yell "Austerity!" and "Inflation!" with no evidence to back them up?
Re: Why would anyone be shocked? (Score:5, Interesting)
Hell not only do they lack evidence they even use their own custom definition of inflation as "increased money supply". The proper definition is "decreased buying power". Those things do not always correlate. A lot of inflation happens in static currencies as well for example. By their definition it is true that austerity prevents inflation but also utterly meaningless since thats just what they defined the word to mean. However they want you to assume the connotations of the proper definition still apply (it does not). Increased money supply does not have to equal decreased buying power. Especially if it is offset by taxes (which they ignore).
So the claim is nothing but a ruse intended to drive a political ideology. It has absolutely no bearing on useful economic analysis.
As an aside ignoring taxes is stupid because austerity reduces tax revenue and provably it always does so by several orders of magnitude more than it saves in expenses. Austerity can, as baseline mathematics, never ever achieve anythiny except to make the deficit much larger much faster.
Re: (Score:2)
Re: (Score:3)
Hell not only do they lack evidence they even use their own custom definition of inflation as "increased money supply". The proper definition is "decreased buying power".
That's because "decreased buying power" is a conflation of a huge number of possible factors, which makes any analysis of "inflation" by your definition meaningless. Regardless of whether you call it "inflation" or something else, the only factor that matters is the change in the money supply.
Re: (Score:2)
Big deal, I was saying in 2004 that the real estate market was way overheated and bound to crash before too long and I'm not even an economist.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Economists (Score:2)
"If all the economists were laid end to end, they'd never reach a conclusion." - George Bernard Shaw
"Reproducibility: Don't cry wolf" (Score:2)
If- (Score:2)
You took all the economists in the world, and laid them end to end, they'd point in different directions.
Wait a minute. (Score:3)
Re:Wait a minute. (Score:5, Interesting)
Yes but the peer review system is flawed. Take someone that is pretty high up in their field and they have their research, people to manage, and they are probably at a university so some classes too. Add in all of their day to day home life stuff that needs to be done. And now they are asked to review a paper from someone else for free. Even if they want to do a great job on it they are on a tight deadline from the publisher. So how thorough of a job do you think that they can do? Can they try to replicate the experiment? Look for an error in an Excel spreadsheet?
It's like what a lot of software shops I've seen do with their testing. The development team races to finish it and gets the application done the day before release and hands it off to QA. Then everyone wonders why the program is full of errors in production.
If you want a better peer review system you are going to have to give the reviewers more time and pay them for it.
Re: (Score:2)
Well, it's one of the sorts of errors peer-review is supposed to catch. And I guess that in this case it eventually did. Compare this to free-market idealism which is also supposed to be caught (there is no free market, there never has been a free market, and it wouldn't even be meta-stable if it existed for an hour somewhere). But this is wiggled around by using terms that are not subject to observation or test. Peer-review is supposed to catch this kind of thing too, but it hasn't in the last 200 year
Lies, damned lies, and statistics (Score:4, Interesting)
I did a brief course in cognitive psychology during my masters. The course was given by a fairly well known name in the field - an editor of one of the standard texts.
He specifically told us that we were to do 'anything we liked' to get our data to say what we wanted. He told us that it was vastly more important to publish and defend than not and get sacked. Very much a "it's easier to ask forgiveness than it is to get permission" sort of atmosphere.
It wouldn't surprise me that this sort of attitude is rampant in other areas of 'science'.
Only the hard sciences seem to have any real legitimacy and even then I wouldn't trust a biologist all that much.
The level of trust I'd give to any statement by someone working in a given area is directly proportional to the area's 'purity': http://xkcd.com/435/
Re: (Score:2)
I don't trust physics. How could you trust a science that solves a problem by just throwing "dark" in front of the name?
Re: (Score:2)
Well, you are right not to trust the physics you get from the newspaper. Given your comment, I presume that's where you've been getting your physics.
Re: (Score:2)
Actually it was play on a joke I heard on the Infinite Monkey Cage podcast made by a chemist.
Re: (Score:2)
If you talk to a decent physicist they'll blank admit that they don't know what the "dark" things are. They've got theories, but non of them have been successfully tested. (They should be as honest about gravitons.)
Re:Lies, damned lies, and statistics (Score:5, Insightful)
Only the hard sciences seem to have any real legitimacy and even then I wouldn't trust a biologist all that much.
I got started in life as an Engineer (3rd or 4th generation, as far as I can tell), and became a Biologist. One of the first things that shocked me is the notion of noise. In Electrical Engineering, noise is well-managed and understood. When you say you have a good fit to your data, it means errors of less than 1%. In Biology, there's so much noise and inherent variability that when you say you have a good fit, it means errors of less than 50%.
There are very few biological processes we understand well enough to say that we really, deeply understand them. Unlike, say, a transistor.
Don't start tugging at this thread... (Score:2)
...or the entire sweater of our existence will come unraveled.
These aren't failures to replicate (Score:5, Informative)
This paper just finds that in many cases when journals require that replication files be posted, they aren't. Of the half of studies that aren't "replicated", the majority are due to the fact that replication files simply aren't available. It's not like these people read the papers, got the data, and reran the analysis on their own. All this paper is saying is that posted replication files often either don't exist or don't work. Their work doesn't show that the results can't be replicated, just that they can't be replicated from public code.
old warning that should be repeated over and over (Score:5, Interesting)
http://www.nytimes.com/2013/04/19/opinion/krugman-the-excel-depression.html
http://www.washingtonpost.com/news/wonkblog/wp/2013/04/16/is-the-best-evidence-for-austerity-based-on-an-excel-spreadsheet-error/
First of all, shame on authors for either not checking their models enough, not asking others to check them, and not opening their models for others to see before publishing "important" results.
Secondly, and perhaps more importantly, shame on the rest of us (and especially policymakers) for relying on such kinds of work so quickly and without validation to support generally political agendas. It's almost the equivalent of funding vaccine-skeptic studies by choosing which doctors will speak in your favor without regard to a rigorous scientific review process.
Re: (Score:2, Insightful)
"The only danger of deficit spending is the risk of inflation; that managed, there is absolutely no issue."
Right. "Managed"... This is up there with fairies and unicorns in that it will never happen in the real world.
Well duh (Score:2, Interesting)
I majored in Economics, an area of study with a huge problem with relying on mathematical formula without worrying about how accurately the formula reproduce reality. I'm honestly surprised even half are reproducible. The honest opinion of many professors therein is "pfft, the numbers agree with themselves perfectly! That's all that matters."
Every paper must come with the code and data, but (Score:4, Interesting)
The solution is pretty simple. Every author must reveal the codes that were used to produce the results.
However, one interesting issue is that once every author reveals the codes, you could find out that half of them code in MATLAB with a proficiency comparable to a 3rd grader who just learned BASIC programming up to about the "goto" statement. Not only there is lots of spaghetti code, but it may also contain serious errors that may filter through into the papers. Hence, I suspect a lot of people will not be happy to reveal their codes.
Re: (Score:2)
That's just a small area of economic research. A lot of studies work with publicly accessible data after it had been 'massaged' one way or another. That's because typical census/survey data sets come in multi-gigabyte mess of numbers, while a typical research question needs to operate on a small subset or a transformation of that subset. How the researcher gets from the raw survey data file down to the data set he is analyzing is not always clear. Also, the statistical codes, which are not always trivial.
The economic theory of dead chickens .. (Score:2)
Liar's Poker [amazon.com] by Michael Lewis
Complain, but don't actually do anything about it. (Score:3)
"...With cancer studies and most recently psychology studies all having replication trouble, these economics papers have some company."
Unless we're actually going to institute some sort of reform when it comes to peer review and documented result validation, the only "company" these papers will be in is an endless sea of ignorance that assumes it won't keep happening over and over again.
There is far too much money to be made in simply publishing papers (a.k.a. bullshit) to actually validate results. Therefore, a whole new breed of fucking liars (sorry, no other words suffice) has been manipulating policy for quite a long time now.
Now society, the onus is on you. What are you going to do about this issue? Sit around and wait for it to happen again? That's what you did the last dozen times.
Re: (Score:2)
You are correct, the scientific validity of a paper does not rest in where it is published or who has reviewed it... but simply in its repeatability.
It should probably be made part of degree progression (somewhere between honors and doctorate) to show or not show the repeatability of existing papers... and only those papers that are shown to be repeatable become part of the accepted knowledge, the unrepeated papers to be considered speculative at best, and the unrepeatable to be abandoned.
IANAE (Score:3)
I am an econometrician (well sort of), which is probably worse, but at least we know that. But economics, independent of any data set availability or actual method problems, is broadly handicapped by the generally unobservable nature of the actual data that would enable the verification (or refutation) of a hypothesis. That is, much of the data is quite noisy with many variables mixed in with each other, and as such a big part of the work is trying to determine the extent to which the data itself is a useful measure of the thing being tested. Sometimes getting to a useful dataset is dependent on some awkward assumptions. As such, one of the biggest faults of Economic Theory is assuming a can opener (https://en.wikipedia.org/wiki/Assume_a_can_opener).
Margin of error is vastly underestimated (Score:3)
For most studies of any kind, the margin for error is around +/-3%. For example, a study covering the United States population using a sample size of 1000 will yield a margin of error of 3.1% [americanre...hgroup.com].
So a study says that texting while driving increases your risk of a fatal crash by 23 times [trw.com]. That sounds like a lot! But hold the phone...the overall rate of traffic fatalities is about 10 per 100,000 people [wikipedia.org], or about 0.01%. Multiply that by 23, and you get 0.23%. A big change, right? But that 0.23% is still well within the margin for error of most any study.
I'm not saying that texting while driving isn't dangerous. I'm just saying it's a lot harder to prove a link than it would seem.
Still, people are wowed by big multipliers, and news writers love to tout dramatic statistics, whether the subject of the study is economics, medicine, or traffic safety. But if you understand statistics, you know that most of these studies don't really tell us much. It's no wonder we keep getting contradictory study results!
Goodhart's law ion action? (Score:2)
Not a science ... (Score:2)
I don't care what anybody says, economics isn't a science, and can't be a science ... because far too much about how you interpret and use economics is determined by how you ideologically believe it should work.
Economics in large part is bad math, with unfounded assumptions, making hand-waving conclusions about something which happened (or you believe should happen) to explain it according how you need it to be explained to match your world view.
Economics is not and never can be an objective science.
Re: (Score:2)
You can stick your head in the ground and ignore economics but since economics affects everything in your life, you do best to pay attention to it.
Re: (Score:2)
I know the scientific method is how you investigate stuff ... I also know economics is pretty much 50% ideology, which means it's wrapping itself up in the claims of being a science while not really being one.
Yes, economics affects our lives ... and in terms of telling us what has historically happened, it has some uses ... and then it falls to shit in terms of being either predictive of what will happen, or being successful in telling us what we should do to achieve an outcome.
But as far as being an object
Social Science Isn't (Score:2)
Science in which experimental controls are illegal, isn't.
Experiments conducted on unwilling human subjects isn't ethical.
All of the social "sciences" are unethical non-sciences.
The solution?
Sort proponents of social theories into governments that test them. [sortocracy.org]
Re: (Score:3)
If you think all economics is Republican, it's evidence that you haven't been paying attention. Of course, the rest of economics is no more reliable than the part you've noticed, but there *are* other schools of economics. There are even Marxian economists.
Re: (Score:3)
"So, the fact that Western economies make life longer...
Get your head out of your ass."
Exactly the kind of thing being talked about here: on these kind of topics people let them go by their gut feelings, even when hard data can be easily found.
Life expectancy at birth, years 1985-1995*1:
USA | Cuba
1985 74.56 | 76.34
1986 74.61 | 76.43
1987 74.77 | 76.34
1988 74.77 | 76.49
1989 75.02 | 76.53
1990 75.21 | 76.53
1991 75.37 | 76.59
1992 75.64 | 76.65
1993 75.42 | 76.65
1994 75.57 |
Re: (Score:2)
Re: (Score:2)
+1