Respect your elders: Fads, fashions, and folderol in psychology - Dunnette (1966)

Some reflections on novelty in psychological science

In the discussion on open data I commented on recently results were reported on data sharing:

Because the authors were writing in APA journals and PLoS One, respectively, they had agreed at the time of submitting that they would share their data according to the journals' policies. But only 26 % and 10 %, respectively, did. (I got the references from a paper by Peter Götzsche, there may be others of which I am unaware.

Yes, there are other studies, interestingly, in the historical record: plus ça change, plus c'est la même chose.

To stress the importance of efforts to change these statistics, an excerpt from Dunnette (1966) who reports a 1962 study found 13.5% authors complied to data requests. Reasons for being unable to comply with a request sound familiar, this is not an issue of "modern" science it seems. (I can recommend the entire article)

THE SECRETS WE KEEP  

We might better label this game "Dear God,  Please Don't Tell Anyone." As the name implies, it incorporates all the things we do to accomplish the aim of looking better in public than we really are. The most common variant is, of course, the tendency to bury negative results.  

I only recently became aware of the massive size of this great graveyard for dead studies when a colleague ex- pressed gratification that only a third of his studies "turned out"—as he put it. 

Recently, a second variant of this secrecy game was discovered, quite inadvertently, by Wolins (1962) when he wrote to 37 authors to ask. for the raw data on which they had based recent journal articles. 

Wolins found that of 32 who replied, 21 reported their data to be either misplaced, lost, or inadvertently destroyed. Finally, after some negotiation, Wolins was able to complete seven re-analyses on the data supplied from 5 authors. 

Of the seven, he found gross errors in three—errors so great as to clearly change the outcome of the results already reported. Thus, if we are to accept these results from Wolins' sampling, we might expect that as many as one-third of the studies in our journals contain gross miscalculations."

30% gross miscalculations might have been a high estimate, but as a 50 year prospective prediction it's not bad: Bakker & Wicherts (2011) found "number of articles with gross errors" across 3 high and 3 low impact journals ranging from 9% to 27.6% 

In the light of these (and other) historical facts & figures, maybe its time for a historical study, lots of recommendations in those publications. 

Again Dunnette (1966):

THE CAUSES
[…]
When viewed against the backdrop of publication pressures prevailing in academia, the lure of large-scale support from Federal agencies, and the presumed necessity to become "visible" among one's colleagues, the insecurities of undertaking research on important questions in possibly untapped and unfamiliar areas become even more apparent. 

THE REMEDY 

[…]
1. Give up constraining commitments to theories, methods, and apparatus!
2. Adopt methods of multiple working hypotheses!
3. Put more eclecticism into graduate education!
4. Press for new values and less pre-tense in the academic environments of our universities!
5. Get to the editors of our psychological journals! 

THE OUTCOME: UTOPIA  

How do I envision the eventual outcome if all these recommendations were to come to pass? What would the psychologizing of the future look like and what would psychologists be up to? Chief among the outcomes, I expect, would be a marked lessening of tensions and disputes among the Great Men of our field.
I would hope that we might once again witness the emergence of an honest community of scholars all engaged in the zestful enterprise of trying to describe, understand, predict, and control human behavior.

Did I already do my déjà vu joke?

References

Bakker, M., & Wicherts, J. M. (2011). The (mis)reporting of statistical results in psychology journals. Behavior research methods, 43(3), 666–78. doi:10.3758/s13428-011-0089-5

Dunnette, M. D. (1966). Fads, fashions, and folderol in psychology. The American psychologist, 21(4), 343–52. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/5910065

Wolins, L. (1962). Responsibility for raw data. The American Psychologist, 17, 657-658. doi: 10.1037/h0038819

天

Truths, Glorified Truths and Statistics (I)

(part 1: Just for the record)

The Appendix should probably be skipped by anyone who reads this

[Just for the record] {

I did not. 

Engage in p-hacking, or any other exploitation of researchers degrees of freedom.

(ok, maybe once, but I did not inhale, or have any relations with the degrees, or the freedoms involved. None that are worth mentioning, or have been caught on tape anyway: See point 1 of the Appendix below)

Some would have us believe that we all studied Cohen, but did not act appropriately, just ignored it all in our daily  practice of science (this was almost literally exclaimed at some point during this very interesting symposium). 

I do not understand how such a thing can happen to a scientist, it appears to me as a post-meditated case of pathological science, or was it just a little sloppy and careless? When you learn about something that should be implemented immediately, then why don't you? Or: Who else will? There is no scientist high council that will decide such things for you.

On the other hand, maybe Cohen was studied very well, as evidenced by the conclusion of the paper entitled What I have learned (so far): "Finally, I have learned that there is no royal road to statistical induction, that the informed judgment of the investigator is the crucial element in the interpretation of data, and that things take time."

Cohen makes a very serious error against formal theory evaluation, but he is in good company, as this is the most common flaw in theory evaluation as it is practiced by the social sciences. In a genuine science, the informed judgement of the investigator plays NO role whatsoever in the evaluation of the accuracy of the prediction by a theory. Quantum physical theories are the best scientific theories ever produced by human minds and there are over 20 informed judgements on how the theory should be interpreted, but that does not have any influence on the empirical accuracy of the theory: highest ever!

Something that I'm picking up in how people are talking about this worries me. There seems to be a tendency to spin all the wrongdoing of the past  as a necessary evil that was inescapable. As if to say: Forgive our ignorance, let's show some penance and go about our business as usual.

I'm not bringing this up because I feel it does not apply to me personally: It is just not true. 

A scientist can never feign ignorance about his or her theorising about the way the universe works. It's either the best and most thorough and profound thinking you can possibly achieve, or it is not solid enough to share with other scientists.

Moreover, what about all those scholars who:

- have spoken out against questionable research practices in the past. 

- argued against the reluctance of scientists to abide by the rules of the scientific method

- out of sheer frustration gave up because their colleagues would not accept falsification in the face of anomalies

- criticised our preferred model of inference, or pointed out those NHST rules are not obeyed at all.

- complained about the logical inconsistencies in psychological theorising and the lack of a proper foundations debate.

To claim ignorance about these matters is at least disrespectful to those who dared to speak out, often at the risk of being marginalised and ridiculed for doing so. I believe it is more than disrespectful and find the idea there could be some kind of cleansing p-hack penance waiting to happen just outrageous.

To whom this may concern: You did not listen, and you should have!

That is what happened, you did not bother to spend time and energy to be educated on important matters of philosophy, mathematics, measurement theory, statistics or whichever discipline of science is somewhat relevant to help you answer your research questions.

Science is not: "That with which you can get away with in peer review." It is about doing everything in your power to get it as right as inhumanly possible and we should not settle for anything less. The point is lucidly made here, this will take time and should bring down the number of studies published. There is no excuse for not being on top of all the most relevant developments from all disciplines of science that could potentially help you get closer to answering the research questions you have.

So let me be clear: There will be no feigning of ignorance tolerated on my watch.

To summarise:

I did not have a life before p-hacking.

}

-------------------------------

[Appendix] {

Want proof?

Of course you do, you're the proud owner of scientific mind!

1. I have not published a single paper in a peer-reviewed journal as a first author before 2013. It just took me a long time to find out exactly what it was I could contribute 

(note: this usually has nothing to do with the importance of those thoughts as perceived by others)

2. Before 2013, I submitted a paper as first author only twice, but they did concern the same study. First journal, they loved the theory, but not the experimental design, so it was rejected. Then I revised it and submitted it to another journal. They saw merit and wanted me to resubmit, again, they loved the theory, but asked me if I could lose 66% of the words I had used. That pretty much settled it. 

(I will not relate here all the encouraging advice I received over the years to become less precise, engage more often in the practice of “huis tuin en keuken” science [probably translates to “middle of the road science”], or to “just send it in and see what reviewers say, because you never know in advance what they will say, they will be pissed off because you cite work that is over 2 years old anyway. Here’s a list of 10 journals, start at the top”)

3. Even so, I have a decent number of publications to which I made substantial contributions either in study design or by performing the data analysis or even the theoretical part, imagine that! I disseminate the work that I do not publish and even teach about it and this is the best way to learn about all the things that I still need to be educated on. Such a resume will not impress any research institute or funding agencies. Thank the goddess I have a permanent teaching job. 

("oh, one of those guys who can only teach and does not know how to write a proper scientific paper")

4. I did not defend my dissertation until I could 100% stand behind every word I wrote.

(but that was already the case more than 5 years ago and still hasn't happened)

Almost, just awaiting some additional results. 

I did postpone, yes, mainly because I seriously considered leaving science, until about a year ago. Things have changed recently as you may have noticed. 

Before the change, I wanted to leave because I realised that a game was being played in which the winners were the ones who interpreted the "facts" of their scientific inquiries in such a way that it would maximally serve their own cause instead of the cause of science, which is to uncover the structure of reality. Decisions about funding, positions, courses in the curriculum, they are not based on quality, but on politics. Good luck with that strategy. 

I have seen too many gifted young students who understood this was the game they were supposed to be playing if they wanted to become a scientist and therefore, could not be saved for science.

If I had wanted to be engaged in an endeavour that interpreted facts any way the wind blows, I would have chosen a career in politics or finance and would have made a much better living out of it in the process. Science is for nerds who want to figure things out, not for bullies who take over the playing ground by loudly shouting out incoherent authoritative arguments to prove they are never wrong about anything.

}

天