(part 1: Just for the record)
The Appendix should probably be skipped by anyone who reads this
[Just for the record] {
I did not.
Engage in p-hacking, or any other exploitation of researchers degrees of freedom.
(ok, maybe once, but I did not inhale, or have any relations with the degrees, or the freedoms involved. None that are worth mentioning, or have been caught on tape anyway: See point 1 of the Appendix below)
Some would have us believe that we all studied Cohen, but did not act appropriately, just ignored it all in our daily practice of science (this was almost literally exclaimed at some point during this very interesting symposium).
I do not understand how such a thing can happen to a scientist, it appears to me as a post-meditated case of pathological science, or was it just a little sloppy and careless? When you learn about something that should be implemented immediately, then why don't you? Or: Who else will? There is no scientist high council that will decide such things for you.
On the other hand, maybe Cohen was studied very well, as evidenced by the conclusion of the paper entitled What I have learned (so far): "Finally, I have learned that there is no royal road to statistical induction, that the informed judgment of the investigator is the crucial element in the interpretation of data, and that things take time."
Cohen makes a very serious error against formal theory evaluation, but he is in good company, as this is the most common flaw in theory evaluation as it is practiced by the social sciences. In a genuine science, the informed judgement of the investigator plays NO role whatsoever in the evaluation of the accuracy of the prediction by a theory. Quantum physical theories are the best scientific theories ever produced by human minds and there are over 20 informed judgements on how the theory should be interpreted, but that does not have any influence on the empirical accuracy of the theory: highest ever!
Something that I'm picking up in how people are talking about this worries me. There seems to be a tendency to spin all the wrongdoing of the past as a necessary evil that was inescapable. As if to say: Forgive our ignorance, let's show some penance and go about our business as usual.
I'm not bringing this up because I feel it does not apply to me personally: It is just not true.
A scientist can never feign ignorance about his or her theorising about the way the universe works. It's either the best and most thorough and profound thinking you can possibly achieve, or it is not solid enough to share with other scientists.
Moreover, what about all those scholars who:
- have spoken out against questionable research practices in the past.
- argued against the reluctance of scientists to abide by the rules of the scientific method
- out of sheer frustration gave up because their colleagues would not accept falsification in the face of anomalies
- criticised our preferred model of inference, or pointed out those NHST rules are not obeyed at all.
- complained about the logical inconsistencies in psychological theorising and the lack of a proper foundations debate.
To claim ignorance about these matters is at least disrespectful to those who dared to speak out, often at the risk of being marginalised and ridiculed for doing so. I believe it is more than disrespectful and find the idea there could be some kind of cleansing p-hack penance waiting to happen just outrageous.
To whom this may concern: You did not listen, and you should have!
That is what happened, you did not bother to spend time and energy to be educated on important matters of philosophy, mathematics, measurement theory, statistics or whichever discipline of science is somewhat relevant to help you answer your research questions.
Science is not: "That with which you can get away with in peer review." It is about doing everything in your power to get it as right as inhumanly possible and we should not settle for anything less. The point is lucidly made here, this will take time and should bring down the number of studies published. There is no excuse for not being on top of all the most relevant developments from all disciplines of science that could potentially help you get closer to answering the research questions you have.
So let me be clear: There will be no feigning of ignorance tolerated on my watch.
To summarise:
I did not have a life before p-hacking.
}
-------------------------------
[Appendix] {
Want proof?
Of course you do, you're the proud owner of scientific mind!
1. I have not published a single paper in a peer-reviewed journal as a first author before 2013. It just took me a long time to find out exactly what it was I could contribute
(note: this usually has nothing to do with the importance of those thoughts as perceived by others)
2. Before 2013, I submitted a paper as first author only twice, but they did concern the same study. First journal, they loved the theory, but not the experimental design, so it was rejected. Then I revised it and submitted it to another journal. They saw merit and wanted me to resubmit, again, they loved the theory, but asked me if I could lose 66% of the words I had used. That pretty much settled it.
(I will not relate here all the encouraging advice I received over the years to become less precise, engage more often in the practice of “huis tuin en keuken” science [probably translates to “middle of the road science”], or to “just send it in and see what reviewers say, because you never know in advance what they will say, they will be pissed off because you cite work that is over 2 years old anyway. Here’s a list of 10 journals, start at the top”)
3. Even so, I have a decent number of publications to which I made substantial contributions either in study design or by performing the data analysis or even the theoretical part, imagine that! I disseminate the work that I do not publish and even teach about it and this is the best way to learn about all the things that I still need to be educated on. Such a resume will not impress any research institute or funding agencies. Thank the goddess I have a permanent teaching job.
("oh, one of those guys who can only teach and does not know how to write a proper scientific paper")
4. I did not defend my dissertation until I could 100% stand behind every word I wrote.
(but that was already the case more than 5 years ago and still hasn't happened)
Almost, just awaiting some additional results.
I did postpone, yes, mainly because I seriously considered leaving science, until about a year ago. Things have changed recently as you may have noticed.
Before the change, I wanted to leave because I realised that a game was being played in which the winners were the ones who interpreted the "facts" of their scientific inquiries in such a way that it would maximally serve their own cause instead of the cause of science, which is to uncover the structure of reality. Decisions about funding, positions, courses in the curriculum, they are not based on quality, but on politics. Good luck with that strategy.
I have seen too many gifted young students who understood this was the game they were supposed to be playing if they wanted to become a scientist and therefore, could not be saved for science.
If I had wanted to be engaged in an endeavour that interpreted facts any way the wind blows, I would have chosen a career in politics or finance and would have made a much better living out of it in the process. Science is for nerds who want to figure things out, not for bullies who take over the playing ground by loudly shouting out incoherent authoritative arguments to prove they are never wrong about anything.
}
天
