Truths, Glorified Truths, and Statistics (III): The power-credibility paradox of empirical phenomena observed in psychological experiments

The power-credibility paradox of empirical phenomena observed in psychological experiments

I started this post in response to an e-mail exchange at openscienceframework@googlegroups.com on the necessity of reporting multiple experiments in one session, which I take to mean one participant.

Joshua Hartshorne wrote:

My concern is that the conversation on this list is often focused on solving the problems of social psychology and then imposing the solutions on other fields,

In our lab, which for the most part is buzzing with social psychological experiments, as a researcher you can opt to run your experiment in a "chunk": One participant gets to press buttons in a cubicle for an hour or so, and is rewarded with a nice round number in course credit. The chunk usually consists of 3-4, 15-20 minute experiments. I often collect more than a thousand data points per participant, so my experiments are not "chunkable". The majority of studies are, and will be conducted this way. I hear other labs use the same system. In a publication I have seen people report something along the lines of: "The data were collected in a 1-hour session in which the participant performed tasks in addition, but unrelated to the present experiment."

Running many manipulations and only reporting the one that "works" is a problem and needs to be discouraged. But we wouldn't want the method of discouragement to result in discouraging the running of multiple EEG experiments in the same session, which is actually a very good idea!

The chunk seems comparable to the multi EEG and fMRI experiments in one session, except for one thing… 

I don't know about fMRI, but having several unrelated experiments in one EEG session is pretty common and certainly not something we want to discourage. Good EEG experiments require that your critical trials be only a small fraction of all trials. The rest can be random filler, but it's more efficient to simply run several studies, with each one using the others as fillers.

… The question is whether unrelated experiments of other researchers are used as fillers in a single EEG / fMRI session, or whether all the experiments run are in fact experiments designed, and / or intended to publish about by the same research group, possibly the same phenomenon?

My guess is the latter will be true in the majority of cases and I will argue that that is not a good idea and there is a lot of cause to impose the solutions of other fields, like the ones suggested for behavioural studies in social psychology, to solve the problems with this particular practice.

First, I very much agree with this important remark by Gustav Nilsonne:

Do you agree that the publications from an EEG experiment should ideally mention what the "filler" experiments were, and which other papers (if any) were based on data from the same experimental runs?

I would add that this should be the case for all experiments measured from the same participant, at the same location for the duration of chunked measurements.

To pre-emptively tackle responses about the feasibility of such an accounting system, Yes: hard, difficult, time, energy, money, ethical concerns, anonymity, but reality is not affected by those matters and neither should your scientific claims about reality be if these issues can in principle be resolved.

(there's much more possible if you let fantasies run wild: If a smart system of token generation and data storage is in place that guarantees anonymity, a participant could be linked to the raw datasets of all the studies they participated in. How many publications would they have contributed to? Are they always the outlier that makes the small effect a moderate one? This potentially provides an interesting window on assessing magnitude of individual variability in different measurement paradigms, perhaps even estimation of compliance and experiment experience / expectation effects)

Back to reality, Gustav wrote: 

In my area this is a real issue. I see fMRI papers from time to time that report one experiment out of several that were performed in the same scanning session, with no mention of the other experiments. Of course there is no way to know in such a case whether there might have been carryover effects from an earlier experiment. This seems to be a fully accepted practice in the fMRI field.

 Carryover effects may be a problem, but what about power and the credibility hurdle of phenomena?

(note: assuming NHST as model for scientific inference)

Suppose you publish a multi-experiment article that concerns close replications of an interesting phenomenon observed in experiment 1. Why do you publish multiple experiments? One reason is to assert credibility for the phenomenon, say at 5 significant independent replications p < .05 you'd be quite convinced there is a structure you bumped into. Then the prob. of making a type I error effectively reduces to .05^5 = .000000312 (Schimmack, 2012). Apparently this is approximately the credibility level used for phenomena of particle physics. What about type II errors? 

This argument was also made recently in the Power Failure article (Button et al., 2013): To maintain an equal power for each sig. replication due to the decrease of effective alpha of the total, you need to increase sample size for each study that adds to the credibility of the phenomenon (in physics you would use / invent a more precise / accurate measurement procedure. Note that this is also a valid strategy for psychology to increase power, just one that is rarely used).

Schimmack (2012) provides a table with the total power of the multiple-experiment study necessary to maintain 80% power/study for large, moderate and small effect sizes, here's an excerpt:

N experiments	total Power needed	Large (d=.8)	Moderate (d=.5)	Small (d=.2)
1	80.0	52	128	788
2	89.4	136	336	2068
5	95.6	440	1090	6750
10	97.8	1020	2560	15820

Now, suppose I want to publish an article about the credibility of observing a phenomenon in our lab by means of chunked experiments. I'll use all the experiments in a randomly selected chunk that was run in our lab to investigate this question, so it's gonna be a multi-experiment article. 

Each experiment in the chunk can provide credibility to the phenomenon of observing chunked phenomena in our lab if whatever effect was examined in the experiment is observed at p < .05.

We have an amazing lab, so all the experiments in the chunk worked out and everyone used G*Power to calculate N needed to detect their well know Effect Sizes. Of course, I am post-hoc evaluating, so I cannot adjust the sample size anymore. Here's what happens to the power of my study to detect whether my lab actually detects phenomena in a chunk if I report an increasing number of successful chunk experiments:

N Sig. chunk	total Power	N for Large	N for Moderate	N for Small
1	80 – 81	52	128	788
2	71 – 72	52	128	788
5	0.5 – 0.9	52	128	788
10	0	52	128	788

Why does this happen?

Because it becomes increasingly unlikely to observe n+1 significant effects out of N attempts to observe the phenomenon at the total power level. The probability of observing 5 significant results in 5 studies whose total power is 50% is 0.0313. So in 3 out of 100 five-experiment studies of the same power, we would expect to see 5 significant results. 

This is the "hurdle" for the credibility of a phenomenon predicted by a theory that needs to be adjusted in order for the phenomenon to maintain its credibility (see also Meehl, 1967).

(In physics this hurdle is even more difficult due to requirements of predicting actual measurement outcomes)

Schimmack (2012) calculates an Incredibility Index (IC-index) as the binomial probability of observing a non-significant result given the observed power to detect an effect, which in this example would simply be 1-total power = 96.9%. That's how incredible my results would be if every effect turned out to be significant. 

Paradoxically, or whatever logic-defying concept applies here, in this case it may not be that bad for science, it's just bad for the phenomenon I am interested in, which is just too incredible to be true. The individual phenomena of the studies in the chunk are likely the result of independent tests of effects predicted by different theories (strictly they are not independent measurements of course). The individual observations could still end up in a very credible multi-study article that contains a lot of nonsignificant results.

Back to the EEG / fMRI filler conditions... it seems much more likely in these cases that the conditions cannot be regarded as independently studied phenomena, as is the case with the chunked experiments querying independent psychological phenomena within the same participant. 

More importantly, suppose the results of 3 conditions that measure different aspects of the same phenomenon measured in one session are published in 3 separate papers (effect of bigram, trigram and quadgram frequency on p123) shouldn't we be worried about increasing the credibility hurdle for each subsequent observation of the phenomenon?

My personal opinion is that we need a (new) measurement theory for psychological phenomena, but that's another story.

天


References

Button, K. S., Ioannidis, J. P. a., Mokrysz, C., Nosek, B. a., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(May). doi:10.1038/nrn3475

Meehl, P. E. (1967). Theory testing in psychology and physics: a methodological paradox. Philosophy of science, 34, 103–115. Retrieved from http://www.jstor.org/stable/10.2307/186099

Schimmack, U. (2012). The ironic effect of significant results on the credibility of multiple-study articles. Psychological methods, 17(4), 551–66. doi:10.1037/a0029487

Truths, Glorified Truths and Statistics (II)

(part 2: "To boldly, go...")

First a disclaimer: I love the work on the p-curve and the estimation of effect sizes. I support the disclosure initiatives (4 questions, 21 words) and the call for more quality and less quantity (however, also see part 1 in which I remind the reader there are many scientists for which there has been no life before p-hacking and a claim of ignorance on these matters is at the very least disrespectful to these scholars)

Let me be the one to spoil all the fun: There is no true effect!

It does not exist as an entity in reality, it is not one of the constituents of the universe, it should be a measurement outcome observed in a measurement context that was predicted by by a theory about a specific domain in reality.

As was pointed out by Klaus Fiedler at the Solid Science Symposium: "What does it mean there is an effect?" (I am quoting from memory, this may be incorrect)

According to the live tweet feed earlier that day:

Solid Science Symposium Tweet Feed - Excellent!

If you believe this is possible, that a true effect can somehow be discovered, out there in reality, like a land mass across the ocean where everyone said there would be dragons, or a new species of silicon based life forms at the other end of the worm-hole, then you show one of the symptoms of participating in a failing system of theory evaluation and revision that I dubbed the [intergalactic] explorer delusion. 

This refers the to the belief expressed by many experimental psychological scientists that the purpose of scientific inquiry is to go where no man has gone before and observe the phenomena that are “out there” in reality waiting to be uncovered by clever experimental manipulation and perhaps some more arbitrary poking about as well. 

A laboratory experiment is however not a field study or an excursion beyond the neutral zone. Even if it were, I would argue that wherever you go as a scientist, boldly, or otherwise, you will be guided and quite possible even be blinded, by a theory or a mathematical formalism about reality that is in most cases implicitly present in your theorising.

Let's analyse this delusion by scrutinising a recent paper by Greenwald (2012) entitled: “There is nothing so theoretical as a good method”, which is a reference to the famous quote by a giant of psychological science, Kurt Lewin (1951). This also allows me to comment on what it actually is that Platt meant to say by the term "strong inference" in his 1964 paper.

Greenwald is explicit about his position towards theory; he is not anti-theoretic, as he acknowledges that theories achieve parsimonious understanding and guide useful applications (but he does not specify… of what?). The author is however also skeptical of theory, because he noticed the ability of theory to restrict open-mindedness. This is indeed a proper description of a theory: It is a specific tunnel-vision, but from the perspective of the Structural Realist (forgive me, I will explain this position more  precisely in the near future), this tunnel-vision is is only temporary.

It will be no surprise I disagree with the following: 

“When alternative theories contest the interpretation of an interesting finding, researchers are drawn like moths to flame. J. R. Platt (1964) gave the approving label “strong inference” to experiments that were designed as crucial empirical confrontations between theories that competed to explain a compellingly interesting empirical result.” (Greenwald, 2012, pp. 99–100, emphasis added)

That is not at all what Platt meant by strong inference, but incidentally we find another symptom of a failing system of theory evaluation, the interpretation fallacy I mentioned in part 1: Theories do not compete for their ability to provide an understandable description or explanation of empirical phenomena. They compete for the ability to predict measurement contexts in which phenomena may be observed and they compete for the accuracy with which measurement outcomes were predicted. And J.R. Platt agrees with this perspective as he describes very clearly:

“Strong inference consists of applying the following steps to every problem in science, formally and explicitly and regularly:

1) Devising alternative hypotheses;

2) Devising a crucial experiment (or several of them), with alternative possible outcomes, each of which will, as nearly as possible, exclude one or more of the hypotheses;

3) Carrying out the experiment so as to get a clean result;

1') Recycling the procedure, making subhypotheses or sequential hypotheses to refine the possibilities that remain; and so on.”

(Platt, 1964, p. 347, emphasis added)

Strong inference starts with devising alternative hypotheses to a problem in science and not with an interesting finding. Platt comments that step 1 and 2 require intellectual invention, which I take the liberty to translate as ‘theorizing about reality’. That is what you do when you device a method.

One source of evidence for his argument concern 13 papers, listed in a table that have started controversies on average 44 years ago in psychological science, but which still have no resolution. The author claims that in order to resolve the controversies, the method of strong inference was applied, which obviously failed. Also, it is claimed that philosophy of science provides no answers to resolve the controversies, because it discusses (apparently endlessly) whether such issues can be resolved empirically in principle. It is clear that Greenwald is referring to the resolution of these controversies as a resolution about the ‘reality’ of the ontology of a theory. This is again a matter of interpretation and is not what formal theory evaluation is about. The constituents of reality posited to exist by a theory are irrelevant in theory evaluation. As long as everything behaves according to the predictions by the theory, we should just accept those constituents as temporary vehicles for understanding. I believe these controversy theories were not properly evaluated for their predictive power and empirical accuracy. I don't know if they can be evaluated in that way, if they cannot, the conclusion must be the theories are trivial.

This impression that ontology evaluation seems to be the problem here is indeed supported by the descriptions provided for the 13 controversies: It is primarily a list of clashes of ontology, e.g., Spreading activation vs. Compound cueing. Further support comes from the examples provided to argue that even if philosophy had an answer, this would not refrain scientists to continue the debate. The fact that scientists do not do this implies to the author there must be another way than strong inference to resolve controversies in science. This is illustrated by examples in which a scientific community was able to achieve consensus about a problem in their discipline (the classification of Pluto as a dwarf planet, HIV as the cause of AIDS and the influence of human activity on global warming). The author suggests that controversies in psychology could be resolved if only a reasonable consensus could be achieved.

I cannot disagree with the author on his wish for a science that worked towards reaching consensus about the phenomena in its empirical record, instead of wasting energy on definite existence proofs for the ontologies of competing theories. Recall the history of the quantum formalism, two very different theoretical descriptions of reality (waves vs. particle ontologies) were found to be the same for all intents and purposes. I am certain that scientists in cosmology, virology and climatology used strong inference to work towards those consensus resolutions, but I did not check it. Strong inference and consensus formalism science go hand in hand.

What I can say is that Platt’s recycling procedure (step 1’) suggests replication attempts should be carried out and apparently there is somewhat of a problem with replication of phenomena in psychological science. So this makes it again very unlikely any strong inference has been applied to resolve theoretical disputes in psychological science. Indeed, one of the authors listed to have caused a controversy that was unresolved by strong inference, recently challenged the discipline to start replicating the ‘interesting findings’ in its empirical record (e.g. Yong, 2012).

(There must be some proverb about dismissing something before its merits have been properly examined...)

A second source of evidence to support his suspicions about the benefits of theorising, Greenwald examines the description of Nobel Prizes for their being rewarded due to theoretical or methodological contributions. The [intergalactic] explorer delusion is obvious here; Greenwald highly values the appearance of the word ‘discovery’: 

“Most “discovery” citations were for methods that permitted previously impossible observations, but in a minority of these, “discovery” indicated a theoretical contribution.” 

He concludes that theory was important for the development of methods, and that novel methods produced inconceivable results, that prompted new theory.

I am quite certain that the referred inconceivable results were predicted by a theory or considered as an alternative hypothesis. They concern measurement contexts one just does not accidentally stumble upon. If outcomes were surprising given the predicted context, an anomaly to the theory was found, and in that case, naturally, a new theory would have to be created. It was however due to an anomaly to a theoretical prediction, not due to a ‘discovery’ of a phenomenon by a method! The Large Hadron Collider (or any other billion-dollar instrument of modern physics) was not built as a method, a vehicle to seek out previously unknown phenomena like the starship U.S.S Enterprise. Theory, very strongly predicted a measurement context in which a boson should be observable that completed the standard model of particle physics. The methods scientists use for obtaining knowledge about the structure of reality is the result of testing predictions by theories, without exception. Satellites are not sent into space equipped with multi-million dollar X-ray detectors just to see what they will find when they get there. 

I conclude by commenting on the way the author describes why Michelson won the Nobel Prize for Physics in 1907. This involves a recurring theme in a paper I am about to submit: the luminiferous Æther. Experimental physicists like Michelson and Morley spent most of their academic careers (and most of their money) on experiments that tested the empirical accuracy of theories that predicted a very specific observable phenomenon called Æther-dragging. Their most famous experiment reported in “On the Relative Motion of the Earth and the Luminiferous Ether” (Michelson & Morley, 1887), showed very accurately and consistently that there was no such thing as an Æther, or at least, that its influence on light and matter was not as large as the Æther-dragging hypothesis predicted it would be. This of course harmed the precision and accuracy of Æther-based theories of the cosmos, but to hint, as Greenwald seems to do, that the method ‘caused’ Einstein to create special relativity theory is farfetched. 

Michelson won the Nobel Prize for Physics in 1907 for the very consistent null-result (yes psychological science, such things can be important) and for the development of the interferometer instruments that meticulously failed to measure any trace of the Æther (cf. Michelson, 1881). Their commitment to the Æther was adamant though. To be absolutely certain that the minute interferences that were occasionally measured were indeed due to measurement error, instruments of increasing accuracy and sensitivity were built. The largest were many meters wide and placed on high altitude on heavy slabs of marble floating on quicksilver in order to avoid vibrations interfering with the measurement process. Now that is a display of ontological commitment! It was however as much motivated by theoretical prediction as the construction of the Large Hadron Collider. Not a theory-less discovery by some clever poking about.

Greenwald admits that the word theory is often used in Michelson and Morley’s 1885 article, so theory must have played an important role in the design of the instruments. The role was not just 'important', without the theory there would have been no method at all. In fact, if a theory of special relativity had been published 20 years before 1905 (physicists knew something like relativity was necessary), there would have been no instruments constructed at all because:

"Whether the ether exists or not matters little - let us leave that to the metaphysicians; what is essential for us is, that everything happens as if it existed, and that this hypothesis is found to be suitable for the explanation of phenomena. After all, have we any other reason for believing in the existence of material objects? That, too, is only a convenient hypothesis; only, it will never cease to be so, while some day, no doubt, the ether will be thrown aside as useless." (Poincaré, 1889/1905, p. 211). 

And indeed, the Æther  was thrown aside as useless, because a method devised to test a prediction by a theory yielded null results. Strong inference means this repeated null-result has consequences for the credibility of the theory that predicted the phenomenon. Apparently, in psychological science, this id a difficult condition to achieve. 

The Structural Realist's take home message is: 

1. We should believe what scientific theories tell us about the structure of the unobservable world, but
2. We should be skeptical about what they tell us about the posited ontology of the unobservable world. 

In this quote by Poincaré may lie the answer to Greenwald's interpretation of current practice of psychological science (which is in fact a very accurate description of the problems we have with theory evaluation, I just do not agree with the interpretation): Why does Poincaré reserve a special place for the hypothesis about material objects, which will never cease to to be so? 

Still believe it is possible to use a method that was not predicted to yield measurement outcomes by a theory about reality? 

Ok.
I'll think of some more examples.

天

References

Greenwald, A. G. (2012). There Is Nothing So Theoretical as a Good Method. Perspectives

on Psychological Science, 7(2), 99–108. doi:10.1177/1745691611434210

Michelson, A. . (1881). The Relative Motion of the Earth and the Luminiferous Ether. American Journal of Science, 22(128), 120–129. Retrieved from http://www.archive.org/details/americanjournal62unkngoog

Michelson, A. ., & Morley, E. W. (1887). On the Relative Motion of the Earth and the Luminiferous Ether. American Journal of Science, 34(203), 333–345. Retrieved from http://www.aip.org/history/gap/PDF/michelson.pdf

Platt, J. (1964). Strong Inference. Science, 146(3642), 347–353. Retrieved from http://clustertwo.org/articles/Strong Inference (Platt).pdf

Poincaré, H. (1905). Science and Hypothesis. New York: The Walter Scott Publishing Co., LTD. Retrieved from http://www.archive.org/details/scienceandhypoth00poinuoft

Yong, E. (2012). Nobel laureate challenges psychologists to clean up their act. Nature. Retrieved from http://www.nature.com/doifinder/10.1038/nature.2012.11535

Truths, Glorified Truths and Statistics (I)

(part 1: Just for the record)

The Appendix should probably be skipped by anyone who reads this

[Just for the record] {

I did not. 

Engage in p-hacking, or any other exploitation of researchers degrees of freedom.

(ok, maybe once, but I did not inhale, or have any relations with the degrees, or the freedoms involved. None that are worth mentioning, or have been caught on tape anyway: See point 1 of the Appendix below)

Some would have us believe that we all studied Cohen, but did not act appropriately, just ignored it all in our daily  practice of science (this was almost literally exclaimed at some point during this very interesting symposium). 

I do not understand how such a thing can happen to a scientist, it appears to me as a post-meditated case of pathological science, or was it just a little sloppy and careless? When you learn about something that should be implemented immediately, then why don't you? Or: Who else will? There is no scientist high council that will decide such things for you.

On the other hand, maybe Cohen was studied very well, as evidenced by the conclusion of the paper entitled What I have learned (so far): "Finally, I have learned that there is no royal road to statistical induction, that the informed judgment of the investigator is the crucial element in the interpretation of data, and that things take time."

Cohen makes a very serious error against formal theory evaluation, but he is in good company, as this is the most common flaw in theory evaluation as it is practiced by the social sciences. In a genuine science, the informed judgement of the investigator plays NO role whatsoever in the evaluation of the accuracy of the prediction by a theory. Quantum physical theories are the best scientific theories ever produced by human minds and there are over 20 informed judgements on how the theory should be interpreted, but that does not have any influence on the empirical accuracy of the theory: highest ever!

Something that I'm picking up in how people are talking about this worries me. There seems to be a tendency to spin all the wrongdoing of the past  as a necessary evil that was inescapable. As if to say: Forgive our ignorance, let's show some penance and go about our business as usual.

I'm not bringing this up because I feel it does not apply to me personally: It is just not true. 

A scientist can never feign ignorance about his or her theorising about the way the universe works. It's either the best and most thorough and profound thinking you can possibly achieve, or it is not solid enough to share with other scientists.

Moreover, what about all those scholars who:

- have spoken out against questionable research practices in the past. 

- argued against the reluctance of scientists to abide by the rules of the scientific method

- out of sheer frustration gave up because their colleagues would not accept falsification in the face of anomalies

- criticised our preferred model of inference, or pointed out those NHST rules are not obeyed at all.

- complained about the logical inconsistencies in psychological theorising and the lack of a proper foundations debate.

To claim ignorance about these matters is at least disrespectful to those who dared to speak out, often at the risk of being marginalised and ridiculed for doing so. I believe it is more than disrespectful and find the idea there could be some kind of cleansing p-hack penance waiting to happen just outrageous.

To whom this may concern: You did not listen, and you should have!

That is what happened, you did not bother to spend time and energy to be educated on important matters of philosophy, mathematics, measurement theory, statistics or whichever discipline of science is somewhat relevant to help you answer your research questions.

Science is not: "That with which you can get away with in peer review." It is about doing everything in your power to get it as right as inhumanly possible and we should not settle for anything less. The point is lucidly made here, this will take time and should bring down the number of studies published. There is no excuse for not being on top of all the most relevant developments from all disciplines of science that could potentially help you get closer to answering the research questions you have.

So let me be clear: There will be no feigning of ignorance tolerated on my watch.

To summarise:

I did not have a life before p-hacking.

}

-------------------------------

[Appendix] {

Want proof?

Of course you do, you're the proud owner of scientific mind!

1. I have not published a single paper in a peer-reviewed journal as a first author before 2013. It just took me a long time to find out exactly what it was I could contribute 

(note: this usually has nothing to do with the importance of those thoughts as perceived by others)

2. Before 2013, I submitted a paper as first author only twice, but they did concern the same study. First journal, they loved the theory, but not the experimental design, so it was rejected. Then I revised it and submitted it to another journal. They saw merit and wanted me to resubmit, again, they loved the theory, but asked me if I could lose 66% of the words I had used. That pretty much settled it. 

(I will not relate here all the encouraging advice I received over the years to become less precise, engage more often in the practice of “huis tuin en keuken” science [probably translates to “middle of the road science”], or to “just send it in and see what reviewers say, because you never know in advance what they will say, they will be pissed off because you cite work that is over 2 years old anyway. Here’s a list of 10 journals, start at the top”)

3. Even so, I have a decent number of publications to which I made substantial contributions either in study design or by performing the data analysis or even the theoretical part, imagine that! I disseminate the work that I do not publish and even teach about it and this is the best way to learn about all the things that I still need to be educated on. Such a resume will not impress any research institute or funding agencies. Thank the goddess I have a permanent teaching job. 

("oh, one of those guys who can only teach and does not know how to write a proper scientific paper")

4. I did not defend my dissertation until I could 100% stand behind every word I wrote.

(but that was already the case more than 5 years ago and still hasn't happened)

Almost, just awaiting some additional results. 

I did postpone, yes, mainly because I seriously considered leaving science, until about a year ago. Things have changed recently as you may have noticed. 

Before the change, I wanted to leave because I realised that a game was being played in which the winners were the ones who interpreted the "facts" of their scientific inquiries in such a way that it would maximally serve their own cause instead of the cause of science, which is to uncover the structure of reality. Decisions about funding, positions, courses in the curriculum, they are not based on quality, but on politics. Good luck with that strategy. 

I have seen too many gifted young students who understood this was the game they were supposed to be playing if they wanted to become a scientist and therefore, could not be saved for science.

If I had wanted to be engaged in an endeavour that interpreted facts any way the wind blows, I would have chosen a career in politics or finance and would have made a much better living out of it in the process. Science is for nerds who want to figure things out, not for bullies who take over the playing ground by loudly shouting out incoherent authoritative arguments to prove they are never wrong about anything.

}

天