AN EXAMINATION OF ALTERNATIVES TO NULL HYPOTHESIS SIGNIFICANCE TESTING

Name: LISA DIRMEIER COTA

Date of Degree: DECEMBER, 2017

Title of Study: AN EXAMINATION OF ALTERNATIVES TO NULL HYPOTHESIS SIGNIFICANCE TESTING

Major Field: PSYCHOLOGY

Science has historically existed independently of statistical inference; since the early 1940’s, however, psychology and many other fields have become increasingly fixated on the idea of making inferences from collected data to a population parameter (Gigerenzer & Marewski, 2014).

Psychology overwhelmingly defaults to using null hypothesis significance testing (NHST) in conjunction with hypothetico-deductive inference (Haig, 2009). Bayesian statistics are an alternative to NHST. A third statistical methodology, Observation Oriented Modeling (OOM; Grice, 2011), provides researchers with the opportunity to explore and explain patterns in their observed data, and to do so at the level of the individual, rather than the aggregate.
 

This study involved a comparison of NHST, a Bayesian two-group analysis, and OOM, in order to describe the similarities and differences between the three statistical techniques. Next, simulation studies were conducted to evaluate the potential bias between sample and population parameters in OOM. Simulation studies indicated that the sample parameter in OOM could potentially over- or under-estimate the population parameter by as much as 16% with small samples (i.e., n = 30), although, on average, the difference between sample and population parameters was negligible. The over/under-estimation was reduced with larger sample sizes. If researchers want to use OOM in conjunction with estimating population parameters, samples sizes of 100+ are suggested but not mandatory.
 

NHST and Bayesian analyses specifically involve estimation of population parameters, whereas OOM is used to evaluate the accuracy of a model based on obtained data. The three methods are quite different in terms of statistics generated and the inferences which should be drawn from those statistics. The three methods are not interchangeable, and inferences from one method should not be applied automatically to analyses from another. Researchers should be mindful of their research question and choose the technique most suited for their purpose.