AN EXAMINATION OF ALTERNATIVES TO NULL HYPOTHESIS SIGNIFICANCE TESTING

Name: LISA DIRMEIER COTA

Date of Degree: DECEMBER, 2017

Title of Study: AN EXAMINATION OF ALTERNATIVES TO NULL HYPOTHESIS SIGNIFICANCE
TESTING

Major Field: PSYCHOLOGY

Science has historically existed independently of statistical inference; since
the early 1940’s, however, psychology and many other fields have become
increasingly fixated on the idea of making inferences from collected data to a
population parameter (Gigerenzer & Marewski, 2014).

Psychology overwhelmingly defaults to using null hypothesis significance testing
(NHST) in conjunction with hypothetico-deductive inference (Haig, 2009).
Bayesian statistics are an alternative to NHST. A third statistical methodology,
Observation Oriented Modeling (OOM; Grice, 2011), provides researchers with the
opportunity to explore and explain patterns in their observed data, and to do so
at the level of the individual, rather than the aggregate.

This study involved a comparison of NHST, a Bayesian two-group analysis, and
OOM, in order to describe the similarities and differences between the three
statistical techniques. Next, simulation studies were conducted to evaluate the
potential bias between sample and population parameters in OOM. Simulation
studies indicated that the sample parameter in OOM could potentially over- or
under-estimate the population parameter by as much as 16% with small samples
(i.e., n = 30), although, on average, the difference between sample and
population parameters was negligible. The over/under-estimation was reduced with
larger sample sizes. If researchers want to use OOM in conjunction with
estimating population parameters, samples sizes of 100+ are suggested but not
mandatory.

NHST and Bayesian analyses specifically involve estimation of population parameters, whereas OOM is used to evaluate the accuracy of a model based on obtained data. The three methods are quite different in terms of statistics generated and the inferences which should be drawn from those statistics. The three methods are not interchangeable, and inferences from one method should not be applied automatically to analyses from another. Researchers should be mindful of their research question and choose the technique most suited for their purpose.