Null Hypothesis

All statistical tests require a null hypothesis (see Formal Hypothesis Testing). From time to time the null hypotheses that are used in statistical tests are not sensible and the consequence of this is that the p-values are not meaningful. As an example, consider the table below, which is a special type of table known as a duplication matrix. This table has the same data shown in both the rows and the columns. Thus, the first column shows that 100% of people who consume Coke consume Coke (not surprisingly), 14% of people who consume Coke also consume Diet Coke, 25% of people who consume Coke also consume Coke Zero, etc.

The arrows on the table show the results of statistical tests. In all of these tests, the null hypothesis is independence between the rows and the columns (see Statistical Tests on Tables for a description of what this null hypothesis entails, although the description is non-technical and the word independence is not used). However, this assumption is clearly not appropriate, as the same data is shown in the rows and columns and thus it cannot be considered independent in any meaningful sense, and a different null hypothesis is required.

To appreciate how the incorrect null hypothesis renders the significance tests meaningless focus on the top-left cell. It suggests that the finding that 100% of people who consume Coke also consume Coke is significantly high. However, it is a logically necessary conclusion and thus cannot, in any sense, be considered to be significant (i.e., the only possible value is 100% and thus the only sensible null hypothesis is that this value is 100% so the test should not be significant). All the other tests in this table are also wrong. The ones in the main diagonal are wrong for the same reason as just discussed. The other tests are wrong because the assumption of independence is not sensible in duplication matrices, as typically buying one brand is correlated with buying other brands (Ehrenberg, A. S. C. (1988). Repeat Buying: Facts, Theory and Applications. New York, Oxford University Press.)