Alpha (Significance Level or Cut-Off)

The key technical output of a significance test is the p-value (see Formal Hypothesis Testing). This p-value is then compared to some pre-specified cut-off, which is usually called  (the Greek letter alpha). For example, most studies use a cut-off of and take conclude that a test is significant if .

Having a standard rule such as this gives the veneer of rigor as it is a transparent and non-subjective process. Unfortunately, when statistical tests are an input into real-world decision making it is generally not ideal to use such a simple process. Rather, it is better to take into account the costs and risks associated with an incorrect conclusion.

Consider a simple problem like a milk company deciding whether or not to change the color of its milk package from white to blue. A study may find that a small increase in sales results if the color change is made. However, the resulting p-value may be 0.06. Thus, if using the 0.05 cut-off the conclusion would be that color makes no difference. However, if it costs the company nothing to make the change, then there is no downside, and they are better off making the change. That is, perhaps the change in packaging will have no impact, in which case there is no downside. And, there is the possibility that making the change will result in a small increase. Now, let's instead suppose that the p-value is 0.04 but that the change in pack size will cost the company millions of dollars. In that situation, it is likely best to conclude that there is no significant effect for a change in pack size, even though the p-value is less than the 0.05 cut-off, as now the 0.04 chance is interpreted as meaning that there is a non-zero chance that the company could spend millions of dollars for no gain (and, the better course of events is for the company to increase the sample size of the study to see if it results in a smaller p-value).

The academic discipline of decision theory focuses on the question of how to weigh up risks and costs in such situations.