This page discusses common faulty or non-respectable statistical practice.
Some abstracts are based on Gigerenzer, Gerd. “Mindless statistics.” The Journal of Socio-Economics 33.5 (2004): 587-606..

Common Types of Statistics

Descriptive statistics
Neyman-Pearson decision theory
Bayesian statistics
Wald’s sequential analysis

The Null Ritual

The null ritual is commonly used, but it is NOT a correct statistical method.
Step 1. set a null hypothesis, but no any alternative hypothesis;
Step 2. report \( p \)-value as \( p \lt 0.05/0.01/0.001… \);
Step 3. always follow the same procedure.

Fisher’s Blame on the Null Ritual

▻▻ Null hypothesis can be something other than nil-hypothesis. For example, the difference is less than 0.5;
▻▻ It is not respectable to use a constant significance level;
▻▻ “Hypotheses” is a plural, which means there are always different approach to perform tests.

Procedure of Testing

Fiser’s Null Hypothesis Theory

Step 1. set up a null hypothesis, which must be a nil-hypothesis (i.e. the difference equals 0);
Step 2. report the exact figure of significance level;
Step 3. use this procedure only when you know very FEW of the problem on hand.

Neyman-Pearson Decision Theory

Step 1. set up both \( H_1 \) and \( H_2 \); decide \( \alpha,\ \beta \).; and sample size before the experiment. The rejection region can thus be determined;
Step 2. if falls into rejection region of \( H_1 \), then accept \( J_2 \), and vice versa;
▻▻ Please note: this does not mean that you believe in it, but only implies that you act as if it were true.
Step 3. use this procedure only when two hypotheses are disjoint, and you can make meaningful cost-benefit trade-offs for \( \alpha,\ \beta \).
This procedure is very useful in quality control.

Level of Significance

The level of significance can mean:
1. more convention (Fisher): take 5% in general:
▻▻ under this level, the non-significant scenario can be ignored.
2. alpha (Neyman-Pearson): the percentage is the % of wrongly rejecting \( H_1 \).
3. the exact figure of significance level.
The differences are:
In Fisher’s theory, it is a property of the data, i.e. a relation between a body of data and a theory;
In Neyman-Pearson’s theory, it is a property of a test, not for data.

Meehl’s Conjecture (1978)

In nonexperimental settings with large sample sizes, the probability of rejecting the null hypothesis of nil group differences in favor of a directional alternative is about 0.50.

Feynman’s conjecture (1998)

To report a significant result and reject the null in favor of an alternative hypothesis is meaningless unless the alternative hypothesis has been stated before the data was obtained.