AP Statistics 3.7-3.8: Carrying Out Tests and Errors

What does a p-value measure in a hypothesis test?

The p-value is the probability, assuming the null hypothesis is true, of obtaining a result at least as extreme as the observed sample result.

How does the size of a p-value affect evidence against the null hypothesis?

A small p-value indicates that the observed result would be unusual under the null hypothesis, giving stronger evidence against it. A large p-value indicates that the result is reasonably plausible under the null hypothesis.

Significance level, $\alpha$

The significance level is a preset probability of making a Type I error: rejecting a true null hypothesis. Common choices include $\alpha=0.05$, $0.01$, and $0.025$.

What is the decision rule using $p$-value and $\alpha$?

Reject $H_0$ when $p\text{-value}<\alpha$. Otherwise, do not reject $H_0$; equivalently, do not reject when $p\text{-value}\ge\alpha$.

Why is 'do not reject the null hypothesis' not the same as 'accept the null hypothesis'?

Failure to reject $H_0$ means only that the sample does not provide sufficient evidence against it. It does not establish that $H_0$ is true.

How should a hypothesis-test conclusion be written?

State the decision about $H_0$ and interpret it in the context of the original claim. For example, say that the data provide sufficient evidence for the alternative claim or do not provide sufficient evidence for it.

Null hypothesis, $H_0$

The null hypothesis is the baseline population claim being tested. It generally contains equality, either explicitly or through a boundary such as $\le$ or $\ge$.

Alternative hypothesis, $H_a$

The alternative hypothesis expresses the claim supported by evidence if the null hypothesis is rejected. It uses $<$, $>$, or $\ne$ and never contains an equality sign.

How should the wording 'the population proportion is greater than $p_0$' be represented in a hypothesis test?

Use $H_a:p>p_0$ and typically $H_0:p\le p_0$. The test is right-tailed because unusually large sample proportions support the alternative.

How should the wording 'the population proportion is less than $p_0$' be represented in a hypothesis test?

Use $H_a:p<p_0$ and typically $H_0:p\ge p_0$. The test is left-tailed because unusually small sample proportions support the alternative.

How should the wording 'the population proportion differs from $p_0$' be represented in a hypothesis test?

Use $H_0:p=p_0$ and $H_a:p\ne p_0$. This is a two-tailed test because results far above or far below $p_0$ can provide evidence against the null.

How does the alternative hypothesis determine the tail(s) of a hypothesis test?

$H_a:p<p_0$ gives a left-tailed test; $H_a:p>p_0$ gives a right-tailed test; and $H_a:p\ne p_0$ gives a two-tailed test.

What is the p-value region for a right-tailed test of a population proportion?

It is the area to the right of the observed test statistic or sample proportion, representing outcomes at least as large as the observed one.

What is the p-value region for a left-tailed test of a population proportion?

It is the area to the left of the observed test statistic or sample proportion, representing outcomes at least as small as the observed one.

What is the p-value region for a two-tailed test?

It includes outcomes at least as far from the null value as the observed result in both directions. For a symmetric distribution, the corresponding tail areas are placed on both sides of the null value.

What is the standard error for a sample proportion under the null hypothesis?

For a one-proportion $z$ test, use $\sigma_{\hat p}=\sqrt{\dfrac{p_0(1-p_0)}{n}}$, where $p_0$ is the proportion specified by $H_0$.

What is the $z$ test statistic for a single population proportion?

$z=\dfrac{\hat p-p_0}{\sqrt{p_0(1-p_0)/n}}$, where $\hat p=x/n$ is the sample proportion and $p_0$ is the null-hypothesis proportion.

Why does a hypothesis test use the null-hypothesis proportion to define the sampling distribution?

The p-value asks how unusual the observed result would be if $H_0$ were true. Therefore, the center and standard error used for the null model are based on $p_0$, the proportion specified by $H_0$.

How can sample size affect the power of a test for a population proportion?

Increasing the sample size decreases the standard error, making real differences from $p_0$ easier to detect. For a fixed $\alpha$ and alternative value, a larger sample generally increases power and decreases $\beta$.

What conditions support a one-proportion $z$ test?

The data should come from a random sample or randomized process, observations should be independent, and the 10% condition should be satisfied when sampling without replacement. Using $p_0$ for the null distribution, the expected numbers of successes and failures should each be at least 10: $np_0\ge10$ and $n(1-p_0)\ge10$.

What is the difference between a Type I and a Type II error?

A Type I error occurs when $H_0$ is rejected even though it is true. A Type II error occurs when $H_0$ is not rejected even though it is false.

Rare event in hypothesis testing

A sample result is a rare event when it would be very unlikely to occur if the null hypothesis were true. Such an outcome provides evidence against the null hypothesis, although it does not prove the alternative hypothesis.

Type I error in a test of a population proportion

A Type I error occurs when the test concludes that the population proportion differs from, is greater than, or is less than the null value when the null hypothesis is actually true. Its probability is controlled by $\alpha$.

Type II error in a test of a population proportion

A Type II error occurs when the test does not provide sufficient evidence for the alternative claim even though the true population proportion satisfies the alternative hypothesis.

What is the power of a hypothesis test?

The power of a test is the probability of correctly rejecting $H_0$ when a particular alternative value is true. It is the probability of detecting a real difference or effect.

How are power and the probability of a Type II error related?

For a specified alternative value, $\text{Power}=1-\beta$, where $\beta$ is the probability of a Type II error. Thus, higher power means a lower probability of failing to detect that alternative.

How does changing $\alpha$ generally affect $\beta$ and power?

For a fixed sample size and effect size, increasing $\alpha$ makes rejection easier, which generally decreases $\beta$ and increases power. Decreasing $\alpha$ generally increases $\beta$ and decreases power, creating a tradeoff between Type I and Type II errors.

A test has $p\text{-value}=0.025$ and $\alpha=0.01$. What decision should be made?

Do not reject $H_0$ because $0.025\ge0.01$. The sample does not provide sufficient evidence at the 1% significance level for the alternative hypothesis.

What does a two-tailed p-value of $0.5485$ imply for testing $H_0:p=0.50$ versus $H_a:p\ne0.50$?

A p-value of $0.5485$ means that, if $p=0.50$, results at least as far from $0.50$ as the observed sample proportion are common. At conventional significance levels such as $0.05$ or $0.01$, do not reject $H_0$.

What does it mean if a sample result is statistically significant?

It means the p-value is smaller than the chosen significance level, so the result would be relatively rare if the null hypothesis were true. Statistical significance does not by itself measure practical importance.