AP Statistics 1.7-1.8: Data Collection and Experimental Design

Experimental design

The planned structure of a study used to investigate how an explanatory variable affects a response variable. A sound design helps produce reliable data and supports valid conclusions.

Population versus sample

The population is the complete group about which a study seeks information; the sample is the subset actually observed. Conclusions about a population are based on data from the sample.

Sampling

The process of selecting a subset of individuals or observations from a population to estimate characteristics of the whole population.

Why is sampling often used instead of conducting a census?

Sampling usually costs less, takes less time, and may be the only practical option when measuring every member of a population is impossible.

Explanatory variable

The variable that a researcher manipulates or uses to explain changes in another variable. In an experiment, its assigned values define the treatments.

Response variable

The outcome measured in an experiment or study. It is expected to change in response to the explanatory variable.

Treatment

A specific condition or level of the explanatory variable imposed on experimental units. For example, aspirin and a placebo are two treatments in a medication experiment.

Experimental unit

The individual object or subject that receives a treatment and is measured in an experiment. When the units are people, they are often called subjects or participants.

Observational study

A study in which researchers observe or measure variables without assigning treatments. Observational associations generally cannot establish causation because groups may differ in lurking variables.

Lurking variable

A variable not included as the main explanatory variable that can affect the response and obscure the relationship being studied. For example, exercise and diet could confound an observational comparison of vitamin E users and nonusers.

Why does observing that vitamin E users are healthier not prove that vitamin E prevents disease?

Vitamin E users may differ from nonusers in other health-related behaviors, such as exercise, diet, or smoking. These lurking variables could explain the observed difference.

Randomized experiment

An experiment in which the researcher assigns treatments to experimental units and measures the resulting responses, using random assignment to treatment groups. Randomization supports cause-and-effect conclusions by balancing lurking variables on average.

Why does random assignment help establish causation?

Random assignment tends to distribute both known and unknown lurking variables among treatment groups. Thus, systematic differences in responses can be attributed to the imposed treatments rather than preexisting group differences.

Random sampling versus random assignment

Random sampling selects units from a population and improves the ability to generalize results. Random assignment places study units into treatments and supports conclusions about cause and effect; they serve different purposes.

Control group

A comparison group that does not receive the active treatment, often receiving a placebo or standard condition. It helps separate treatment effects from effects caused by participation, expectations, or time.

Placebo

A treatment designed to resemble the active treatment but that should not affect the response variable. It provides a comparison for evaluating effects beyond participants' expectations.

Placebo effect

A change in a participant's response caused by expectations about receiving treatment rather than by the treatment's active component. A placebo control helps identify this effect.

Blinding

Keeping people involved in a study unaware of which treatment a participant receives. Blinding reduces effects caused by expectations or differential treatment of participants.

Double-blind experiment

An experiment in which both participants and the researchers who interact with or evaluate them do not know treatment assignments. The treatment code is typically revealed only after data collection or according to a prespecified procedure.

Can every randomized experiment be double-blind?

No. Some treatments are obvious to participants, such as a floral scent versus no scent or texting versus not texting. Researchers or outcome assessors may still be blinded when feasible.

How would you identify variables in an aspirin heart-attack experiment?

The experimental units are the individual participants; the explanatory variable is assigned medication; the treatments are aspirin and placebo; and the response is whether each participant experiences a heart attack during the study.

Repeated-measures design

A design in which each participant or experimental unit receives multiple treatments and provides responses under each condition. It can reduce variation between individuals, but order, carryover, fatigue, and learning effects must be considered.

How can randomizing treatment order improve a study in which every participant receives both treatments?

Randomizing whether participants receive treatment A or B first helps balance order effects, such as practice, fatigue, or learning, between conditions. This is especially important in repeated-measures or crossover designs.

Why could birth order not be assigned in a randomized experiment?

Birth order is a preexisting characteristic that researchers cannot randomly impose. Without random assignment, groups may differ in many other ways, so a study of its association with personality cannot by itself establish causation.

What design issue arises when comparing distracted and undistracted driving?

Randomly assigning drivers to text while driving could create an unsafe and unethical experiment. A safer design might use a simulator, and if each participant experiences both conditions, treatment order should be randomized.

What participant characteristics should be considered in a driving-performance study?

Researchers should define the target population and recruit participants who appropriately represent relevant drivers. Age, driving experience, vision, texting experience, and other factors may affect response time and should be measured or balanced.

Representative sample

A sample whose relevant characteristics reasonably reflect those of the population of interest. Representativeness depends on the sampling method, not merely on having a large sample.

Sampling frame

A list or operational procedure that identifies the units eligible to be selected for a sample. If it omits or overrepresents parts of the target population, estimates may be biased.

Probability sample

A sample in which every population unit has a known, nonzero probability of selection. Known selection probabilities allow researchers to use appropriate weighting and quantify sampling uncertainty.

Why can unequal selection probabilities still produce a probability sample?

Equal probabilities are not required. A sample is a probability sample as long as each unit's chance of selection is known and greater than zero; analysis can account for unequal chances using weights.

How are sampling weights used?

A sampled unit may represent itself and other population units that had similar selection opportunities. A common weight is related to the inverse of its probability of selection, such as a weight of $2$ when the selection probability is $1/2$.

Simple random sample

A probability sample in which every possible sample of a specified size has an equal chance of selection. It requires a suitable sampling frame and random selection.

Stratified sampling

A method that divides the population into meaningful subgroups called strata and randomly samples within each subgroup. It can ensure representation of important groups and improve precision.

Cluster sampling

A method that divides the population into naturally occurring groups called clusters, randomly selects clusters, and surveys units within selected clusters. It can reduce cost but may be less precise when units in the same cluster are similar.

Systematic sampling

A method that selects units from an ordered sampling frame by choosing a random starting point and then selecting every $k$th unit. The interval $k$ is determined by the population and desired sample size.

Convenience sampling

Selecting units that are easiest to reach. It can produce systematic bias because accessible individuals may differ from those who are missed.

Voluntary response sample

A sample formed when individuals choose themselves to participate, often by responding to an open invitation. It is vulnerable to voluntary response bias because people with strong opinions may be more likely to respond.

Undercoverage bias

Bias that occurs when some groups in the population are inadequately represented or excluded from the sampling frame. The results may be biased if the omitted groups differ from those included.

Nonresponse bias

Bias that occurs when selected individuals do not participate or cannot be contacted and those individuals differ meaningfully from respondents. Researchers should make reasonable follow-up efforts and report the limitation.

What is the problem with selecting a neighborhood block because it is convenient for the researcher?

The resulting convenience sample may not represent the entire community. The researcher should use a defensible random or probability-based selection process and attempt to contact all selected households.

How can selective omission of households bias a community survey?

Skipping households or failing to revisit absent residents can exclude groups such as working families. If their characteristics differ from respondents', the estimated community results will be biased.

Completely randomized design

An experimental design in which all experimental units are randomly assigned to one of the treatments, with no separate grouping or blocking. It is appropriate when the units are reasonably similar or when no important source of variation is identified.

Block design

An experimental design that first divides experimental units into blocks of similar individuals based on a variable expected to affect the response, then randomly assigns treatments within each block. Blocking reduces variation from the blocking variable and allows clearer treatment comparisons.

Completely randomized design versus block design

A completely randomized design randomizes all units together. A block design randomizes separately within groups of similar units, which can improve precision when the blocking variable explains differences in the response.

Replication in an experiment

Applying each treatment to many experimental units rather than relying on a single unit or observation. Replication helps distinguish genuine treatment effects from random variation and makes results more reliable.

Three essential principles of experimental design

Random assignment helps balance lurking variables, control provides a meaningful comparison, and replication reduces the influence of chance variation. Blinding can additionally reduce expectation and observer effects.