The core idea
Correlation measures the direction and strength of a linear association. Least-squares regression selects the line with the smallest sum of squared residuals. Confounding occurs when intertwined influences prevent a simple comparison from isolating the effect of interest.
1. Plot the pairs before compressing them
Each observation supplies a pair (x, y) belonging to the same unit. A scatterplot preserves those pairs and can reveal direction, curvature, clusters and unusual points. Do not sort the x and y columns separately: doing so invents new pairings. A rising pattern indicates that larger x values tend to accompany larger y values. It does not establish that changing x will change y. A curved relationship can be strong while its linear correlation is close to zero. For instance, the three points (−1, 1), (0, 0), (1, 1) lie exactly on y = x² but have zero correlation. Also check units, missing values and whether several observations come from the same person. A single summary number cannot show all these features.
Sources: OpenStax: The Regression Equation ↗ · OpenStax: Scatter Plots ↗
2. Correlation standardises a linear pattern
Let x̄ and ȳ be the sample means. Define Sxx = Σ(x−x̄)², Syy = Σ(y−ȳ)² and Sxy = Σ(x−x̄)(y−ȳ), summing over paired observations. Pearson correlation is r = Sxy/√(SxxSyy), provided both variables vary. Its value lies between −1 and 1 and has no units. Its sign gives the direction of linear association. Values near either endpoint indicate a close straight-line pattern; values near zero need not indicate no relationship. Converting hours to minutes does not change r, although it changes a regression slope’s units. If every x is identical, Sxx is zero and correlation is undefined, not evidence of no association. Outlying points can strongly affect r, particularly in small datasets. Interpret it with the plot and the data-generating context.
Sources: OpenStax: The Regression Equation ↗
3. A fitted line minimises squared residuals
A prediction line has form ŷ = a+bx. For an observed pair, the residual is e = y−ŷ: positive means the point is above the line. Ordinary least squares chooses a and b to minimise Σe². Squaring prevents positive and negative errors cancelling and gives larger deviations greater influence. With Sxx > 0, b = Sxy/Sxx and a = ȳ−bx̄. The fitted line passes through (x̄, ȳ). Its slope describes the change in fitted y per one-unit change in x, within the fitted relationship. The intercept is the fitted value at x = 0, which may have no useful practical meaning if zero lies far outside the observed range. The optimisation criterion defines a descriptive line; additional assumptions are needed for uncertainty intervals or population claims.
Sources: OpenStax: The Regression Equation ↗ · NIST: Linear Least Squares Regression ↗
4. Worked case: fit and check a small invented dataset
Use four invented practice records: hours x = 0, 1, 2, 3 and marks out of ten y = 2, 4, 5, 7. Their means are x̄ = 1.5 and ȳ = 4.5. Calculating deviations gives Sxx = 5, Sxy = 8 and Syy = 13. Therefore b = 8/5 = 1.6 and a = 4.5−1.6(1.5) = 2.1. The line is ŷ = 2.1+1.6x. Fitted values are 2.1, 3.7, 5.3, 6.9; residuals are −0.1, 0.3, −0.3, 0.1, whose squares sum to 0.20. Correlation is 8/√65 ≈ 0.9923. This deliberately tidy example teaches arithmetic, not a real effect of study time. Four observations with a high correlation are not sufficient grounds for a broad educational conclusion.
Sources: OpenStax: The Regression Equation ↗
5. Fit quality is not prediction certainty
For this least-squares fit with an intercept, R² = 1−SSE/Syy = 1−0.20/13 ≈ 0.9846, also equal to r². It describes the fraction of the observed squared variation around ȳ accounted for by the fitted line. It is not a 98.46% probability of a correct prediction and does not measure causation. At x = 2.5 the line predicts 6.1 marks, an interpolation within the observed range. At x = 10 it predicts 18.1 marks, impossible for a ten-mark assessment. Extrapolation exposes the model’s limits. Plot residuals against x: a curve suggests a missing nonlinear pattern; a widening spread suggests changing variability. Predicting one new learner is more uncertain than estimating the average response at a given x. Honest prediction reports that uncertainty and uses a suitable population and model.
Sources: OpenStax: The Regression Equation ↗ · OpenStax: Prediction ↗
6. Worked case: group composition can reverse a comparison
Consider invented observational results for study methods A and B, separated by prior preparation. Among well-prepared learners, A succeeds for 9/10 = 90% and B for 72/90 = 80%. Among less-prepared learners, A succeeds for 18/90 = 20% and B for 1/10 = 10%. A has the higher observed success rate in both preparation groups. Yet overall A succeeds for 27/100 = 27%, while B succeeds for 73/100 = 73%. Most A users were in the harder-to-succeed group; most B users were already well prepared. The totals use different group weights, producing a reversal. This is an illustration of Simpson’s paradox. Comparing within preparation groups addresses one imbalance but still does not establish a causal advantage: other differences, selection and chance can remain.
Inspect the pattern and the groups behind it
| Preparation | Method A | Method B |
|---|---|---|
| Well prepared | 9/10 = 90% | 72/90 = 80% |
| Less prepared | 18/90 = 20% | 1/10 = 10% |
| Overall | 27/100 = 27% | 73/100 = 73% |
Sources: OpenStax: Contingency Tables ↗ · OpenStax: Data, Sampling, and Variation in Data and Sampling ↗
7. Design determines which claims are supportable
A plausible confounder affects the outcome and is associated with the explanatory variable, making their influences difficult to separate. In the method example, prior preparation is related to both method choice and success. Random assignment of methods can balance such factors in expectation and support a causal comparison when an experiment is well conducted; it does not guarantee identical groups in every finite sample. Random sampling serves a different purpose: supporting generalisation to a population. An observational regression may adjust for measured differences, but unmeasured confounding can remain. Adding every available variable mechanically is not a substitute for understanding the process. Keep three claims distinct: describing these observations, predicting similar future observations, and estimating the effect of an intervention. Each requires its own evidence.
Sources: OpenStax: Experimental Design and Ethics ↗ · OpenStax: Data, Sampling, and Variation in Data and Sampling ↗
PUT IT INTO PRACTICE
Practice: check a fitted line and its limits
- For the four-point dataset, use ŷ = 2.1+1.6x to calculate the fitted value and residual at x = 2, where y = 5.
- Express the same slope when x is measured in minutes instead of hours. Keep the predicted marks unchanged.
- Calculate a prediction at 150 minutes, and identify why a ten-hour prediction is not credible for a ten-mark assessment.
- Solution: at two hours the fitted value is 5.3 and residual is −0.3. The slope becomes 1.6/60 marks per minute, so 150 minutes still predicts 6.1. Ten hours predicts 18.1, violating the score range and extrapolating far beyond the data. Changing units preserves predictions; extending a model beyond its scope does not preserve validity.
Check your understanding
Can zero correlation coexist with a strong relationship?
Yes. Correlation describes linear association; a symmetric curved pattern such as y = x² can have zero correlation.
Why square residuals?
It prevents opposite signs cancelling and defines a criterion that penalises large deviations more strongly. It also makes outliers influential.
Does r = 0.99 demonstrate causation?
No. A common cause, selection mechanism or other process may create the association. Its strength does not settle its cause.
Why did the overall method comparison reverse?
The methods served very different mixtures of preparation groups. Overall rates weighted the harder and easier groups differently.
Are random sampling and random assignment interchangeable?
No. Sampling concerns who enters a study; assignment concerns which treatment participants receive. They address different inferential problems.
Does R² tell us how often a prediction is correct?
No. It summarises fitted squared variation in the observed response. Prediction accuracy for new data requires separate evaluation.