Lesson 5 of 5 · 8 min · Shape and relationships

Correlation: measuring relationships, and why it is not causation

Covers: Python 3.14 statistics module and standard definitions. Updated 13 Nov 2026. Part of the Data Analyst roadmap.

What you will learn

  • Draw and read a scatter plot
  • Calculate and interpret the correlation coefficient
  • Know when to use Spearman instead of Pearson
  • See how outliers and hidden groups distort correlation
  • Explain correlation versus causation in plain words

In simple words

Correlation asks: when one thing goes up, does another tend to go up (or down) with it? Hours studied and exam marks, advertising spend and sales, temperature and ice-cream sales. The correlation number runs from minus one to plus one and shows how tightly two things move together. But moving together is not the same as one causing the other, which is the most important idea in this lesson.

Words to know

  • Correlation: how strongly two variables move together.
  • Scatter plot: a chart with one dot per row, one variable on each axis.
  • Correlation coefficient (r): a number from -1 to +1 summarising the strength and direction of a straight-line relationship.
  • Confounder: a hidden third factor that influences both variables.

Start with a scatter plot

Advertising spend (thousands of rupees) and sales (thousands) for six months:

Spend: 10, 20, 30, 40, 50, 60
Sales: 25, 38, 52, 58, 77, 81
import matplotlib.pyplot as plt
plt.scatter(spend, sales)
plt.xlabel("Ad spend (thousand INR)"); plt.ylabel("Sales (thousand INR)")
plt.show()

Dots rising from lower left to upper right show a positive relationship; falling shows a negative one; a shapeless cloud shows none. Always plot before calculating anything, because a single number can hide the shape (curves, clusters and outliers).

The correlation coefficient

The Pearson correlation coefficient, r, ranges from -1 to +1:

r Reading
+1 perfect rising straight line
+0.7 to +0.9 strong positive
+0.3 to +0.7 moderate positive
around 0 no straight-line relationship
negative values the same strengths, but as one rises the other falls
-1 perfect falling straight line

For the advertising data, r is about 0.99: spend and sales rise together almost in a line. The squared value, r squared, is the share of the variation in one variable that a straight line on the other explains: 0.99 squared is about 0.98, so 98 percent of the sales variation here lines up with spend. Treat the verbal labels as rough, and note that what counts as "strong" depends on the field: 0.5 is impressive in social science and weak in physics.

Calculate it:

import statistics as st
st.correlation(spend, sales)          # 0.99  (Python 3.10+)
df["spend"].corr(df["sales"])         # pandas
df.corr(numeric_only=True)            # a correlation matrix of all numeric columns

Excel: =CORREL(A2:A7, B2:B7). SQL (PostgreSQL and others): SELECT CORR(sales, spend) FROM monthly;.

Pearson and Spearman

Pearson measures straight-line relationships and is sensitive to outliers. Spearman converts values to ranks first and measures whether one variable consistently rises with the other, in any steady (monotonic) way, curved or not. It is robust to outliers.

Make the last sales figure a data-entry error, 500 instead of 81. The Pearson correlation falls from 0.99 to about 0.72, even though the order of values still rises perfectly; Spearman stays at 1.0. Use Spearman for ranked data, skewed data and outliers:

df["spend"].corr(df["sales"], method="spearman")
st.correlation(spend, sales, method="ranked")      # Python 3.12+

What can distort a correlation

  • Outliers: one extreme point can create or destroy a correlation. Plot and inspect.
  • Restricted range: looking only at a narrow slice of values (only the top students) weakens the apparent relationship.
  • Curves: a U-shaped relationship can have r close to zero although the variables are strongly related.
  • Hidden groups: two groups with different baselines can show a strong overall correlation that disappears inside each group (Simpson's paradox). Split by group and check.
  • Small samples: with six points, a high r can happen by chance. Report the sample size, and for important decisions test whether it is statistically significant (scipy.stats.pearsonr gives a p-value).
  • Time trends: two series that both rise over time correlate even if unrelated (the number of mobile phones in India and the number of cars). Compare changes rather than levels.

Correlation is not causation

If A and B are correlated, there are several explanations:

  1. A causes B (more advertising brings more sales).
  2. B causes A (a company with rising sales can afford to advertise more).
  3. A third factor causes both: ice-cream sales and drowning deaths rise together because both rise in hot weather. The hidden factor is a confounder.
  4. Coincidence: with enough variables, some will correlate by chance.

In the advertising example, festival season might raise both spend and sales; the correlation would be real but advertising may not be the driver. To claim causation you need stronger evidence: a controlled experiment or A/B test (randomly show ads to some customers and not others), a before-and-after comparison with a control group, or careful adjustment for confounders. Until then, say "is associated with", not "causes".

Practical wording in a report: "Months with higher ad spend had higher sales (r = 0.99, six months of data). This does not show that spend caused the increase; seasonality may affect both. A test in two comparable cities would tell us."

Using correlation in analysis

  • Screen many variables with a correlation matrix and a heatmap to find relationships worth investigating.
  • Look for strongly correlated inputs (multicollinearity) before building models; they carry the same information.
  • Always follow a surprising correlation with a plot, a check for outliers and groups, and a question: what else could explain it?

Common mistakes

  • Saying "causes" from a correlation.
  • Calculating r without a scatter plot.
  • Using Pearson on curved relationships or data with outliers.
  • Ignoring sample size.
  • Correlating two things that both trend over time.

Try it yourself

  1. Plot the advertising data and calculate Pearson r. Then replace the last sales value with 500 and compare Pearson and Spearman.
  2. Compute the correlation matrix of numeric columns in a dataset you have, and pick the strongest pair. Plot it.
  3. Write two sentences, one wrong ("A causes B") and one correct, about a correlation you found.
  4. Think of a confounder for: the number of fire engines at a fire and the damage done.

Check yourself

What does a correlation of -0.8 mean?

A strong negative relationship: as one variable rises, the other tends to fall.

Why is a scatter plot important before relying on r?

r hides the shape; the plot reveals curves, outliers and groups that make r misleading.

When would you use Spearman instead of Pearson?

For ranked data, curved but steadily rising relationships, or data with outliers.

What is a confounder?

A third variable that influences both of the variables you are comparing and can create a correlation with no direct cause between them.

Interview questions

Sales and ad spend are highly correlated. Does advertising drive sales?

Not necessarily. Correlation shows association, not cause. Reverse causation, seasonality or another confounder could explain it; run an experiment or A/B test or control for confounders before claiming causation.

What is Simpson's paradox?

A trend that appears in several groups can reverse or disappear when the groups are combined, because group sizes and baselines differ. Always check relationships within segments.

What does r squared tell you?

The share of the variance in one variable explained by a straight-line relationship with the other; for example 0.64 means about 64 percent.

Summary

  • A scatter plot first, then the correlation coefficient r (from -1 to +1).
  • Pearson measures straight-line links and is sensitive to outliers; Spearman is rank-based and robust.
  • Outliers, hidden groups, curves, small samples and shared time trends can mislead.
  • Correlation is not causation; use experiments to establish cause.

Going deeper

See how much of this you can explain

A free mock interview asks you new questions on Statistics for analysts for Data Analyst and scores every answer, with what to fix.