Lesson 1 of 5 · 9 min · Metrics that mean something
Defining KPIs: from a business goal to a number you can trust
What you will learn
- Turn a business goal into a measurable KPI
- Write a KPI definition that two people cannot read differently
- Tell vanity metrics from actionable ones
- Choose between counts, rates and averages
- Add guardrail metrics so you do not improve one number by hurting another
In simple words
A KPI (key performance indicator) is the one number a team watches to know if it is winning. "Make the app better" is a wish; "increase the share of new users who finish their first lesson within a day" is a KPI. The hard part is not calculating it, it is defining it so exactly that everyone gets the same figure, and choosing one that actually tells you something you can act on.
Words to know
- KPI: a key number that shows progress towards an important goal.
- Metric: any measurement; a KPI is a metric chosen because it matters.
- Numerator and denominator: the top and bottom of a rate, such as paid users divided by all users.
- Guardrail metric: a number that must not get worse while you improve the main KPI.
From goal to KPI
Start from the business question, not the data you happen to have.
- Goal: what is the team trying to achieve? "More students should get value from our practice tests."
- Question: what would tell us it is happening? "Do students come back and practise regularly?"
- Metric: pick something measurable that answers it. "Weekly active learners," "tests completed per active learner," "percentage of new students who complete 3 tests in their first 14 days."
- Target: what level and by when? "From 22 percent to 30 percent by March."
A good KPI is measurable, relevant to the goal, within the team's influence, and time-bound. If nobody can change it, it is a statistic, not a KPI.
Write the definition down
Most arguments about numbers are really arguments about definitions. Write a one-page specification for each KPI:
| Item | Example: "Activation rate" |
|---|---|
| Name | Activation rate (14 days) |
| Business meaning | New students who start practising seriously |
| Formula | users with at least 3 completed tests within 14 days of signup / users who signed up |
| Population | all new signups; exclude test accounts and staff |
| Time window | signup date in the period; 14 days to activate |
| Source tables | users, test_attempts |
| Grain and refresh | daily, by signup date |
| Owner | head of product |
| Known issues | users before 1 March have missing device data |
| Target | 30 percent by 31 March |
Questions the definition must settle: Which users count (do you remove refunded, duplicate and test users)? What counts as an "active" day? Which time zone? What happens to partial periods? Is the denominator "all signups" or "signups who verified their email"? A KPI whose definition changes silently destroys trust and breaks trends; if you must change it, version it ("Activation v2") and restate history.
Vanity versus actionable metrics
- Vanity metrics go up easily and tell you little: total signups ever, page views, app downloads, total registered users. They only grow (cumulative) and rarely suggest what to do.
- Actionable metrics change when you change something and point at a decision: conversion rate of visitors to signups, percentage of signups who activate, repeat purchase rate in 60 days.
A simple test: "If this number went down, would we know what to investigate?" Cumulative totals fail. Rates and per-user numbers over a fixed window usually pass.
Counts, rates and averages
- Counts (orders, signups) show scale but rise with traffic; they cannot tell you whether the product got better.
- Rates and ratios (conversion, churn, defect rate) normalise for size and are easier to compare across periods and groups. Always state the denominator: "conversion" of visits, of sessions, or of unique visitors differ.
- Averages (average order value, average resolution time) hide skew; add the median or a percentile (see the statistics course).
- Per-unit metrics (revenue per active user, cost per acquisition) connect effort and result.
When a rate moves, ask whether the numerator, the denominator or both changed. Conversion can "improve" because the denominator (low-quality traffic) fell, which is not a win.
North star and a metric tree
Large teams pick one north star metric that captures the value customers get ("tests completed by active students each week"). Beneath it sits a metric tree, breaking it into parts the teams can influence:
Revenue = Visitors x Signup rate x Paid conversion x Average order value
Each branch has an owner. When revenue changes, the tree shows where to look (the drop-diagnosis lesson uses this).
Guardrails and unintended effects
People and algorithms optimise whatever you measure. This is often stated as Goodhart's law: when a measure becomes a target, it stops being a good measure. If you reward "tickets closed per hour", agents close tickets without solving problems. Pair each KPI with guardrail metrics that must not worsen:
| Main KPI | Guardrail |
|---|---|
| Signup conversion | quality of signups (activation rate), refund rate |
| Support tickets closed | customer satisfaction, reopen rate |
| Delivery speed | damage rate, cost per delivery |
| Revenue per user | churn, complaints |
Report the KPI and its guardrails together.
Computing a KPI in SQL
Activation rate by signup month, following the definition above:
WITH signups AS (
SELECT user_id, signup_date
FROM users
WHERE is_test = false
),
activated AS (
SELECT s.user_id
FROM signups s
JOIN test_attempts t ON t.user_id = s.user_id
AND t.completed_at < s.signup_date + INTERVAL '14 days'
GROUP BY s.user_id
HAVING COUNT(*) >= 3
)
SELECT DATE_TRUNC('month', s.signup_date) AS month,
COUNT(*) AS signups,
COUNT(a.user_id) AS activated,
ROUND(100.0 * COUNT(a.user_id) / COUNT(*), 1) AS activation_pct
FROM signups s
LEFT JOIN activated a ON a.user_id = s.user_id
GROUP BY 1
ORDER BY 1;
Notice the left join, so signups with no tests still count in the denominator. Forgetting that (using an inner join) is the classic way to overstate a conversion rate. Also be careful to give recent signups the full 14 days before including them: a user who signed up yesterday cannot yet have activated, so partial periods should be excluded or marked.
Common mistakes
- Choosing the metric from the data available rather than from the decision to be made.
- Leaving the definition unwritten, so each team calculates it differently.
- Reporting cumulative totals as if they were performance.
- A rate without a stated denominator.
- No guardrails, so the target is met by damaging something else.
- Including incomplete periods, making recent cohorts look worse.
Try it yourself
- Pick a product you use. Write its goal and three possible KPIs; mark which are vanity and which actionable.
- Write the one-page definition for one KPI, including population, window and owner.
- Draw a metric tree for revenue for an online course seller.
- Choose a guardrail metric for "reduce average support response time" and explain why.
- Write the SQL (or pandas) to calculate your KPI by month, using a left join for the denominator.
Check yourself
What makes a good KPI?
It is measurable, tied to a goal, within the team's influence, clearly defined and time-bound.
Why is "total registered users" usually a vanity metric?
It only grows, so it does not show whether things are getting better and gives no hint about what to do.
What is a guardrail metric?
A metric that must not get worse while you improve the main KPI, such as refund rate alongside signup conversion.
Why must a KPI definition state the denominator?
Rates change with the denominator; "conversion of visits" and "conversion of unique visitors" give different numbers, and teams will disagree without a precise definition.
Interview questions
How would you choose KPIs for a new feature?
Start with the goal the feature serves, define the user behaviour that shows success, choose a rate with a clear denominator and time window, set a target and baseline, and add guardrails for side effects.
What is the difference between a metric and a KPI?
A metric is any measurement. A KPI is a metric chosen because it directly reflects a key business goal and drives decisions.
What is Goodhart's law and how do you guard against it?
When a measure becomes a target, people optimise the number rather than the goal. Use guardrail metrics, review the definition regularly, and look at outcomes as well as activity.
Summary
- Start from the goal and the decision, then choose a measurable, actionable KPI.
- Write a precise definition: formula, population, window, source, owner, target.
- Prefer rates with a clear denominator to cumulative counts; pair KPIs with guardrails.
- Use a left join for the denominator and exclude incomplete periods.
Going deeper
- Service level objectives (Google SRE Book): how Google defines indicators, targets and why percentiles beat averages.
- Aggregate expressions in PostgreSQL:
FILTERandCOUNT(DISTINCT ...)for rate calculations. - PostgreSQL date and time functions: intervals and
DATE_TRUNCfor time windows.
See how much of this you can explain
A free mock interview asks you new questions on Metrics, KPIs and business analysis for Data Analyst and scores every answer, with what to fix.