The Metric That Lied Challenge

What You'll Be Doing

Marlowe Athletic is a fitness chain: 84 clubs, roughly 310,000 members, three membership tiers. You are an analyst on a four-person data team.

The VP of Membership is presenting to the CEO on Thursday. Her headline is on the ops dashboard: “Monthly member churn improved from 4.1% to 3.2% — our best quarter in two years.” She wants to repeat the January promotion in Q3.

Here is the underlying table.


Month	Members at month start	Cancellations in month	Churn rate shown on dashboard
December	268,400	10,870	4.05%
January	271,900	11,150	4.10%
February	309,600	10,220	3.30%
March	314,100	10,050	3.20%

Four other things you know:

The January promotion — $0 join fee, first month $5 — ran from January 8 to February 5 and brought in 44,300 new members.
Every Marlowe membership carries a 90-day minimum term. A member cannot cancel before day 91.
The dashboard computes churn as cancellations in month ÷ members at month start.
An AI assistant auto-generates the commentary under the tile. This month it says: “Churn improved 0.9 points quarter over quarter, driven by improved member satisfaction following the January retention initiative. Recommend expanding the initiative to Q3.”

Write the VP a short note before Thursday.

Constraints to consider

You cannot change the dashboard. It is a vendor BI package and the next release window is six weeks out. Whatever you recommend has to work alongside the number the CEO is going to see.
The VP gets one page and one recommended metric — not a metric framework, not a list of options.
You do not have member-level cancellation dates for the promotion cohort. That export takes a week. Reason with the table above and state what you would confirm when it lands.
You may not propose an A/B test or a holdout. The promotion already ran nationally. There was no control group and there will not be one retroactively.
The Q3 repeat is already approved. Your note has to be useful whether or not it gets stopped.

AI Usage Guidance

We expect you to use AI tools. We evaluate how you use them — not whether you use them. Evidence of iteration, redirection, and critical evaluation scores higher than a polished output with no process documentation.

The single highest-signal indicator: your video answer to the mandatory AI question. If you cannot name a specific moment where you redirected AI output, evaluators will assume you did not.

Mandatory AI question for your video: Walk me through one moment where you disagreed with, pushed back on, or redirected what the AI gave you — and what you did instead. Name the specific moment. Explain what the AI produced that didn’t meet the bar, what you did differently, and why.

Speak naturally. Communication is assessed on clarity of technical ideas and logical structure — not verbal polish, accent, or filler words.

Submission: Upload each deliverable as a separate file directly on the Provn platform.

What You'll Accomplish

Identify the mechanism by which a metric moved, separately from the story attached to it

Distinguish a rate change caused by the numerator from one caused by the denominator

Propose a single replacement measure precise enough for someone else to compute

Evaluate a machine-generated causal claim against the evidence available

Explain a measurement problem to a non-technical executive in under four minutes

How Your Work Will Be Scored

Metric Diagnosis: Identifies the mechanism behind the movement using the numbers given, rather than describing the trend Measurement Redesign & Scrutiny of AI Commentary: Proposes one precisely specified replacement metric and separates the AI commentary’s supported claims from its invented ones Video Walkthrough & Communication: Explains a measurement problem to a non-technical executive clearly and in sequence AI Fluency: Shows directed, iterative, critically evaluated AI use

What to Submit

File 1 — Note to the VP

Document · 250 words max, one pageRequired

Format: .pdf, .doc, .docx, .rtf, .txt, .md

What the churn number is actually telling her, the one metric you would put next to it, and what you expect to happen next.

File 2 — README Document

DocumentRequired

Format: .pdf, .doc, .docx, .rtf, .txt, .md

Three required sections:

Section A — Written analysis: the mechanism. Show the arithmetic that supports your reading, name what you would predict for April and May, and state the one thing in the table that could still prove you wrong. 150–250 words.
Section B — Your replacement metric: state it precisely — numerator, denominator, population, and time window — such that another analyst would compute exactly the same number. Then say, in one or two sentences, what you would not put in front of the CEO from the AI-generated commentary, and why.
Section C — AI Usage Log (Mandatory): This is not a trick. We want to see how you work with AI — not whether you used it. In a short section of your README, document your AI collaboration process. For each significant interaction with an AI tool, briefly note: what you asked the AI to help with / what it gave you / what you kept, changed, or rejected — and why. Three interactions documented is sufficient. The log does not need to be exhaustive.

File 3 — Video Walkthrough

Video · 3–4 minutesRequired

Format: .mp4, .mov, .webm

Record as MP4 or MOV and upload directly on the Provn platform as a separate file.

Cover: (1) what the churn number is actually measuring, ~60 sec; (2) your replacement metric and what you predict for April–May, ~60 sec; (3) the mandatory AI question, ~60 sec; (4) what you would check first when the cohort export lands, ~30 sec. Speak naturally — we are assessing your reasoning, not verbal polish.