Should We Ship the Recommendation Algorithm?
Description You're joining Fathom as their first Senior Data Scientist. Fathom is a subscription fitness app with 340,000 paying subscribers. The Growth team just ran an A/B test on a new workout-recommendation algorithm and is ready to…
Create a free account to upload your work. Your progress saves as a draft until you submit.
What You'll Be Doing
Description
You're joining Fathom as their first Senior Data Scientist. Fathom is a subscription fitness app with 340,000 paying subscribers. The Growth team just ran an A/B test on a new workout-recommendation algorithm and is ready to ship it to 100% of users — they've already reallocated next quarter's roadmap assuming it ships.
The VP of Product has asked you to review the results before final sign-off: "This looks like a clear win to Growth. I need your read before I take it to the board in a week."
The experiment data:
| Metric | Treatment (new algorithm) | Control (existing algorithm) |
|---|---|---|
| Users | 82,000 | 81,400 |
| Test duration | 9 days | 9 days |
| Weekly active minutes (primary metric) | +4.2% vs. control (p = 0.008) | — |
| Revenue per active user, 7-day | −0.3% vs. control (p = 0.61, not significant) | — |
| Support tickets tagged "recommendations felt repetitive/irrelevant" | +18% vs. control | — |
| Engagement lift, users with <30 days tenure | +9.1% | — |
| Engagement lift, users with >90 days tenure | −1.2% (not significant) | — |
| Number of secondary engagement metrics tested | 14 total; this is the only one significant at p < 0.05 | — |
Additional context: The Growth team's own postmortems on the last three recommendation-algorithm experiments found that engagement lifts from launch-week tests faded to statistically null within roughly three weeks of full rollout in two of the three cases.
You have 45 minutes to form your recommendation.
Constraints to Consider
- No further testing before the decision: The board conversation is in one week. You cannot extend the experiment or run a follow-up test before you need to give your recommendation.
- No new data collection: You have the experiment logs and the historical postmortem summaries above — no ability to instrument new metrics or pull additional segments before the deadline.
- Reversing course has a real cost: Growth has already reallocated next quarter's roadmap assuming this ships. A "don't ship" or "ship partially" recommendation is not free — it has planning and political cost, and you should account for that honestly rather than ignoring it.
- The board is expecting a decision, not a request for more time: Leadership has already told the board a call would be made this quarter. "We need to gather more data" is not an acceptable recommendation on its own.
- You do not own the recommendation model's implementation — you are being asked for your analytical judgment and recommendation, not a technical redesign of the algorithm itself.
AI Usage Guidance
We expect you to use AI tools. We evaluate how you use them — not whether you use them. Evidence of iteration, redirection, and critical evaluation scores higher than a polished output with no process documentation.
The single highest-signal indicator: your video answer to the mandatory AI question. If you cannot name a specific moment where you redirected AI output, evaluators will assume you did not.
Mandatory AI question for your video: Walk me through one moment where you disagreed with, pushed back on, or redirected what the AI gave you — and what you did instead. Name the specific moment. Explain what the AI produced that didn't meet the bar, what you did differently, and why.
Video guidance for this role: Speak naturally, as if you're briefing the VP of Product directly. We're listening for the quality of your statistical judgment and your honesty about what the data does and doesn't support — not verbal polish.
Submission: Upload each deliverable as a separate file directly on the Provn platform: your Analysis & Recommendation Memo, your README document (Sections A, B, and C), and your video walkthrough (MP4 or MOV).
What You'll Accomplish
Demonstrate the ability to diagnose validity threats in an experiment result from imperfect, aggregate data
Translate a statistical finding into a specific, actionable business recommendation with an honest confidence statement
Show honest engagement with a result that complicates the "obvious" conclusion, not a smoothed-over success narrative
Connect a recommendation to a post-launch monitoring plan anchored to the scenario's own data
Show how you worked with AI to accelerate your process — including at least one moment where you redirected it
How Your Work Will Be Scored
What to Submit
File 1 — Analysis & Recommendation Memo:
Format: .pdf, .doc, .docx, .rtf, .txt, .md
A structured memo for the VP of Product presenting your read of the experiment, your recommendation (ship fully, ship to a segment, hold and re-test post-launch, or another option), and the single biggest risk in that recommendation. No required format or template — structure it as you would for a VP audience. Length: 1–2 pages equivalent.
Sign in to upload files
File 2 — README Document:
Format: .pdf, .doc, .docx, .rtf, .txt, .md
Three required sections — Section A — Statistical Diagnosis (300–500 words): Walk through what you believe is actually driving the observed result, what alternative explanation you considered and rejected, and the single biggest assumption that would change your recommendation if it turned out to be wrong. Section B — Business Communication & Monitoring Plan: Two parts required. Part B1: Your recommendation translated for the VP audience, and the specific pushback you'd expect from the Growth team, with your answer to it. Part B2: The monitoring metric and threshold you'd use to catch a fading effect or a regression after launch, anchored to a number in the data above, with a timeframe and what you'd change if you don't catch it in time. Section C — AI Usage Log (Mandatory): In a short section of your README, document your AI collaboration process. For each significant interaction with an AI tool, briefly note: what you asked the AI to help with / what it gave you / what you kept, changed, or rejected — and why. Three interactions documented is sufficient.
Sign in to upload files
File 3 — Video Walkthrough (8–10 minutes):
Format: .mp4, .mov, .webm
Record as MP4 or MOV and upload directly on the Provn platform as a separate file. Cover: (1) your diagnosis — what you believe is actually driving the result, and why; (2) your recommendation and the trade-off you made; (3) your business communication and monitoring plan highlights; (4) mandatory AI question (see AI Usage Guidance above); (5) what you'd do differently with more time or the ability to re-test.
Sign in to upload files
Create a free account to upload your work. Your progress saves as a draft until you submit.
On this page