You're joining Fathom as their first Senior Data Scientist. Fathom is a subscription fitness app with 340,000 paying subscribers. The Growth team just ran an A/B test on a new workout-recommendation algorithm and is ready to ship it to 100% of users — they've already reallocated next quarter's roadmap assuming it ships.
The VP of Product has asked you to review the results before final sign-off: "This looks like a clear win to Growth. I need your read before I take it to the board in a week."
The experiment data:
| Metric | Treatment (new algorithm) | Control (existing algorithm) |
|---|---|---|
| Users | 82,000 | 81,400 |
| Test duration | 9 days | 9 days |
| Weekly active minutes (primary metric) | +4.2% vs. control (p = 0.008) | — |
| Revenue per active user, 7-day | −0.3% vs. control (p = 0.61, not significant) | — |
| Support tickets tagged "recommendations felt repetitive/irrelevant" | +18% vs. control | — |
| Engagement lift, users with <30 days tenure | +9.1% | — |
| Engagement lift, users with >90 days tenure | −1.2% (not significant) | — |
| Number of secondary engagement metrics tested | 14 total; this is the only one significant at p < 0.05 | — |
Additional context: The Growth team's own postmortems on the last three recommendation-algorithm experiments found that engagement lifts from launch-week tests faded to statistically null within roughly three weeks of full rollout in two of the three cases.
You have 45 minutes to form your recommendation.
We expect you to use AI tools. We evaluate how you use them — not whether you use them. Evidence of iteration, redirection, and critical evaluation scores higher than a polished output with no process documentation.
The single highest-signal indicator: your video answer to the mandatory AI question. If you cannot name a specific moment where you redirected AI output, evaluators will assume you did not.
Mandatory AI question for your video: Walk me through one moment where you disagreed with, pushed back on, or redirected what the AI gave you — and what you did instead. Name the specific moment. Explain what the AI produced that didn't meet the bar, what you did differently, and why.
Video guidance for this role: Speak naturally, as if you're briefing the VP of Product directly. We're listening for the quality of your statistical judgment and your honesty about what the data does and doesn't support — not verbal polish.
Submission: Upload each deliverable as a separate file directly on the Provn platform: your Analysis & Recommendation Memo, your README document (Sections A, B, and C), and your video walkthrough (MP4 or MOV).
Demonstrate the ability to diagnose validity threats in an experiment result from imperfect, aggregate data
Translate a statistical finding into a specific, actionable business recommendation with an honest confidence statement
Show honest engagement with a result that complicates the "obvious" conclusion, not a smoothed-over success narrative
Connect a recommendation to a post-launch monitoring plan anchored to the scenario's own data
Show how you worked with AI to accelerate your process — including at least one moment where you redirected it
On this page
