Set the Bar for Ask Marlowe Challenge

What You'll Be Doing

Marlowe Athletic wants to roll Ask Marlowe — the plain-English analytics assistant from the Medium challenge — out to all 84 club general managers. You own the recommendation and the readiness bar.

Here is the pilot record: six clubs, eight weeks.

1,840 questions asked. 1,470 (79.9%) returned an answer. 370 returned an error or a refusal.
The data team hand-audited 120 answers, sampled from the 1,470. Results: 89 correct (74%), 18 directionally right but with a wrong number (15%), 13 confidently wrong (11%).
Median answer latency 14 seconds; p95 41 seconds.
Inference cost: $0.31 per question.
Auditing one answer takes an analyst about 12 minutes, including reproducing the query.
Projected steady-state volume at 84 clubs: roughly 55 questions per GM per month — about 4,620 questions a month.
The data team is four people, roughly 640 working hours a month in total. They currently spend about 30% of that — 190 hours — answering ad-hoc GM questions by hand. That saving is the entire business case.
GM sentiment: 8 of 11 pilot GMs said they would “rather have a fast answer they can sanity-check than wait three days.”
One incident. The GM at Club 12 cut Saturday morning staffing based on an Ask Marlowe answer about attendance. The answer applied the wrong time zone and undercounted 6–9 a.m. bookings by 40%. Two Saturdays ran understaffed before anyone noticed.

Write the readiness bar and make the call.

Constraints to consider

The data team stays at four people through fiscal year end. No hiring, no contractors, no borrowing from another function.
You cannot change the model or the vendor. You can change the schema exposed to it, the prompt, the question types it will accept, what the answer displays, and how answers are verified.
Any number that appears in a board deck or an investor update must be produced by the data team, not by the assistant. No exceptions, no launch-then-formalise.
Your users are 84 general managers who are not analysts. Any guardrail you propose has to work without the GM understanding why it is there.
Decision in five weeks. Whatever launches must be operable inside the team’s existing 640 monthly hours, net of whatever you claim to save.

AI Usage Guidance

We expect you to use AI tools. We evaluate how you use them — not whether you use them. Evidence of iteration, redirection, and critical evaluation scores higher than a polished output with no process documentation.

The single highest-signal indicator: your video answer to the mandatory AI question. If you cannot name a specific moment where you redirected AI output, evaluators will assume you did not.

Mandatory AI question for your video: Walk me through one moment where you disagreed with, pushed back on, or redirected what the AI gave you — and what you did instead. Name the specific moment. Explain what the AI produced that didn’t meet the bar, what you did differently, and why.

Speak naturally. Communication is assessed on clarity of technical ideas and logical structure — not verbal polish, accent, or filler words.

What You'll Accomplish

Define readiness for an analytical AI system as numeric thresholds on named measures

Design an evaluation that produces a threshold rather than a general impression of quality

Treat different classes of wrong answer as different problems with different tolerances

Reason quantitatively about the cost of verification against a fixed analyst-hour budget

Defend a launch decision to a leadership audience with the arithmetic intact

How Your Work Will Be Scored

Evaluation Design & Readiness Bar Error Taxonomy & Verification Economics Strategic Judgment & Recommendation Video Walkthrough & Communication AI Fluency

What to Submit

File 1 — Readiness Brief

Document · 2 pages maximumRequired

Format: .pdf, .doc, .docx, .rtf, .txt, .md

Your readiness criteria with explicit numeric thresholds, your go / limited go / no-go recommendation, and the five-week plan behind it.

File 2 — README Document

DocumentRequired

Format: .pdf, .doc, .docx, .rtf, .txt, .md

Three required sections:

Section A — Written analysis: the verification economics. Show the arithmetic that gets from 4,620 questions a month to an audit policy the four-person team can actually run, and state what you are choosing not to check. 350–450 words.
Section B — Evaluation, thresholds, and failure handling: (i) the evaluation you would build before launch — what the question set is, roughly how many questions, how a correct answer is defined, who judges it, and how the set stays current; (ii) the numeric threshold that gates launch, and separately the threshold that would trigger a rollback after launch; (iii) the one class of wrong answer you consider unacceptable at any rate, and the mechanism — not the policy — that prevents it reaching a GM.
Section C — AI Usage Log (Mandatory): This is not a trick. We want to see how you work with AI — not whether you used it. In a short section of your README, document your AI collaboration process. For each significant interaction with an AI tool, briefly note: what you asked the AI to help with / what it gave you / what you kept, changed, or rejected — and why. Three interactions documented is sufficient. The log does not need to be exhaustive.

File 3 — Video Walkthrough

Video · 6-8 minutesRequired

Format: .mp4, .mov, .webm

Record as MP4 or MOV and upload directly on the Provn platform as a separate file. Cover: (1) your recommendation and the readiness bar, with at least one threshold number spoken aloud, ~60 sec; (2) the evaluation design and the gating threshold, ~2 min; (3) the verification arithmetic — how many answers get checked, by whom, and what that costs, ~90 sec; (4) the Club 12 incident and what stops the next one, ~60 sec; (5) the mandatory AI question, ~60 sec; (6) the residual risk you are accepting, ~30 sec. Speak naturally, as if briefing Marlowe’s leadership team — clarity of reasoning matters, verbal polish does not.