The Launch Readiness Bar Challenge

What You'll Be Doing

Northline’s trip planner is being rebuilt as an agent: it can now read the customer’s order history, check live inventory, look up weather for the trip window, assemble a gear list, and place the order — taking actions on the customer’s behalf rather than only making suggestions.

It has been in internal use for six weeks and in early access with 400 customers for three. Leadership wants general availability in five weeks. You are the PM, and you own the go/no-go call.

Here is what you know:

On an internal 120-case task suite, the agent completes the full booking flow correctly in 82% of cases. Of the 18% failures, roughly half are wrong-item selections and half are incomplete carts.
Early access CSAT is 4.4, but 400 customers is a self-selected enthusiast group.
Median cost per completed conversation is $0.41. At projected GA volume that is roughly $47,000/month against a category the finance team budgeted $18,000/month for.
Median latency is 9 seconds; p95 is 34 seconds.
Legal has not yet signed off on the agent placing orders without an explicit confirmation step.
Two of the three engineers roll off to another team in six weeks, whatever you decide.

Write the readiness bar and make the call.

Constraints to consider

You cannot retrain the model or change vendors; you can change the prompt, the scaffolding, the retrieval, the guardrails, and the interface.
The $18,000/month budget is fixed for the quarter. Finance will not revisit it.
Whatever ships must be operable by one engineer after week six.
Any capability that places an order without confirmation requires documented legal sign-off before launch — no exceptions, no launch-then-fix.
You will present this to a leadership group that includes at least one person who thinks the whole agent direction is a mistake.

AI Usage Guidance

We expect you to use AI tools. We evaluate how you use them — not whether you use them. Evidence of iteration, redirection, and critical evaluation scores higher than a polished output with no process documentation.

The single highest-signal indicator: your video answer to the mandatory AI question. If you cannot name a specific moment where you redirected AI output, evaluators will assume you did not.

Mandatory AI question for your video: Walk me through one moment where you disagreed with, pushed back on, or redirected what the AI gave you — and what you did instead. Name the specific moment. Explain what the AI produced that didn’t meet the bar, what you did differently, and why.

Communication clarity matters for this role. We assess structure, stakeholder awareness, and ability to simplify complex ideas — not presentation style or accent.

Submission: Upload each deliverable as a separate file directly on the Provn platform.

What You'll Accomplish

Define launch readiness criteria for a probabilistic system, with thresholds rather than adjectives

Design an eval suite that measures agent task success, not just response quality

Reason explicitly about the capability / latency / cost / reliability tradeoff with real numbers

Build a governance and rollback position that survives legal and executive scrutiny

Defend a go/no-go decision to a leadership audience that includes a skeptic

How Your Work Will Be Scored

Launch Readiness & Eval Design Unit Economics & Reliability Tradeoffs Risk, Safety & Governance Video Walkthrough & Executive Communication AI Fluency

What to Submit

File 1 — Launch Readiness Brief

Document · 2 pages maximumRequired

Format: .pdf, .doc, .docx, .rtf, .txt, .md

Your readiness criteria with explicit thresholds, your go / limited-GA / no-go recommendation, and the five-week plan behind it.

File 2 — README Document

DocumentRequired

Format: .pdf, .doc, .docx, .rtf, .txt, .md

Three required sections:

Section A — Written analysis: the tradeoff argument. What you are trading against what, with the numbers attached. Include how you get from $47,000 to $18,000 per month, or why you would not launch at that volume. 350–450 words.
Section B — Readiness, evals, and rollback: (i) the eval suite you would run before GA — what the task suite measures, how big, how success is judged per case, and what threshold gates the launch; (ii) the one failure mode you consider unacceptable at any rate, and the guardrail that prevents it; (iii) your rollback plan — what the trigger is, who calls it, and how long it takes.
Section C — AI Usage Log (Mandatory): This is not a trick. We want to see how you work with AI — not whether you used it. In a short section of your README, document your AI collaboration process. For each significant interaction with an AI tool, briefly note: what you asked the AI to help with / what it gave you / what you kept, changed, or rejected — and why. Three interactions documented is sufficient. The log does not need to be exhaustive.

File 3 — Video Walkthrough

Video · 6–8 minutesRequired

Format: .mp4, .mov, .webm

Record as MP4 or MOV and upload directly on the Provn platform as a separate file. Cover: (1) your recommendation and the readiness bar in one minute; (2) the eval suite and the gating threshold, ~2 min; (3) the cost and latency tradeoff with the numbers, ~90 sec; (4) how you answer the person in the room who thinks the agent is a mistake, ~60 sec; (5) the mandatory AI question, ~60 sec; (6) what you’d do differently with more time, ~30 sec. Speak naturally, as if briefing your leadership team directly — strategic clarity matters, verbal polish does not.