The 9% That Costs Money Challenge

What You'll Be Doing

You are the engineer who owns Deskmate at Halcyon Freight Systems. The validation problem from the Practice challenge got a proper look. Here is what the holdout set says.

Your team hand-labeled 200 real shipper emails and ran the extractor against them. Field-level results:


Field	Cases	Exact match	Wrong value	Missing
origin_zip	200	197	2	1
dest_zip	200	195	4	1
pickup_date	200	181	17	2
weight_lbs	200	172	21	7
commodity	200	188	9	3
special_handling	200	149	12	39
All six fields correct	200	182	—	—

The costs are not symmetric:

A wrong weight_lbs that crosses the 45,000 lb equipment threshold sends the wrong truck. Average dry-run charge: $850, and Halcyon eats it. Happened 11 times last quarter.
A wrong pickup_date is caught by the shipper before dispatch roughly 80% of the time. The other 20% is a missed pickup and an angry customer, no direct charge.
A missing special_handling (“tarped,” “team drivers,” “hazmat”) is the one nobody notices until the driver arrives. Two of these last quarter became claims. One was $6,200.
Wrong ZIPs are almost always caught by the routing engine downstream, which rejects unroutable lanes.

Design the gate. What gets auto-created, what gets held for a human, and what gets rejected outright.

Constraints to consider

Halcyon’s ops team is four people. They can review at most 120 emails a week on top of their existing work. Deskmate processes 2,800 a week — so your human-review budget is about 4.3% of volume.
The model endpoint Halcyon is on does not return token log-probabilities or any confidence score. There is no per-field confidence available from the vendor and you cannot change vendors. Anything that looks like a confidence number, you have to build.
Whatever you add cannot push p95 end-to-end latency above +400ms. Deskmate runs inline while a broker waits.
You cannot change POST /loads, which is still owned by the platform team and still not idempotent.
Half a sprint of one engineer. Ship one thing.

AI Usage Guidance

We expect you to use AI tools. We evaluate how you use them — not whether you use them. Evidence of iteration, redirection, and critical evaluation scores higher than a polished output with no process documentation.

The single highest-signal indicator: your video answer to the mandatory AI question. If you cannot name a specific moment where you redirected AI output, evaluators will assume you did not.

Mandatory AI question for your video: Walk me through one moment where you disagreed with, pushed back on, or redirected what the AI gave you — and what you did instead. Name the specific moment. Explain what the AI produced that didn’t meet the bar, what you did differently, and why.

Speak naturally. Communication is assessed on clarity of technical ideas and logical structure — not verbal polish, accent, or filler words.

What You'll Accomplish

Design a gating policy for a probabilistic component using asymmetric error costs rather than a single accuracy number

Build deterministic validation where no model confidence signal is available

Allocate a fixed human-review budget to the highest-cost error classes

Specify an evaluation that measures the gate itself, not just the extractor

Explain a risk-based engineering tradeoff to a mixed engineering and operations audience

How Your Work Will Be Scored

Engineering Execution Probabilistic Systems Design Engineering Judgment & Tradeoffs Video Walkthrough & Communication AI Fluency

What to Submit

File 1 — The Gate

Any FileRequired

Format: no restrictions

A single Python (or TypeScript) file containing your validation and routing logic — the function that takes an extracted load and returns one of auto_create / hold_for_review / reject, with the reason. It does not need to run against a real service, and stubs are fine. It does need to be code, not pseudocode, and the routing thresholds must be visible in it.

File 2 — README Document

DocumentRequired

Format: .pdf, .doc, .docx, .rtf, .txt, .md

Three required sections:

Section A — Written analysis: why your gate routes the way it does. Show the arithmetic: given your rules and the table above, roughly how many of the 2,800 weekly emails land in each bucket, and does that fit the 120-per-week review budget. Name the error class you are knowingly letting through and what it costs. 300–400 words.
Section B — Edge cases and measurement. Two parts.
- B1 — Edge-case reasoning (complete this part without AI assistance). List at least five inputs or conditions that would defeat your gate — cases where it says auto_create and should not have, or says hold_for_review on something perfectly fine. One line each on what happens. Write these from your own reasoning; we are reading for whether you can attack your own design. Note at the top of this block that it was written unaided.
- B2 — How you would measure the gate: what you would track weekly, the pass criterion for “the gate is working,” and how you would sample to get it. Include the metric that tells you the gate is holding back too much, not just too little.
Section C — AI Usage Log (Mandatory): This is not a trick. We want to see how you work with AI — not whether you used it. In a short section of your README, document your AI collaboration process. For each significant interaction with an AI tool, briefly note: what you asked the AI to help with / what it gave you / what you kept, changed, or rejected — and why. Three interactions documented is sufficient. The log does not need to be exhaustive.

File 3 — Video Walkthrough

Video · 4-5 minutesRequired

Format: .mp4, .mov, .webm

Record as MP4 or MOV and upload directly on the Provn platform as a separate file. Cover: (1) your routing policy in one sentence, ~45 sec; (2) a walk through the code and where the thresholds live, ~90 sec; (3) the arithmetic — what fraction goes to review and why that fits, ~60 sec; (4) the mandatory AI question, ~60 sec; (5) what you’re knowingly letting through, ~30 sec. Speak naturally — we assess the quality of your reasoning, not verbal polish.