The Pilot That’s Stalling Challenge

What You'll Be Doing

Week 5 of the six-week Meridian Freight pilot. The draft-quote assistant is live in three service centers. The renewal decision is in eight days, and Dana presents to Meridian’s board the day after that.

Here is the telemetry.


Metric	Week 1	Week 4	Change
Quote requests routed to the assistant	610	2,240	+267%
Draft quotes produced	402 (66%)	1,940 (87%)	+21 pts coverage
Drafts approved by broker with no edit	51%	23%	−28 pts
Drafts edited, then sent	38%	44%	+6 pts
Drafts discarded	11%	33%	+22 pts
Median broker time-to-send	6.2 min	9.8 min	+58%
Median end-to-end quote turnaround	21 min	28 min	+33%
Quote win rate — assistant-assisted	26%	19%	−7 pts
Quote win rate — manual control group (60 brokers)	18%	19%	+1 pt
Drafts citing a spreadsheet lane older than 60 days	4%	29%	+25 pts
Kestrel API 429 (rate-limited) responses	0	1,830/week	—
p95 draft latency	8s	34s	+325%

Three things happened during the pilot that are not in the table:

In week 3 Meridian won a new shipper and expanded Kestrel from 4,100 to 5,600 contract lanes.
Also in week 3, Ron went on leave. The spreadsheet has not been updated since.
Broker feedback themes, collected across three service centers: “the accessorials are wrong” (Fresno, 22 mentions), “it quotes lanes we don’t run anymore” (Kansas City, 17), “at 9am it’s slower than just doing it myself” (all sites, 31).

Priya has said, in writing, “I told you the spreadsheet was a problem.”

Diagnose it and tell Dana what you are going to do with the three weeks you have left.

Constraints to consider

Three weeks, and you are the only engineer on it. Nobody is joining.
Still no write access to Kestrel, and a second API key would require a security review that takes six weeks. The 60/min limit is fixed for the rest of the engagement.
Any new data source must live inside Meridian’s VPC and be on their approved list.
You cannot get Ron back. Plan for the spreadsheet as it is.
Whatever you change, you must be able to tell Dana in one sentence what it does to the number she is putting in front of her board. If you can’t say it in one sentence, it isn’t in the plan.

AI Usage Guidance

We expect you to use AI tools. We evaluate how you use them — not whether you use them. Evidence of iteration, redirection, and critical evaluation scores higher than a polished output with no process documentation.

The single highest-signal indicator: your video answer to the mandatory AI question. If you cannot name a specific moment where you redirected AI output, evaluators will assume you did not.

Mandatory AI question for your video: Walk me through one moment where you disagreed with, pushed back on, or redirected what the AI gave you — and what you did instead. Name the specific moment. Explain what the AI produced that didn’t meet the bar, what you did differently, and why.

Speak naturally. Communication is assessed on clarity of technical ideas and logical structure — not verbal polish, accent, or filler words.

What You'll Accomplish

Build a causal chain from mixed telemetry and qualitative field signal to a ranked diagnosis

Separate cause from symptom, and say what the available data cannot distinguish

Design the check that would have caught the regression in week 2 instead of week 5

Re-scope a stalling engagement against a single-engineer constraint and a fixed deadline

Deliver bad news to a customer executive whose credibility is attached to the outcome, without losing the relationship

How Your Work Will Be Scored

Diagnosis & Re-Scoping AI System Design in a Customer Environment Commercial & Stakeholder Judgment Video Walkthrough & Customer Communication AI Fluency

What to Submit

File 1 — Diagnosis and Three-Week Plan

Document · 2 pages maximumRequired

Format: .pdf, .doc, .docx, .rtf, .txt, .md

What is actually happening, ranked by contribution to the decline, and exactly what you do with the three weeks.

File 2 — README Document

DocumentRequired

Format: .pdf, .doc, .docx, .rtf, .txt, .md

Three required sections:

Section A — Written analysis: your causal argument. Which movements are causes, which are downstream symptoms, and what the data as given cannot separate. State the evidence that would confirm or kill your leading explanation. 350–450 words.
Section B — The check you should have had, and the sentence you owe Dana: design the check that would have caught this in week 2. Specify what it samples and how often, what it asserts, the threshold that would have alerted you, whether it is automated or human-judged, and who would have received the alert. Then write the one sentence you will say to Dana about what your plan does to the number she is presenting — and, separately, the one sentence you will say if the plan does not land in time.
Section C — AI Usage Log (Mandatory): This is not a trick. We want to see how you work with AI — not whether you used it. In a short section of your README, document your AI collaboration process. For each significant interaction with an AI tool, briefly note: what you asked the AI to help with / what it gave you / what you kept, changed, or rejected — and why. Three interactions documented is sufficient. The log does not need to be exhaustive.

File 3 — Video Walkthrough

Video · 5-6 minutesRequired

Format: .mp4, .mov, .webm

Record as MP4 or MOV and upload directly on the Provn platform as a separate file. No ZIP files, no repository links, no third-party video platforms. Cover: (1) your diagnosis in one sentence, ~30 sec; (2) the evidence chain behind it, ~90 sec; (3) your revised three-week scope and what you tell Dana, ~90 sec; (4) the mandatory AI question, ~60 sec; (5) what you are least sure about, ~45 sec. Speak naturally — we assess reasoning, not verbal polish.