The AI Feature That’s Losing Trust
Northline shipped the trip planner in a reduced form. Ten weeks in, here is what the data says. | | | | | | ------------------------------------------------ | ---------- | ---------- | ---------- | | Metric | Week 1 | Week 10 | Change | |…
Create a free account to upload your work. Your progress saves as a draft until you submit.
What You'll Be Doing
Northline shipped the trip planner in a reduced form. Ten weeks in, here is what the data says.
| Metric | Week 1 | Week 10 | Change |
| Weekly conversations | 3,900 | 11,400 | +192% |
| Conversations reaching a gear list | 71% | 78% | +7 pts |
| Add-to-cart rate from a gear list | 34% | 19% | −15 pts |
| Order completion from assistant-attributed carts | 61% | 58% | −3 pts |
| Support tickets mentioning the assistant | 22 | 310 | +1,309% |
| Assistant CSAT (1–5, n≈900/wk) | 4.1 | 3.2 | −0.9 |
| Repeat use within 30 days | 28% | 12% | −16 pts |
| Median response latency | 2.1s | 6.8s | +224% |
| Returns rate, assistant-attributed orders | 8% | 17% | +9 pts |
| Returns rate, site-wide | 9% | 9% | flat |
Two additional facts from the support queue: the three most common ticket themes are “recommended something out of stock” (41%), “the list was wrong for my trip” (26%), and “it was slow / I gave up” (18%). Separately, the team shipped a prompt change in week 6 to make responses “more helpful and complete”; average response length roughly doubled after that change.
Usage is up. Trust is down. Diagnose it and propose what to do.
Constraints to consider
- You cannot retrain or change the model vendor.
- The catalog stock feed refreshes every four hours and your team does not own it.
- One engineer for the next four weeks. The other two are on Black Friday infrastructure.
- Any rollback of the week-6 prompt change requires you to state what you would lose.
- No new instrumentation can land in under two weeks — if your diagnosis depends on data you don’t have, say so and say what you’d do in the meantime.
AI Usage Guidance
We expect you to use AI tools. We evaluate how you use them — not whether you use them. Evidence of iteration, redirection, and critical evaluation scores higher than a polished output with no process documentation.
The single highest-signal indicator: your video answer to the mandatory AI question. If you cannot name a specific moment where you redirected AI output, evaluators will assume you did not.
Mandatory AI question for your video: Walk me through one moment where you disagreed with, pushed back on, or redirected what the AI gave you — and what you did instead. Name the specific moment. Explain what the AI produced that didn’t meet the bar, what you did differently, and why.
Communication clarity matters for this role. We assess structure, stakeholder awareness, and ability to simplify complex ideas — not presentation style or accent.
What You'll Accomplish
Diagnose a decline in a model-powered product from a mixed quantitative and qualitative signal set
Separate correlation from cause and state what evidence would confirm the diagnosis
Design an eval that would have caught the regression before customers did
Prioritize a four-week plan against a single-engineer constraint
Communicate a diagnosis and a plan to a skeptical leadership audience
How Your Work Will Be Scored
What to Submit
File 1 — Diagnosis and Four-Week Plan
Format: .pdf, .doc, .docx, .rtf, .txt, .md
What is actually happening, ranked by contribution to the decline, and what the one engineer does for four weeks.
Sign in to upload files
File 2 — README Document
Format: .pdf, .doc, .docx, .rtf, .txt, .md
Three required sections:
- Section A — Written analysis: your causal argument. Which metric movements are causes, which are symptoms, and what you cannot yet distinguish. State the evidence that would confirm or kill your leading hypothesis. 350–450 words.
- Section B — The eval you would have run: design the evaluation that would have caught this before customers did. Specify the task set and roughly how many cases, what each case checks, the pass criteria, whether each check is automated or human-judged, and the regression bar that would have blocked the week-6 prompt change. Also name the one metric you would instrument that does not exist today, and why it is the one.
- Section C — AI Usage Log (Mandatory): This is not a trick. We want to see how you work with AI — not whether you used it. In a short section of your README, document your AI collaboration process. For each significant interaction with an AI tool, briefly note: what you asked the AI to help with / what it gave you / what you kept, changed, or rejected — and why. Three interactions documented is sufficient. The log does not need to be exhaustive.
Sign in to upload files
File 3 — Video Walkthrough
Format: .mp4, .mov, .webm
ecord as MP4 or MOV and upload directly on the Provn platform as a separate file. Cover: (1) your diagnosis in one sentence, ~30 sec; (2) the evidence chain behind it, ~90 sec; (3) your eval design and the regression bar, ~90 sec; (4) the mandatory AI question, ~60 sec; (5) what you’re least sure about, ~45 sec. Speak naturally — we assess reasoning, not verbal polish.
Sign in to upload files
Create a free account to upload your work. Your progress saves as a draft until you submit.
On this page