Challenge Library/Program Manager, Operations Manager/The Step That Shouldn’t Be Automated

The Step That Shouldn’t Be Automated

Merrow Field Services repairs commercial refrigeration and HVAC for grocery and restaurant chains. Every service call comes into one of six regional dispatch centers, where it gets a priority code and a trade code before a technician is…

AI ops
bite-sized
practice
human-in-the-loop
Estimated Time:
20 minutes
Difficulty:Intermediate
Status:Not started
Start this challenge

Create a free account to upload your work. Your progress saves as a draft until you submit.

What You'll Be Doing

Merrow Field Services repairs commercial refrigeration and HVAC for grocery and restaurant chains. Every service call comes into one of six regional dispatch centers, where it gets a priority code and a trade code before a technician is assigned. 

  • P1 — Emergency. Contractual response within 4 hours. Applies when product loss, food safety, or a store closure is at stake. 
  • P2 — Same day. 
  • P3 — Scheduled within 5 business days. 

Ten weeks ago Merrow put an AI triage assistant in front of that step. It reads the caller’s description, assigns the priority and trade codes, and routes the work order. It auto-assigns when its confidence is above a threshold and sends the rest to a human dispatcher. 

Here is where things stand. 

MeasureBeforeNow
Inbound service calls per week4,3004,300
Share auto-assigned with no human review0%81%
Median time from call to technician assigned14 min6 min
Dispatcher headcount across six centers3122
P1 miss rate (a true emergency coded P2 or P3)0.4%2.1%
P1 over-call rate (a non-emergency coded P1)1.1%0.9%
Trade code errors (wrong specialty dispatched)3.2%3.4%
SLA compliance on P1 response97%92%

Four calls the assistant handled last week, pulled from the exception review: 

Call A — “The walk-in in the back is running warm, maybe 50 degrees, we’ve got about $18,000 of product in there.” → coded P2, assigned next morning. Store lost the product. 

Call B — “Front door air curtain is making a noise.” → coded P3. Correct. 

Call C — “It’s fine, no rush, but the compressor on unit 4 has been cycling constantly since Tuesday and now the case temps are climbing.” → coded P3 by the assistant. A dispatcher happened to see it and re-coded to P1. Unit failed the next day. 

Call D — “Everything’s down, the whole store, we have nothing.” → coded P1. Correct. 

You are the program manager who owns this workflow. Write a short note to the VP of Dispatch. 

Constraints to consider 

  • Headcount is frozen. You cannot add dispatchers back, and the 22 you have are already scheduled across a 24/7 rota with three people on overnight covering all six regions. 
  • The triage model is a vendor product. You cannot retrain it or change vendors. You can change the confidence threshold, the routing rules around it, and what the dispatcher sees. 
  • Your engineering partner has capacity for one change this month. One, not three. 
  • Anything you propose must work at 2am with three people on shift — if it creates review volume, say how much and who absorbs it. 

AI Usage Guidance 

We expect you to use AI tools. We evaluate how you use them — not whether you use them. Evidence of iteration, redirection, and critical evaluation scores higher than a polished output with no process documentation. 

The single highest-signal indicator: your video answer to the mandatory AI question. If you cannot name a specific moment where you redirected AI output, evaluators will assume you did not. 

Mandatory AI question for your video: Walk me through one moment where you disagreed with, pushed back on, or redirected what the AI gave you — and what you did instead. Name the specific moment. Explain what the AI produced that didn’t meet the bar, what you did differently, and why. 

Communication clarity matters for this role. We assess structure, stakeholder awareness, and ability to simplify complex ideas — not presentation style or accent. 

Submission: Upload each deliverable as a separate file directly on the Provn platform.

What You'll Accomplish

Distinguish an automation accuracy problem from a routing and threshold problem

Identify which step in an automated workflow needs a human returned to it, and justify the choice by cost of failure rather than by error rate

Specify a control with a threshold, a trigger, and an owner

Size the operational load a proposed control creates against a fixed staffing constraint

Explain an operational diagnosis to a functional leader in under four minutes

How Your Work Will Be Scored

Operational Diagnosis & Human-in-the-Loop Judgment: Separates the error classes by consequence rather than frequency, and puts the human back at the step where being wrong is most expensive Control & Measurement Design: Specifies a control an operator could implement — trigger, threshold, owner — and sizes the load it creates Video Walkthrough & Communication: Explains the diagnosis to a functional leader clearly and in sequence AI Fluency: Shows directed, iterative, critically evaluated AI use

What to Submit

File 1 — Dispatch Note

Document · 250 words max, one pageRequired

Format: .pdf, .doc, .docx, .rtf, .txt, .md

Your diagnosis. What is actually going wrong, which single step you would put a human back into (or which routing rule you would change), and what it costs.

Sign in to upload files

File 2 — README Document

DocumentRequired

Format: .pdf, .doc, .docx, .rtf, .txt, .md

Three required sections: 

  • Section A — Written analysis: the one change you would ship this month, and what you are consciously not fixing. State the review volume your change creates and who absorbs it at 2am. 150–250 words. 
  • Section B — Your control: write the control as an operator would implement it. State the trigger condition, the threshold or rule, what happens when it fires, who owns it, and the one number you would watch weekly to know whether it worked. 
  • Section C — AI Usage Log (Mandatory): This is not a trick. We want to see how you work with AI — not whether you used it. In a short section of your README, document your AI collaboration process. For each significant interaction with an AI tool, briefly note: what you asked the AI to help with / what it gave you / what you kept, changed, or rejected — and why. Three interactions documented is sufficient. The log does not need to be exhaustive.

Sign in to upload files

File 3 — Video Walkthrough

Video · 3–4 minutesRequired

Format: .mp4, .mov, .webm

Record as MP4 or MOV and upload directly on the Provn platform as a separate file.

Cover: (1) what is going wrong and why the P1 miss rate matters more than the trade code error rate, ~60 sec; (2) your one change and the control behind it, ~60 sec; (3) the mandatory AI question, ~60 sec; (4) what you would check next with more time, ~30 sec. Speak naturally — we’re assessing your thinking, not verbal polish.

Sign in to upload files