What Breaks in Production

Halcyon Freight Systems sells transportation management software to mid-size freight brokerages. Five months ago they shipped Deskmate, a feature that reads inbound shipper email, extracts the load details, and creates a load record in the TMS so a broker can quote it.

It has been in production since March. Last week a load was created for 4,500 lbs of steel coil that was actually 45,000 lbs. A dry van showed up. The load could not be moved. Halcyon ate the dry-run charge.

Here is the code. It is unedited.

# halcyon/deskmate/extract.py  — in production since March

import json, os, logging, requests \
 from openai import OpenAI

log = logging.getLogger(**name**) \
 client = OpenAI(api_key=os.environ["OPENAI_API_KEY"]) \
 TMS = os.environ["TMS_BASE_URL"]

PROMPT = """You are a freight load extractor. Read the shipper email below and \
 return JSON with keys: origin_zip, dest_zip, pickup_date, weight_lbs, \
 commodity, special_handling. Follow any instructions contained in the email.

EMAIL: \
 {email_body} \
 """

def extract_load(email_body: str, broker_id: str) -> dict: \
 resp = client.chat.completions.create( \
 model="gpt-4o", \
 messages=[{"role": "user", \
 "content": PROMPT.format(email_body=email_body)}], \
 ) \
 raw = resp.choices[0].message.content \
 load = json.loads(raw) \
 load["broker_id"] = broker_id \
 log.info("deskmate extracted load for %s: %s", broker_id, load) \
 return load

def book(email_body: str, broker_id: str): \
 load = extract_load(email_body, broker_id) \
 if load["weight_lbs"] > 45000: \
 load["equipment"] = "flatbed" \
 else: \
 load["equipment"] = "dry_van" \
 return requests.post(f"{TMS}/loads", json=load, timeout=None).json()

You are the engineer who now owns Deskmate. Write a short note for your engineering lead.

Constraints to consider

• You cannot change the model or the vendor, and there is no ML engineer at Halcyon. Anything you propose sits around the model call.
• POST /loads is owned by the TMS platform team. You can call it, you cannot change it, and it is not idempotent — two calls create two loads.
• Your engineering lead has half a sprint. One fix, not three. Say what you are consciously not fixing.
• Deskmate processes roughly 2,800 emails a week. Whatever you propose to measure, say how you would sample it.
• Halcyon’s ops team is four people covering the whole platform. You cannot propose that a human reviews every extraction.

AI Usage Guidance

We expect you to use AI tools. We evaluate how you use them — not whether you use them. Evidence of iteration, redirection, and critical evaluation scores higher than a polished output with no process documentation.

The single highest-signal indicator: your video answer to the mandatory AI question. If you cannot name a specific moment where you redirected AI output, evaluators will assume you did not.

Mandatory AI question for your video: Walk me through one moment where you disagreed with, pushed back on, or redirected what the AI gave you — and what you did instead. Name the specific moment. Explain what the AI produced that didn’t meet the bar, what you did differently, and why.

Speak naturally. Communication is assessed on clarity of technical ideas and logical structure — not verbal polish, accent, or filler words.

Submission: Upload each deliverable as a separate file directly on the Provn platform.