The Agent Got Slower and Dumber

Deskmate is an agent now. It has four tools — search_rate_guidance, lookup_shipper_history, check_carrier_capacity, create_load_draft — and it runs a loop until it can produce a quote draft for a broker to approve.

It shipped eight weeks ago. Here is the data.

Metric	Week 1	Week 8	Change
Emails processed / week	2,800	4,100	+46%
Drafts accepted by broker with no edit	64%	41%	−23 pts
Median tool calls per email	3.1	7.4	+139%
p50 end-to-end latency	4.2s	11.9s	+183%
p95 end-to-end latency	12s	74s	+517%
Median input tokens per email	9,400	38,600	+311%
Cost per processed email	$0.031	$0.147	+374%
Loop hit the 12-iteration cap	0.4%	9.6%	+9.2 pts
Extraction field accuracy (200-case holdout)	91%	88%	−3 pts
Emails escalated to the ops queue	4.1%	14.8%	+10.7 pts
Dry-run charges attributed to Deskmate	$1,700/mo	$9,400/mo	+453%

Two changes shipped during the window, both by engineers no longer on the team:

• Week 5: retrieval for search_rate_guidance changed from top-3 chunks to top-10 “for better recall.” No eval was run.
• Week 6: a new tool, lookup_shipper_history, was added. It returns the shipper’s last 50 loads as raw JSON.

Here is the loop.

# halcyon/deskmate/agent.py 
MAX_ITERS = 12 
 
def run(email_body: str, broker_id: str) -> dict: 
    messages = [ 
        {"role": "system", "content": SYSTEM_PROMPT}, 
        {"role": "user", "content": email_body}, 
    ] 
    for i in range(MAX_ITERS): 
        resp = client.chat.completions.create( 
            model="gpt-4o", messages=messages, tools=TOOL_SCHEMAS 
        ) 
        msg = resp.choices[0].message 
        messages.append(msg) 
        if not msg.tool_calls: 
            return {"draft": msg.content, "iters": i} 
        for call in msg.tool_calls: 
            result = TOOLS[call.function.name](**json.loads(call.function.arguments)) 
            messages.append({ 
                "role": "tool", 
                "tool_call_id": call.id, 
                "content": json.dumps(result),   # raw, untruncated 
            }) 
    log.warning("deskmate hit iteration cap for %s", broker_id) 
    return {"draft": None, "iters": MAX_ITERS} 

Usage is up. Everything else is down. Diagnose it and say what you would do.

Constraints to consider

• You cannot change the model or the vendor, and there is still no ML engineer at Halcyon.
• search_rate_guidance calls a service owned by the pricing team. It returns in ~400ms at p50 and you cannot change it or its response shape.
• One engineer for the next three weeks. The rest of the team is on the carrier-portal migration and will not be pulled off it.
• Finance has capped Deskmate at $0.05 per processed email by end of quarter. That number is fixed and is not being renegotiated.
• If you roll back the week-5 retrieval change, you must state specifically what capability is lost.
• No new instrumentation can ship in under two weeks. If your diagnosis depends on data you do not have, say so and say what you would do meanwhile.

AI Usage Guidance

We expect you to use AI tools. We evaluate how you use them — not whether you use them. Evidence of iteration, redirection, and critical evaluation scores higher than a polished output with no process documentation.

The single highest-signal indicator: your video answer to the mandatory AI question. If you cannot name a specific moment where you redirected AI output, evaluators will assume you did not.

Mandatory AI question for your video: Walk me through one moment where you disagreed with, pushed back on, or redirected what the AI gave you — and what you did instead. Name the specific moment. Explain what the AI produced that didn’t meet the bar, what you did differently, and why.

Speak naturally. Communication is assessed on clarity of technical ideas and logical structure — not verbal polish, accent, or filler words.