Challenge Library/Software Engineer/Diagnose Silent Order Failures in a Live E-Commerce Pipeline

Diagnose Silent Order Failures in a Live E-Commerce Pipeline

You are a senior engineer on the platform team at a mid-size e-commerce company. Your team owns the order processing pipeline — the system that takes a confirmed checkout, validates payment, assigns inventory, routes to a fulfillment…

software

engineer

senior

Estimated Time:

1 hour

Difficulty:Intermediate

Status:Not started

Start this challenge

Create a free account to upload your work. Your progress saves as a draft until you submit.

What You'll Be Doing

For the past three weeks, customers have been experiencing silent failures on the confirmation step: payment is captured and inventory is reserved, but no confirmation email arrives and the order status page shows “Processing” indefinitely. The issue affects roughly 4–6% of orders during peak hours (evenings and weekends). Affected customers contact support, often hours later. Some cancel and repurchase, causing duplicate inventory reservations.

Your engineering manager has asked you to investigate, propose a fix, and ship the first meaningful step within four weeks.

Starter Code

The following Python service handles the post-payment confirmation step. Read it carefully — it may contain issues beyond the primary incident.

# order_confirmation_service.py

import json, boto3, smtplib, logging
from email.mime.text import MIMEText

sqs = boto3.client('sqs', region_name='us-east-1')
QUEUE_URL = 'https://sqs.us-east-1.amazonaws.com/123456789/order-confirmations'
SMTP_HOST, SMTP_PORT = 'smtp.internal.example.com', 587

def poll_and_process():
    while True:
        response = sqs.receive_message(
            QueueUrl=QUEUE_URL,
            MaxNumberOfMessages=10,
            WaitTimeSeconds=20
        )
        messages = response.get('Messages', [])
        for msg in messages:
            order = eval(msg['Body'])  # parse order payload
            send_confirmation(order)
            sqs.delete_message(
                QueueUrl=QUEUE_URL,
                ReceiptHandle=msg['ReceiptHandle']
            )

def send_confirmation(order):
    body = f"Hi {order['customer_name']}, your order {order['order_id']} is confirmed!"
    msg = MIMEText(body)
    msg['Subject'] = 'Order Confirmed'
    msg['From'] = 'orders@example.com'
    msg['To'] = order['customer_email']
    with smtplib.SMTP(SMTP_HOST, SMTP_PORT) as server:
        server.sendmail('orders@example.com', [order['customer_email']], msg.as_string())

if __name__ == '__main__':
    poll_and_process()

CONSTRAINTS

Honor all four constraints. A strong submission will address each one explicitly. Generic solutions that ignore these constraints will score lower regardless of technical quality.

Infrastructure: Your stack is AWS (SQS, Lambda, CloudWatch, SES, RDS). You do not have budget or approval to introduce new infrastructure services (no Kafka, no new databases, no third-party observability platforms). Work within what exists.
External dependency: The fulfillment routing service is a third-party vendor API. You can observe its inputs and outputs but cannot modify its behavior or access its internals. Your solution must treat it as a black box.
Scope: Your first deliverable must be shippable in four weeks by a team of two engineers. Identify what is in scope for that window and what comes later. Do not propose a six-month rewrite.
Ownership: Your team will own this service in production, including overnight incidents. Whatever you build, you are on call for it. Design accordingly.

How Your Work Will Be Scored

Technical Execution & Code Quality - 30%Solution Design & Trade-offs - 25%Production Ownership - 15%Communication & Collaboration - 15%AI Fluency - 15%

What to Submit

Fixed & Improved Service Code

DocumentRequired

Format: .pdf, .doc, .docx, .rtf, .txt, .md

Produce a corrected and improved version of order_confirmation_service.py that addresses the bugs you identify.
- Your code does not need to be production-ready — it should be a clear, well-reasoned proof-of-concept that demonstrates your approach.
- Include inline comments where you made a deliberate design decision or departed from the original structure.
- Aim for code that a teammate could review and understand without a separate explanation.

README

DocumentRequired

Format: .pdf, .doc, .docx, .rtf, .txt, .md

Your README must contain exactly three sections:

Section A — Diagnosis & Recommendation (300–500 words)

What is causing the 4–6% silent failure rate? Walk through your diagnosis.
What is your recommended fix? Explain why you chose this approach over alternatives.
What trade-offs does your approach introduce? Name at least one thing you are giving up.
What risks remain after your four-week fix? What would you address in the next phase?

Section B — Production Readiness

Part B1 — Incident Runbook

Write a brief runbook for the on-call engineer who gets paged for this service at 2am.
Include: how to detect the issue, how to triage it quickly, and the immediate mitigation step available before a code fix is deployed.
Assume the on-call engineer is familiar with AWS but has not worked on this service before.

Part B2 — Required Reasoning Question (answer without AI assistance)

Describe a scenario where an AI coding assistant would give you a plausible but incorrect answer for this type of problem — and explain specifically how you would catch it. What would the incorrect output look like, and what would you check to identify the error before acting on it?

Section C — AI Usage Log (Mandatory)

This is not a trick. We want to see how you work with AI — not whether you used it.

In a short section of your README, document your AI collaboration process.

For each significant interaction with an AI tool, briefly note:

What you asked the AI to help with
What it gave you
What you kept, changed, or rejected — and why

Three interactions documented is sufficient. The log does not need to be exhaustive.

Video Walkthrough

Video · 8–10 minutesRequired

Format: .mp4, .mov, .webm

Record your walkthrough as an MP4 or MOV file and upload it directly on the Provn platform as a separate file.

Structure your video around these five sections:

Opening (60 seconds): State the core problem and your recommended fix in plain language. No jargon yet — explain it as you would to a non-technical stakeholder.
Code walkthrough (2–4 minutes): Walk through your fixed code. Explain the key decisions, not just what you changed. Why did you make each choice? What did you consider and reject?
Runbook walkthrough (1–2 minutes): Walk through your Section B1 runbook. How would an on-call engineer use it at 2am?
Mandatory AI question (1–2 minutes): Answer this question directly:

“Walk me through one moment where you disagreed with, pushed back on, or redirected what the AI gave you — and what you did instead. Name the specific moment. Explain what the AI produced that didn’t meet the bar, what you did differently, and why.”

Reflection (30–60 seconds): What would you do differently if you had more time? Be specific — name the actual decision you’d revisit.

Speak naturally. Communication is assessed on clarity of technical ideas and logical structure — not verbal polish, accent, or filler words.

Start this challenge

Create a free account to upload your work. Your progress saves as a draft until you submit.

On this page