Diagnose Silent Order Failures in a Live E-Commerce Pipeline

You are a senior engineer on the platform team at a mid-size e-commerce company. Your team owns the order processing pipeline — the system that takes a confirmed checkout, validates payment, assigns inventory, routes to a fulfillment center, and emits the order confirmation to the customer.

For the past three weeks, customers have been experiencing silent failures on the confirmation step: payment is captured and inventory is reserved, but no confirmation email arrives and the order status page shows “Processing” indefinitely. The issue affects roughly 4–6% of orders during peak hours (evenings and weekends). Affected customers contact support, often hours later. Some cancel and repurchase, causing duplicate inventory reservations.

Your engineering manager has asked you to investigate, propose a fix, and ship the first meaningful step within four weeks.

Starter Code

The following Python service handles the post-payment confirmation step. Read it carefully — it may contain issues beyond the primary incident.

# order_confirmation_service.py

import json, boto3, smtplib, logging
from email.mime.text import MIMEText

sqs = boto3.client('sqs', region_name='us-east-1')
QUEUE_URL = 'https://sqs.us-east-1.amazonaws.com/123456789/order-confirmations'
SMTP_HOST, SMTP_PORT = 'smtp.internal.example.com', 587

def poll_and_process():
    while True:
        response = sqs.receive_message(
            QueueUrl=QUEUE_URL,
            MaxNumberOfMessages=10,
            WaitTimeSeconds=20
        )
        messages = response.get('Messages', [])
        for msg in messages:
            order = eval(msg['Body'])  # parse order payload
            send_confirmation(order)
            sqs.delete_message(
                QueueUrl=QUEUE_URL,
                ReceiptHandle=msg['ReceiptHandle']
            )

def send_confirmation(order):
    body = f"Hi {order['customer_name']}, your order {order['order_id']} is confirmed!"
    msg = MIMEText(body)
    msg['Subject'] = 'Order Confirmed'
    msg['From'] = 'orders@example.com'
    msg['To'] = order['customer_email']
    with smtplib.SMTP(SMTP_HOST, SMTP_PORT) as server:
        server.sendmail('orders@example.com', [order['customer_email']], msg.as_string())

if __name__ == '__main__':
    poll_and_process()

CONSTRAINTS

Honor all four constraints. A strong submission will address each one explicitly. Generic solutions that ignore these constraints will score lower regardless of technical quality.

• Infrastructure: Your stack is AWS (SQS, Lambda, CloudWatch, SES, RDS). You do not have budget or approval to introduce new infrastructure services (no Kafka, no new databases, no third-party observability platforms). Work within what exists.

• External dependency: The fulfillment routing service is a third-party vendor API. You can observe its inputs and outputs but cannot modify its behavior or access its internals. Your solution must treat it as a black box.

• Scope: Your first deliverable must be shippable in four weeks by a team of two engineers. Identify what is in scope for that window and what comes later. Do not propose a six-month rewrite.

• Ownership: Your team will own this service in production, including overnight incidents. Whatever you build, you are on call for it. Design accordingly.