Challenges/Arrivia/Full Stack Engineer, Software Engineer/Full Stack AI Engineer — Identity Risk Scorer

Full Stack AI Engineer — Identity Risk Scorer

You're joining the BuilderEx team at a travel loyalty company that operates identity and authentication for multiple partner brands. The team is mid-migration to a unified OAuth 2.0/OpenID Connect identity provider — three legacy auth…

Estimated Time:

40 minutes

Difficulty:Intermediate

Status:Not started

Start this challenge

Create a free account to upload your work. Your progress saves as a draft until you submit.

What You'll Be Doing

The problem you've been handed on your second week: The fraud team has flagged a pattern — 847 logins over the past 30 days succeeded (valid credentials, valid OIDC token issued) but showed anomalous post-auth behavior: unusual geolocation, device switches within minutes of login, then immediate redemption requests against high-value loyalty accounts.

The current system trusts a valid token completely. There is no post-authentication risk assessment.

Your manager asks you to build a risk-scoring middleware that sits in the post-authentication flow. After a valid OIDC token is issued, your middleware receives a login event payload and returns a structured risk_decision object that downstream systems use to determine whether to allow, step-up-authenticate, or block the session.

Constraints:

You cannot modify the OIDC token issuance flow — the identity provider is a managed system; your middleware hooks in after token issuance.
The explanation field must be readable by a compliance officer without translation — "risk score exceeds threshold" is not acceptable.
Scope your PoC to what's achievable in a two-week sprint — heuristic, rule-based, lightweight ML, or a combination is fine; it just needs to be functional, observable, and improvable.
Your team will own this in production, including on-call at 2am when the scorer starts generating false positives and blocking loyal platinum-tier members. Design accordingly.

AI Usage Guidance: Using AI tools is expected and encouraged — how you use AI is part of what's evaluated. Your video walkthrough must include this mandatory question: "Walk me through one moment where you disagreed with, pushed back on, or redirected what the AI gave you — and what you did instead. Name the specific moment. Explain what the AI produced that didn't meet the bar, what you did differently, and why."

What You'll Accomplish

Demonstrate ability to design a post-authentication risk scoring system that integrates with an existing OAuth 2.0/OIDC flow without modifying it

Build a risk scoring implementation that balances security signal accuracy with the real cost of false positives on legitimate users

Show production engineering judgment: code structure, error handling, and observability appropriate for an on-call ownership model

Design an API output that serves both downstream engineering systems and non-technical compliance audit requirements

Document architectural trade-offs, incident response thinking, and AI collaboration process

How Your Work Will Be Scored

IAM Architecture & SSO Execution — 27% — Middleware correctly positioned in the post-auth flow, OIDC constraint honored, latency and failure-mode trade-offs addressedAI/ML Risk Scoring Integration — 22% — Signal selection observable at runtime, explanation field serves a compliance officer, false positive cost explicitly consideredFull-Stack Production Engineering — 18% — Code structured for on-call ownership, error handling covers real failure modes, incident runbook provides actionable stepsDeveloper Experience & API Design — 13% — risk_decision output is self-documenting, Section A reads as a design document, system is extensible without a rewriteAI Fluency — 10% — AI Usage Log, Section B2 reasoning, and video AI question together demonstrate genuine judgment about when/how to use AIResume & Background — 10% — Evaluated separately from challenge artifacts

What to Submit

Risk Scorer Implementation

Any File · No page limit — code file(s) onlyRequired

Format: no restrictions

Your working risk-scoring middleware as a code file or files. Use Python, TypeScript, or Node.js (your choice).

Requirements:

Accept the sample event payload (or an equivalent structure of your design)
Evaluate at least 3 behavioral risk signals from the event data
Return a risk_decision object with risk_level, confidence, signals, and explanation fields
Include at least one unit test
A callable function/module with a clear entry point is sufficient — no full server or deployed infra required
Include a brief comment at the top of your entry file explaining how to run it

Sample input payload:

{
  "event_type": "auth_success",
  "user_id": "u_8821aa",
  "account_tier": "platinum",
  "session_id": "sess_94kz",
  "issued_at": 1751000000,
  "ip_address": "203.0.113.55",
  "geo_country": "US",
  "geo_city": "Chicago",
  "device_id": "dev_abc123",
  "device_type": "mobile",
  "user_agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 17_0)",
  "last_login": {
    "timestamp": 1750996200,
    "ip_address": "185.220.101.42",
    "geo_country": "RO",
    "geo_city": "Bucharest",
    "device_id": "dev_abc123"
  },
  "account_flags": {
    "recent_password_change": false,
    "failed_attempts_24h": 0,
    "redemption_pending": true
  }
}

Expected output shape:

{
  "risk_level": "low | medium | high",
  "confidence": "0.0-1.0",
  "signals": ["signal_name_1", "signal_name_2"],
  "explanation": "A plain-English explanation readable by a compliance officer."
}

README

Document · Section A: 300–500 words; no strict limit for B and CRequired

Format: .pdf, .doc, .docx, .rtf, .txt, .md

Three required sections:

Section A — Architecture Rationale (300–500 words)

Why did you place the middleware where you did in the post-auth flow?
Which signals did you choose and why? What signals did you consider and reject?
What does "medium risk" look like in your design, and what happens next — hard block, step-up challenge, something else?
What are the most important things you'd add or change with more time?

Section B — Production Ownership B1 (Incident Runbook): Precision drops overnight — 30% of flagged sessions are legitimate platinum-tier members getting blocked mid-booking. Write a runbook for the on-call engineer who gets the 2am alert. At least 3 concrete, actionable steps. First step should not be "wake up the author." B2 (AI Reasoning, complete without AI assistance): Describe a scenario where an AI coding assistant would give a plausible but incorrect implementation for this type of risk-scoring problem, and how you'd catch it before acting on it.

Section C — AI Usage Log (Mandatory) For each of at least 3 significant AI interactions, note: what you asked, what it gave you, and what you kept/changed/rejected and why.

Video Walkthrough

Video · 8-10 minutesRequired

Format: .mp4, .mov, .webm

8–10 minute video (MP4 or MOV). Do not submit a link to a video platform.

Structure:

Summary (60 sec): problem, approach, recommendation
Code walkthrough (3–4 min): signal logic, output structure, one key design decision
Section B walkthrough (2 min): incident runbook + AI reasoning answer
Mandatory AI question (1–2 min): the specific moment you pushed back on the AI's output
Reflection (30–60 sec): what you'd do differently or add with more time

Communication is assessed on clarity of technical ideas and logical structure — not verbal polish, accent, or filler words.

Start this challenge

Create a free account to upload your work. Your progress saves as a draft until you submit.

On this page