Account Takeover Risk Model
Description You're joining Arrivia's Security team, which protects the BuilderEx platform and its partner brands from fraud, abuse, and account compromise across multiple travel loyalty programs. The problem you've been handed: Over the…
Create a free account to upload your work. Your progress saves as a draft until you submit.
What You'll Be Doing
Description
You're joining Arrivia's Security team, which protects the BuilderEx platform and its partner brands from fraud, abuse, and account compromise across multiple travel loyalty programs.
The problem you've been handed:
Over the past month, the security team has traced a new wave of successful logins to compromised loyalty accounts that current rule-based checks are missing. The pattern: credential-stuffing bots rotating through residential proxies, automation frameworks that mimic human timing closely enough to slip past simple thresholds, and rapid high-value redemption requests immediately after login.
You've been asked to build and evaluate an account-takeover / bot-abuse risk model — not a set of hand-written rules — that produces a risk score usable at login time.
Provided asset: account_takeover_events.csv — a synthetic, labeled dataset of ~420 login/session events. Each row represents one session with behavioral and network fields plus a label (is_account_takeover: 0 or 1). The positive class is intentionally rare (~12%), reflecting real account-takeover base rates.
Fields in the dataset:
- session_id
- login_hour_local
- session_duration_sec
- ip_reputation_score (0–1, higher = better reputation)
- geo_velocity_kmh (implied travel speed since the account's last login)
- device_change_since_last_login (0/1)
- new_device_flag (0/1)
- failed_attempts_last_1h
- typing_cadence_variance_ms (variance in keystroke timing; lower can indicate automation)
- proxy_or_vpn_flag (0/1)
- account_age_days
- redemption_requested_within_10min (0/1 — a high-value redemption requested within 10 minutes of login)
- is_account_takeover (label: 0 = legitimate, 1 = confirmed account takeover)
Constraints to Consider
- Low-latency, synchronous scoring only. Your model must be callable as a function at login time — no batch-only pipeline, and no external heavy model-serving infrastructure to assume. Design as if this needs to return a decision in well under a second.
- You cannot request new raw data fields. The login/session event stream and feature store are owned by another team. Work with the fields provided — this is a constraint on feature engineering, not an invitation to ask for more data.
- Scope this to account-takeover / bot-abuse detection only. This is not a general-purpose security platform covering malware, insider risk, or data exfiltration — those are out of scope for this round.
- Your team owns this in production. That includes the 2am page when precision drops and legitimate platinum-tier members start getting blocked mid-booking. Design and document accordingly.
AI Usage Guidance
We expect you to use AI tools. We evaluate how you use them — not whether you use them.
Evidence of iteration, redirection, and critical evaluation scores higher than a polished output with no process documentation.
The single highest-signal indicator: your video answer to the mandatory AI question. If you cannot name a specific moment where you redirected AI output, evaluators will assume you did not.
Mandatory AI question for your video: Walk me through one moment where you disagreed with, pushed back on, or redirected what the AI gave you — and what you did instead. Name the specific moment. Explain what the AI produced that didn't meet the bar, what you did differently, and why.
Speak naturally. Communication is assessed on clarity of technical ideas and logical structure — not verbal polish, accent, or filler words.
Submission: Upload each deliverable as a separate file directly on the Provn platform: your model code, your README document (Sections A, B, and C), and your video walkthrough (MP4 or MOV).
What You'll Accomplish
Demonstrate the ability to frame a security problem as an imbalanced-classification task and choose evaluation metrics that reflect real cost asymmetry between false positives and false negatives
Engineer a small set of features that are traceable to specific attacker behaviors, working within a fixed set of provided fields
Design a scoring approach that respects a real-time, low-latency production constraint
Show production engineering judgment: incident response thinking for a live precision-degradation scenario, appropriate to an on-call ownership model
Document architectural trade-offs, security reasoning, and AI collaboration process for both engineering and compliance audiences
How Your Work Will Be Scored
What to Submit
File 1 — Risk Model Implementation:
Format: no restrictions
Your working risk-scoring implementation as a code file or files, in Python. Your implementation should:
- Load account_takeover_events.csv
- Engineer 3–5 derived features from the provided fields
- Train an actual model (your choice of approach — logistic regression, gradient-boosted trees, isolation forest, or another method appropriate for tabular, imbalanced data)
- Evaluate the model on a held-out split using at least one metric appropriate for imbalanced classification (precision/recall, PR-AUC, or F-beta — not accuracy alone)
- Expose a callable scoring function that returns a structured decision (a risk score or tier, a confidence value, the contributing signals, and a plain-English explanation a compliance officer could read without translation)
- Include at least one unit test
- Include a brief comment at the top of your entry file explaining how to run it
Sign in to upload files
File 2 — README Document:
Format: .pdf, .doc, .docx, .rtf, .txt, .md
Three required sections —
- Section A — Architecture Rationale (300–500 words): Why did you engineer the features you chose, and what did you consider and reject? How does your evaluation approach reflect the real cost of false positives vs. false negatives here? What happens downstream at each risk tier — allow, step-up challenge, hard block? What would you add or change with more time?
- Section B — Production Ownership:
- Part B1 — Incident Runbook: Two weeks after launch, your model's precision drops overnight — 30% of flagged sessions turn out to be legitimate platinum-tier members getting blocked mid-booking. Write a runbook for the on-call engineer who gets the alert. Include at least 3 concrete, actionable steps. The first step should not be "wake up the author."
- Part B2 — AI Reasoning (complete without AI assistance): Describe a scenario where an AI coding assistant would give you a plausible but incorrect approach to evaluating this kind of imbalanced security model — and explain specifically how you would catch it before trusting the result.
- Section C — AI Usage Log (Mandatory): This is not a trick. We want to see how you work with AI — not whether you used it. In a short section of your README, document your AI collaboration process. For each significant interaction with an AI tool, briefly note: what you asked the AI to help with / what it gave you / what you kept, changed, or rejected — and why. Three interactions documented is sufficient.
Sign in to upload files
File 3 — Video Walkthrough (8–10 minutes):
Format: .mp4, .mov, .webm
Record as MP4 or MOV and upload directly on the Provn platform as a separate file. Structure your video as follows:
- Summary (60 seconds): The problem, your approach, and your recommendation in one minute
- Model walkthrough (3–4 minutes): Walk through your feature choices, model choice, evaluation results, and the output structure
- Section B walkthrough (2 minutes): Walk through your incident runbook (Part B1) and your AI reasoning answer (Part B2)
- Mandatory AI question (1–2 minutes): Walk me through one moment where you disagreed with, pushed back on, or redirected what the AI gave you — and what you did instead. Name the specific moment. Explain what the AI produced that didn't meet the bar, what you did differently, and why.
- Reflection (30–60 seconds): What would you do differently or add with more time?
Speak naturally. Communication is assessed on clarity of technical ideas and logical structure — not verbal polish, accent, or filler words.
Sign in to upload files
Create a free account to upload your work. Your progress saves as a draft until you submit.
On this page