Senior Big Data Developer/Big Data Developer Challenge
The Scenario You are joining the Financial Crime Data Engineering team at a large financial institution. The team builds and maintains the data pipelines that feed the institution's anti-money laundering (AML) and fraud detection systems.…
Create a free account to upload your work. Your progress saves as a draft until you submit.
What You'll Be Doing
The Scenario
You are joining the Financial Crime Data Engineering team at a large financial institution. The team builds and maintains the data pipelines that feed the institution's anti-money laundering (AML) and fraud detection systems. These pipelines ingest transaction data from multiple source systems, enrich it with entity-level behavioral profiles, and deliver structured outputs to downstream case management and regulatory reporting platforms.
The institution processes approximately 3.5 million financial transactions per hour across retail banking, wire transfers, ACH, and card networks. All data processing runs on Google Cloud Platform — Dataproc clusters for Spark workloads, BigQuery for analytical queries and historical lookups, and Cloud Storage for raw data landing and intermediate staging.
The Problem
A recent regulatory examination identified a gap in the institution's transaction monitoring coverage: the current pipeline does not adequately detect structuring behavior — patterns where individuals deliberately split transactions to stay below the $10,000 Currency Transaction Report (CTR) threshold. The existing pipeline processes individual transactions in isolation. It does not build rolling behavioral windows at the entity level that would reveal whether a customer has made multiple deposits of $9,500 across different branches over a three-day period.
The compliance team has mandated a 90-day remediation timeline. Your team has been assigned to design and build a new pipeline module — the Structuring Detection Enrichment Layer — that sits between raw transaction ingestion and the downstream case management platform. This module must:
- Aggregate transactions into rolling entity-level windows (24-hour, 72-hour, and 30-day periods) to surface structuring patterns
- Enrich each transaction record with the entity's aggregated behavior profile (total volume, frequency, amount distribution, branch diversity)
- Produce output records conforming to the downstream case management platform's fixed input schema (you cannot modify this schema)
- Maintain full data lineage from source transaction to enriched output for regulatory audit purposes
What You Know About the Current System
- Source data lands in Cloud Storage as partitioned Parquet files (partitioned by
transaction_dateandsource_system) - Entity resolution has already been performed upstream — each transaction record includes a resolved
entity_id - The existing daily batch pipeline runs on a 12-node Dataproc cluster (n1-highmem-16 instances) and currently completes in approximately 3.5 hours
- Historical transaction data for lookback windows lives in BigQuery, partitioned by
transaction_date, clustered byentity_id - The downstream case management platform consumes data via a fixed API contract (JSON-over-HTTPS, max 5,000 records per batch call, 200ms timeout per call)
- Compliance requires that all intermediate data artifacts be retained for 7 years and be reproducible from source
Constraints
Honor the following constraints in your solution. These reflect the real operating environment for this role.
-
GCP infrastructure only: Your solution must use the existing GCP stack — Dataproc (Spark), BigQuery, and Cloud Storage. Do not propose migrating to a different cloud provider or introducing tools not already in the environment (e.g., no Kafka, no Snowflake, no Databricks). Work within what exists.
-
Fixed downstream API contract: The case management platform consumes enriched records via a fixed API (JSON-over-HTTPS, max 5,000 records per batch call, 200ms timeout). You cannot modify this contract. Your output stage must conform to it.
-
4-week first deliverable: The regulatory timeline is 90 days, but your first sprint deliverable must be scoped to 4 weeks. Identify what you would deliver first and what comes later — do not propose a 90-day monolithic build.
-
Audit trail non-negotiable: Every enriched output record must be traceable back to its source transactions. No transformations that lose lineage. Compliance requires 7-year retention of all intermediate artifacts. This is a regulatory requirement, not a nice-to-have.
-
Existing cluster resources: Assume the existing 12-node Dataproc cluster (n1-highmem-16). You may propose configuration changes or scaling recommendations, but your proof-of-concept must demonstrate it can run within this resource envelope. No unlimited-compute assumptions.
What You'll Accomplish
Design a multi-window aggregation pipeline that computes rolling entity-level behavioral profiles (24-hour, 72-hour, and 30-day windows) over high-volume transaction data using PySpark on Dataproc, demonstrating your ability to move beyond row-level processing to stateful, entity-centric analytics.
Integrate batch and historical data sources by orchestrating reads from both Cloud Storage (raw Parquet ingestion) and BigQuery (historical lookback), managing the handoff between landing-zone data and warehoused history within a single coherent pipeline module.
Produce schema-compliant enriched output that conforms to a fixed downstream API contract you cannot change — requiring you to map your enrichment logic to an external system's expectations while handling batching constraints, payload limits, and delivery reliability.
Implement end-to-end data lineage and auditability so that every enriched output record can be traced back to its constituent source transactions, satisfying a 7-year regulatory retention requirement and demonstrating your understanding of compliance-grade data engineering practices
Make and justify infrastructure and performance trade-offs — including partitioning strategies, cluster sizing considerations, and pipeline scheduling decisions — that show you can reason about operational cost, processing latency, and system reliability in a production GCP environment handling millions of transactions per hour.
How Your Work Will Be Scored
What to Submit
Code Artifact — Structuring Detection Enrichment Module
Format: .pdf, .doc, .docx, .rtf, .txt, .md
Write a proof-of-concept implementation of the core enrichment logic. Your code should demonstrate:
- PySpark: The primary pipeline logic — reading source data, computing rolling entity-level aggregation windows, and producing enriched output records.
- Python: Orchestration, configuration management, and any utility functions (e.g., schema validation, data quality checks, output formatting for the downstream API contract).
- Scala: At least one performance-critical component where Scala's strengths over PySpark are relevant — for example, a custom aggregation function, a UDF for entity behavior scoring, or a specialized partitioner. Include a brief code comment explaining why you chose Scala for this component.
This is a proof-of-concept, not production-ready code. Focus on demonstrating your approach, your understanding of the data processing patterns, and your fluency across all three languages. Include sample input/output data or schema definitions so an evaluator can understand what the code processes.
Sign in to upload files
README Document — Three Sections
Format: .pdf, .doc, .docx, .rtf, .txt, .md
Section A — Architecture Assessment (300–500 words)
Describe your pipeline architecture for the Structuring Detection Enrichment Layer. Include:
- Data flow from source (Cloud Storage Parquet files) through enrichment to downstream delivery (case management API)
- How you handle the three rolling aggregation windows (24h, 72h, 30-day) — processing strategy, storage decisions, and why you chose this approach over alternatives
- At least one explicit design trade-off: what you chose, what you deprioritized, and why
- How your design maintains audit-trail lineage from source transaction to enriched output
Section B — Production Operations Plan
This pipeline will run daily in production, unsupervised, processing millions of records for regulatory compliance. Address the following:
Part B1 — Monitoring & Failure Recovery
- What metrics would you monitor to detect silent data quality degradation (not just job failures)?
- If the pipeline fails at 2am during the daily batch window, what is your recovery procedure? What do you check first, and what is your escalation path?
- How do you handle late-arriving transactions that fall within an already-computed aggregation window?
Part B2 — Required Reasoning Question (answer without AI assistance)
Describe a scenario where an AI coding assistant would give you a plausible but incorrect answer for this type of pipeline problem — and explain specifically how you would catch it. What would the incorrect output look like, and what would you check to identify the error before deploying it? Be specific to the financial crime data processing context.
Section C — AI Usage Log (Mandatory)
This is not a trick. We want to see how you work with AI — not whether you used it.
In a short section of your README, document your AI collaboration process. For each significant interaction with an AI tool, briefly note:
- What you asked the AI to help with
- What it gave you
- What you kept, changed, or rejected — and why
Three interactions documented is sufficient. The log does not need to be exhaustive.
Sign in to upload files
Video Walkthrough
Format: .mp4, .mov, .webm
Screen-Share Video Submission
Record a screen-share video walking through your submission. Upload it as an MP4 or MOV file directly on the Provn platform as a separate file.
Your video should cover:
- Summary (60 seconds): The problem and your recommended approach in plain language.
- Code walkthrough (3–4 minutes): Walk through your proof-of-concept code. Explain your key design decisions — why you structured the pipeline this way, why you chose Scala for the component you did, and how the pieces fit together.
- Operations plan (1–2 minutes): Walk through your monitoring and failure recovery approach from Section B.
- Mandatory AI question (1–2 minutes): Walk me through one moment where you disagreed with, pushed back on, or redirected what the AI gave you — and what you did instead. Name the specific moment. Explain what the AI produced that didn't meet the bar, what you did differently, and why.
- Reflection (30–60 seconds): What would you do differently with more time?
Speak naturally. Communication is assessed on clarity of technical ideas and logical structure — not verbal polish, accent, or filler words.
Sign in to upload files
Create a free account to upload your work. Your progress saves as a draft until you submit.
On this page