Provn
    How it worksBrowse jobsFor companiesBlogLog in

    © 2026 Provn Inc. All rights reserved.

    About•Blog•Terms of Service•Privacy Policy

    Made with love in Seattle

    Challenges/TalentReach/General/AI Experience Engineer Challenge

    AI Experience Engineer Challenge

    AI
    LLM
    Agent Design
    System Architecture
    FinTech
    API Integration
    User Experience
    Estimated Time:
    1 hour
    Status:Not started

    What You'll Be Doing

    This challenge assesses your ability to design and implement an AI-powered workflow assistant for PortX, a company that powers integrations between banks, credit unions, and fintech companies.

    Scenario — "AI Workflow Assistant for Integration Setup":

    PortX powers integrations between banks, credit unions, and fintechs. New customers often ask: "How do I connect my banking core to a fintech partner for payments or onboarding?" Right now, this setup process involves reading documentation and configuring API endpoints manually.

    Your task is to design an AI Workflow Assistant that can guide users through connecting a new integration safely, interactively, and with human-level clarity.

    Record a video walking us through your approach, and submit two short design documents outlining your system architecture and reliability strategy.

    What You'll Accomplish

    Design orchestration layers for LLM-powered agents

    Implement safety guardrails and human-in-the-loop workflows

    Build transparent, explainable AI interactions

    Balance reliability with flexibility in agentic systems

    Create audit trails for compliance and trust

    How Your Work Will Be Scored

    System Design & Architecture — 40% AI Reliability & Safety — 30% User Experience & Interaction Clarity — 20% AI Usage Transparency — 10%

    What to Submit

    Required Submissions:

    1. System Design Sketch (Required)

    • Format: PDF or Document
    • Length: 1 page maximum
    • Content: Provide a simple diagram or outline showing:
      • The main components (agent logic, data connectors, model orchestration)
      • How the system handles retrieval, planning, and action execution
      • How it maintains safety and traceability (logs, approval prompts, or tool permissions)

    Focus on architecture and reasoning — not full code. A hand-drawn diagram is acceptable if clear.

    Strong Example: "I'd design an orchestration layer with LangChain's agent executor and custom PortX tools: ListPartnersTool for available integrations, SetupEndpointTool for authenticated API setup. Each step prompts user confirmation ('I'll create this endpoint — okay?') and logs it in Datadog. Guardrails: restrict API actions to read-only until confirmation; use retry on timeout. Success metric: 90% of users complete setup with zero support tickets."

    Weak Example: "Use GPT to generate configs and push them to the API automatically."


    2. Reliability & Feedback Plan (Required)

    • Format: PDF or Document
    • Length: 1 page maximum
    • Content: Describe how you'd ensure high-quality, trustworthy AI outputs in production:
      • What guardrails or fallback strategies would you use (e.g., model confidence thresholds, rule-based checks, human-in-the-loop)?
      • How would you capture and use user feedback to improve performance over time?
      • How would you measure success — accuracy, completion rate, latency, or user trust?

    Think about both user experience and system health.


    3. Video Recording (Required)

    • Format: MP4
    • Length: 7-10 minutes (continuous recording, one take)
    • Must include all three components detailed below

    Submission Guidelines

    Video Requirements:

    Your 7-10 minute video must include ALL three components:

    1. Introduction (1-2 minutes)

    Briefly describe one AI-powered experience you've built or prototyped. What made it effective for users, and how did you validate its performance?


    2. Traits Assessment (2-3 minutes)

    Which of these situations fits your natural working style better — and why?

    Scenario A — "The Launch Sprint" (Execution): You're asked to ship an AI feature that summarizes customer feedback from support tickets. The models and APIs are known — your challenge is tuning prompts, handling edge cases, and delivering it into production by the end of the week. Would that kind of focused, high-velocity delivery excite you?

    Scenario B — "The Discovery Loop" (Exploration): You're asked to design a new "AI Workflow Assistant" that helps financial institutions connect data pipelines to fintech partners. No one knows exactly how it should work yet — you'll explore, prototype, and define the core interaction model. Would you enjoy that kind of open-ended, product-defining work?

    There's no right or wrong answer — we just want to understand how you like to work.


    3. Challenge Response (3-5 minutes)

    Walk us through how you would design and implement the AI Workflow Assistant described above:

    • How would you structure the agent or LLM orchestration?
    • How would you connect it safely to PortX APIs or partner data?
    • How would you make it reliable, explainable, and user-friendly?

    AI Usage

    At the end of your video, please tell us if you used AI. If you did, let us know how you used it.

    On this page

    Top of Page
    What You'll Be Doing
    How It's Scored
    What to Submit