Provn
    How it worksBrowse jobsFor companiesBlogLog in

    © 2026 Provn Inc. All rights reserved.

    About•Blog•Terms of Service•Privacy Policy

    Made with love in Seattle

    Challenges/Gordian Software/Operations Manager/Fixing a Faulty AI Travel Response

    Fixing a Faulty AI Travel Response

    Operations
    AI Quality Assurance
    Prompt Engineering
    Process Improvement
    Customer Support
    Estimated Time:
    30 minutes
    Status:Not started

    What You'll Be Doing

    Scenario — "The Refund Mix-Up"

    Context: Axel's AI chat agent helps travelers understand their flight options including when they're eligible for a refund or credit.

    A customer sent this message: "Hi, my flight was canceled by the airline. Can I get my money back?"

    The AI responded: "Since your ticket is non-refundable, you are not eligible for a refund, but you can use the credit for future travel."

    The response was incorrect — when an airline cancels a flight, passengers are entitled to a refund, even on non-refundable tickets.

    Your Task:

    You've been asked to:

    1. Find what went wrong — what about the prompt or rule likely caused the incorrect reply?
    2. Fix it — write or describe an updated prompt snippet or instruction that would lead the model to give the right answer
    3. Test it — show what the correct AI reply should look like for the same user message
    4. Explain how you'd validate the fix before rolling it out (e.g., sample test chats, review process)

    Submission Guidelines

    Video Requirements:

    Your 7-10 minute video must include ALL three components:

    1. Introduction (1-2 minutes)

    • Share one example of a time you improved how information was communicated — by a person, a system, or a process. What changed as a result? We want to know how you think.

    2. Traits Assessment (2-3 minutes)

    Which situation sounds more like the kind of work that energizes you — and why?

    • Scenario A — "The Thursday Fix" (Execution): You notice the AI agent is giving wrong answers about baggage fees for one airline. You quickly verify the correct rule, update the prompt instructions, and confirm improved responses in testing. Would you enjoy that kind of direct, fast-turnaround problem solving?
    • Scenario B — "The Pattern Builder" (Discovery): You see that refund-related mistakes are happening across multiple airlines. You spend time reviewing transcripts, identifying the root cause, and proposing a new set of logic or classification improvements. Would you prefer diagnosing larger patterns and designing fixes that scale?

    There's no right or wrong answer — just tell us which type of challenge fits how you work best.

    3. Challenge Response (3-5 minutes)

    Walk us through how you'd fix the faulty AI response in the scenario:

    • What's wrong with the AI's answer?
    • How would you adjust the prompt or rule?
    • How would you test that it now works reliably?

    Keep it simple and practical — focus on your thought process and clarity.

    AI Usage Declaration:

    At the end of your video, please state:

    • ☐ I did not use any AI tools
    • ☐ I used AI tools (briefly describe how, including what I changed or adapted)

    What You'll Accomplish

    Diagnose AI response quality issues

    Validate and test AI behavior improvements

    Balance accuracy with user experience Monitor and improve operational quality

    Monitor and improve operational quality

    Write clear prompt instructions and guidelines

    How Your Work Will Be Scored

    Creative Problem Solving & Accuracy-25%Systems Thinking & QA Design-20%High Agency / Ownership — 20% Automation & AI Fluency-15%Customer Empathy & Communication-15%AI Usage Transparency-5%

    What to Submit

    Required Submissions:

    1. Prompt Fix & Example Output (Required)
      • Format: PDF or Document
      • Length: 1 page maximum
      • Show your corrected prompt or guideline text
      • Include an example of the right AI reply the user should see
      • Briefly explain what changed and why
    2. Quality Assurance Plan (Required)
      • Format: PDF or Document
      • Length: 1 page maximum
      • Describe how you'd:
        • Test the fix on a few similar chat examples
        • Monitor performance once deployed (e.g., error rate, customer feedback)
        • Share insights with engineering or training teams for longer-term improvements
      • Keep it lightweight — this is about how you think, not formal testing scripts
    3. Video Recording (Required)
      • Format: MP4
      • Length: 7-10 minutes (continuous recording, one take)
      • Must include all three components detailed below

    Submission Guidelines

    Video Requirements:

    Your 7-10 minute video must include ALL three components:

    1. Introduction (1-2 minutes)

    • Share one example of a time you improved how information was communicated — by a person, a system, or a process. What changed as a result?

    2. Traits Assessment (2-3 minutes)

    Which situation sounds more like the kind of work that energizes you — and why?

    • Scenario A — "The Thursday Fix" (Execution): You notice the AI agent is giving wrong answers about baggage fees for one airline. You quickly verify the correct rule, update the prompt instructions, and confirm improved responses in testing. Would you enjoy that kind of direct, fast-turnaround problem solving?
    • Scenario B — "The Pattern Builder" (Discovery): You see that refund-related mistakes are happening across multiple airlines. You spend time reviewing transcripts, identifying the root cause, and proposing a new set of logic or classification improvements. Would you prefer diagnosing larger patterns and designing fixes that scale?

    There's no right or wrong answer — just tell us which type of challenge fits how you work best.

    3. Challenge Response (3-5 minutes)

    Walk us through how you'd fix the faulty AI response in the scenario:

    • What's wrong with the AI's answer?
    • How would you adjust the prompt or rule?
    • How would you test that it now works reliably?

    Keep it simple and practical — focus on your thought process and clarity.

    AI Usage Declaration:

    At the end of your video, please state:

    • ☐ I did not use any AI tools
    • ☐ I used AI tools (briefly describe how, including what I changed or adapted)

    On this page

    Top of Page
    What You'll Be Doing
    How It's Scored
    What to Submit