TryCrucible — Prove your AI engineering skills

Scenario

You are given 120 customer messages with expected JSON extractions and model outputs. Some outputs contain simple regressions such as invalid JSON, wrong product, missed priority, missing required fields, or unsupported hallucinated fields. Build an eval suite that detects these cases and reports useful failure categories.

Expected output

Eval suite correctly identifies regressions in the 20 held-out cases and explains the failure category

Dataset

📦

Dataset files

Scoring rubric

Clarity5%

Eval report is easy to read and maps each failure to an actionable category

LLM usage20%

If LLM-as-judge is used, prompts are deterministic and limited to ambiguous cases

Robustness10%

Handles invalid JSON, missing fields, extra fields, and near-miss values

Correctness25%

Correctly marks held-out outputs as pass/fail and identifies the failure mode

Architecture20%

Eval checks are separated into validity, required fields, exact field matching, and unsupported fields

Language-free evaluation

Build your solution in any language or framework — Python, TypeScript, Go, Rust, Java, C#, or anything else. The dataset artifacts may be in one language; your implementation does not need to match. TryCrucible evaluates the behaviour of your system, the quality of your AI workflow, your verification strategy, and the reproducibility of your submission — not your language choice.

Submission requirements

A public GitHub repository link
A Dockerfile in the repo root — any language or framework; the evaluator builds and runs your container
Your solution reads test_inputs.json from the working directory and writes results.json — standard I/O contract across all challenges
A decisions.md — 3–5 sentences on the key architectural and AI-workflow choices you made
The system must be fully reproducible — we clone, build, and run it against real test inputs

Evaluation contract

When you submit, the evaluator runs these steps in order:

1Clone your public GitHub repository
2Build your container from the Dockerfile in the repo root
3Mount test_inputs.json into the working directory
4Run your solution in a network-isolated sandbox (5 min limit, 512 MB RAM)
5Read results.json from the working directory
6Score correctness against hidden ground truth, then score architecture, AI workflow, robustness, and clarity

// I/O contract — same schema for every challenge

Input (provided by evaluator)

// test_inputs.json
[
  { "id": "t1", "input": { ... } },
  { "id": "t2", "input": { ... } }
]

Output (written by your solution)

Create a free account to start

Already have one? Sign in

Write evals for prompt regression testing