TryCrucible — Prove your AI engineering skills

Scenario

NimbusCloud has 300 technical support tickets. Build a hybrid retrieval system (BM25 + vector) with a re-ranking step before generation. Your system must outperform a naive vector-only baseline on the provided test queries.

Expected output

Hybrid retrieval outperforms vector-only baseline, re-ranking improves top-1 accuracy by at least 10%

Dataset

📦

Dataset files

Scoring rubric

Clarity5%

Benchmark results comparing baseline vs hybrid are included in the submission

LLM usage20%

Context selection is tight — only top-k after re-ranking, not full retrieval set

Robustness10%

Handles queries where BM25 and vector disagree, degrades gracefully under re-ranker errors

Correctness25%

Answers match ground truth, ranked sources show re-ranking improved top-1 over baseline

Architecture20%

BM25 and vector scores are combined correctly, re-ranker is a separate composable step

Language-free evaluation

Build your solution in any language or framework — Python, TypeScript, Go, Rust, Java, C#, or anything else. The dataset artifacts may be in one language; your implementation does not need to match. TryCrucible evaluates the behaviour of your system, the quality of your AI workflow, your verification strategy, and the reproducibility of your submission — not your language choice.

Submission requirements

A public GitHub repository link
A Dockerfile in the repo root — any language or framework; the evaluator builds and runs your container
Your solution reads test_inputs.json from the working directory and writes results.json — standard I/O contract across all challenges
A decisions.md — 3–5 sentences on the key architectural and AI-workflow choices you made
The system must be fully reproducible — we clone, build, and run it against real test inputs

Evaluation contract

When you submit, the evaluator runs these steps in order:

1Clone your public GitHub repository
2Build your container from the Dockerfile in the repo root
3Mount test_inputs.json into the working directory
4Run your solution in a network-isolated sandbox (5 min limit, 512 MB RAM)
5Read results.json from the working directory
6Score correctness against hidden ground truth, then score architecture, AI workflow, robustness, and clarity

// I/O contract — same schema for every challenge

Input (provided by evaluator)

// test_inputs.json
[
  { "id": "t1", "input": { ... } },
  { "id": "t2", "input": { ... } }
]

Output (written by your solution)

Create a free account to start

Already have one? Sign in

Production RAG with hybrid search and re-ranking