Project idea · ungraded

Agent Evaluation Suite

Build a reusable test suite for an agent: scripted scenarios with expected tool calls and answers, metrics such as task success, tool accuracy and step count, and a report that shows what changed between two versions of the agent.

You will use

Milestones

Build the smallest working version first. Ticks are saved in this browser only.

01Write 20 test cases: input, expected final answer (or a checker function) and expected tools.
02Run an agent on every case and record the messages, tool calls, steps and time.
03Compute task success, tool precision and recall, and average steps per task.
04Add safety cases that must end in refusal or approval, and count violations.
05Compare two agent versions in one table and highlight regressions.

Stretch goals

Use a second model as a grader for open-ended answers, and check its agreement with your own grades.
Run each case several times to measure variability.
Run the suite automatically on every change with a CI workflow.

Data

Write the test cases yourself, for the study-planner or club-assistant agent. Scripted fake models make the first version of the suite fully deterministic.

Run it yourself

pip install langgraph langchain-core pandas
python evaluate.py --agent planner