AI Evals Playground

Build a prompt, get the model's reply, then grade it four ways — a deterministic keyword check, an AI-as-judge score, a token-budget cost guardrail, and a human thumbs up/down. One reply, four families of eval.

Prompt → reply
System
User
Assistant reply
Press Send & grade to get the model's reply.
Sampling
1.0
0 · focused2 · wild
512
16 · short1024 · long
Evals

Keyword check

Rule-based —

Passes if the reply contains this word or phrase (ignoring capitalization). Fast, free, and fully deterministic — but only as good as the word you pick.

AI as judge

Model-graded —

A second model call scores the reply 1–10 against a rubric and gives a one-line rationale.

/ 10

Token budget

Cost guardrail —

Passes if the reply used no more than this many completion tokens — a deterministic cost/latency guardrail read straight from usage.

Human review

Human —

The ground truth every automated eval is trying to approximate. You read the reply and decide.

Chat model via OpenAI's /v1/chat/completions endpoint, called through Netlify Functions that hide the API key. The reply is graded by four evals spanning the three families: rule-based (keyword, token budget), model-graded (AI as judge), and human (thumbs). The keyword and token-budget checks run entirely in the browser; the AI judge is a second server call.

Source on GitHub.

Judge system prompt

This is the system prompt sent to the grading model. It defines the rubric and asks for a 1–10 score plus a one-line rationale as JSON. Edit it to change how strictly — or on what basis — the reply is graded.