Generative evaluation
"It looked good when I tried it" is not a quality bar. Pradra Studio evaluates generative capabilities against a golden set — examples you curate, with known-good answers — and turns the results into named, numeric metrics that the same acceptance criteria machinery can gate.
The golden set
A golden set is a collection of input → expected-output examples for the capability: real tickets with their correct summaries, real invoices with their correct extracted fields. It is your ground truth, independent of any model — which is exactly why it can judge one.
Keep it honest and it compounds in value: every production mistake you notice becomes a new golden example the next prompt version must get right.
The metrics
An evaluation runs every golden example through the capability's pinned model and prompt version, then scores:
| Metric | Measures |
|---|---|
structural_valid_rate | Did extraction produce valid, contract-shaped output? |
field_match_rate / exact_match_rate | How many extracted fields / whole answers matched? |
similarity_score | How close generated text came to the expected text. |
safety_flag_rate | How often the content filter fired. |
golden_set_score | The headline: overall quality on your ground truth. |
Optionally, an LLM judge can add a rubric score — but it is deliberately auxiliary: the golden set is the anchor, and if the judge is unavailable the evaluation records that fact rather than inventing a zero. The judge model and rubric version are recorded with every report, so scores are comparable over time.
The gate
Set acceptance criteria over these metrics (for example,
golden_set_score >= 0.9) and the rules are enforced at
prompt activation and fine-tune promotion:
an unevaluated or failing version cannot become the serving one. One gate for
classic ML, generative, and agents alike.
The content filter
Alongside evaluation, a serving-path content filter (deterministic patterns you configure — no model in the loop) guards live traffic:
- A match on the input blocks the request before the model is called (and before any tokens are paid for).
- A match on the output withholds the flagged text — the consumer receives a fixed "response withheld" notice with a flag it can handle, and the event is recorded for audit.
Filter hits are counted per capability, so a spike is visible in monitoring.
Working with evaluation in the Studio
- Open the generative capability's Evaluation tab in AI Studio.
- Build the golden set: add examples (input fields + expected output).
- Run an evaluation against the prompt version you're considering; the report shows every metric.
- If it passes your criteria, activate the version from the Prompts tab — the gate re-checks automatically.
Next
- Ground answers in your own documents: Knowledge sources.