Evaluations

Evaluations help you test an Action against known examples and compare the actual result with the result you expected.

Use evaluations when you want confidence that an Action still behaves correctly after changing prompts, steps, tools, filters, or AI model settings.

What Evaluations Are For

An evaluation dataset is a collection of test cases for one Action. Each test case contains representative input data and the expected result.

Evaluations help answer questions such as:

• Does the Action produce the expected final output?
• Did the Action call the expected tools with the expected arguments?
• Which test cases fail after a configuration change?
• Is the behavior stable across common, edge, and failure-prone inputs?
• Can a cheaper or faster AI model handle this Action, or does it need a more capable model?

Typical Evaluation Process

1. Pick an Action you want to validate.
2. Create or open an evaluation dataset for that Action.
3. Add test cases with realistic inputs and expected outputs.
4. Run the dataset as a dry run or live test.
5. Review passed and failed cases.
6. Adjust the Action, prompts, model choices, tools, or expectations.
7. Run the dataset again and compare results.

This gives you a repeatable way to verify changes instead of relying on one manual run.

The Test Datasets panel on the Evaluations page lists every dataset in the tenant. When the list is long, it scrolls inside the panel, and the search box above it filters datasets by name, description, or Action name.

Creating Test Cases From Runs

The easiest way to create useful test cases is often from real runs.

When you inspect a run, you can add it to an evaluation dataset. Studio uses the run data to prefill the test case, including the input context and observed outputs. You can then edit the expected result to describe what should happen in future runs.

This is useful because real runs include realistic input shapes, tool responses, and edge cases that are easy to miss when writing test data manually.

What a Test Case Contains

A test case can include:

• input parameters for the Action,
• expected tool calls,
• expected final output,
• optional evaluation metrics,
• whether the case is enabled for runs.

Expected tool calls are useful when you need to verify that the Action not only returns the right answer, but also uses the right systems in the right way.

Expected final output is useful for validating the final business result, such as extracted invoice fields, classification decisions, or generated summaries.

Evaluation Metrics

Exact output matching is strict: any difference in the final output fails the case. Evaluation metrics let you check only the fields that matter, using comparisons such as exact match, contains, numeric comparison, regular expressions, or string similarity with a threshold.

Each metric has a JSON path (for example $.InvoiceData.currency), a comparison type, and an expected value. When you add a test case from a run, use Pick from Output to click a field from the run's output and create a metric for it with the observed value prefilled.

Metrics on intermediate step outputs

By default a metric is evaluated against the final output, which is the output of the last executed step in the Action Flow. A metric can instead target the output of any executed step, which is useful when the interesting result is produced in the middle of the flow. For example, you can require that an Analysis step always extracts USD as the currency, even when a later step transforms or summarizes that result.

To create a step metric, open Pick from Output, choose the step in Pick fields from, and click a field from that step's output. The metric card shows the step name, and the Evaluate selector on the card lets you switch a metric between the final output and any step of the run. When editing a test case later, Studio loads the steps of the run the case was created from, or of its most recent evaluation run, so the same choices are available. A test case that was created manually and never evaluated has no step outputs to pick from yet; run the dataset once and the steps appear.

A step metric fails when the step did not run in the evaluation, for example because it was disabled or its switch branch was not taken. Steps are matched by their identity across draft and published versions, so renaming or reordering steps does not break existing metrics.

Turn on Evaluate metrics only when the metrics should decide the result on their own. The expected final output is then kept for reference but no longer compared.

Running a Dataset

Run a dataset when you want to verify multiple cases together.

Use a dry test when you want to avoid making live external changes where supported. Use a live test when the Action must interact with real tools or systems to validate the full Action Flow.

Select the test cases to include, choose the Action version to test, and start the run with Run Dry Test or Run Live Test. After the run finishes, Studio shows summary counts and case-level results.

Repeating Test Cases

Agent Steps can produce a different output for the same prompt from one run to the next, so a test case that passes once is not necessarily stable. Use the Repeat selector next to the Action version to execute every selected test case up to 10 times within one run.

The run then counts each repetition as its own execution: the summary shows the total number of executions, and the results are grouped per test case. Each group shows how many repetitions passed and whether the outcome was consistent:

• Consistent pass: every repetition passed.
• Consistent fail: every repetition failed, which points to a wrong expectation or a real regression.
• Inconsistent: some repetitions passed and others failed, which points to an unstable prompt, model, or expectation.

Expand a group to inspect the individual repetitions, each with its own output diff, metrics, and link to the Action run log.

Running All Datasets

Use Dry Run All or Live Run All on the Evaluations page to start a run for every dataset that has test cases. Studio then shows a batch overview with one row per dataset: its status, how many cases have finished, and the passed and failed counts. The overview updates while the runs are executing and a notification appears when every dataset has finished.

Click a dataset row to open that run's detailed results. If a dataset could not be started, the row shows the error and offers a retry. You can hide the overview while runs continue and reopen it from the status banner under the run buttons.

Reviewing Failed Cases

The test run view shows which cases passed and failed. For each case, you can inspect mismatches in tool calls, final output, metrics, and error messages.

If a case fails, open the linked test case from the run results and decide whether to:

• fix the Action because the output is wrong,
• update the expected output because the intended behavior changed,
• split the scenario into more specific test cases,
• adjust the model or prompt because the result is unstable (repeat the run to confirm the instability before changing anything).

Use failed cases as a focused debugging queue. They tell you exactly which input scenario needs attention.

Asking Shepherd about results

Dooap Shepherd can read test datasets, test cases, and test run results. With a test run open, ask it to summarize the results or to explain why a case failed: it reads the metric expected and actual values, the mismatched tool calls, and the linked action run, then tells you whether the Action regressed, the expected value is stale, or a metric's JSON path no longer resolves. With an Action open, ask which datasets cover it and how the latest run went.

Shepherd can also start a run for you. Ask it to run a dataset, approve the card it shows (dry or live run, Action version, number of cases), and it summarizes the results when the run finishes and compares them with the previous run. Ask "what changed since the last run" on any test run to get the cases that flipped between passing and failing and how their metric values moved.

Evaluating AI Model Choices

Evaluations are especially useful when comparing AI model configurations.

For example, you might want to know whether an Action can use a cheaper and faster model or whether it requires a more capable model. Instead of guessing, run the same dataset with different model settings and compare pass rates, output quality, latency, and cost.

Practical model comparison questions include:

• Does the smaller model still pass the critical test cases?
• Are failures limited to edge cases, or do they affect common scenarios?
• Does the faster model call tools correctly?
• Is the quality gain from a stronger model worth the added cost?
• Can different steps use different models based on complexity?

For credit and model cost considerations, see Credits.

Good Evaluation Habits

• Start with a small dataset of high-value scenarios.
• Include both successful and failure-prone examples.
• Add real cases from runs whenever possible.
• Keep expected outputs specific enough to catch regressions.
• Re-run evaluations after changing prompts, tools, models, or step logic.
• Keep test cases enabled only when they represent behavior you currently want to enforce.

Evaluations work best when they become part of normal Action maintenance: change, run, review, improve, and repeat.

---