A reproducible method for AI agent evaluation
Agent evaluation should expose where a system succeeds, fails, recovers and consumes resources—not compress every behaviour into one opaque score. This page defines the proposed Neul Agent Quality Harness and clearly separates the research design from results that have not yet been published.
Research status: method proposed; public benchmark results pending. No provider, model or framework ranking is claimed on this page.
Six evaluation layers
Each layer answers a different question. A project uses only the layers relevant to its intended behaviour and risk.
Deterministic contract tests
Schema validation, allowed tool selection, argument constraints, state transitions, budgets, prohibited actions and known recovery paths.
Task-level scenarios
Representative multi-step tasks with explicit success conditions, controlled fixtures and labelled partial or unsafe outcomes.
Adversarial cases
Prompt injection, untrusted tool output, secret requests, cross-tenant access, excessive recursion, retry amplification and unavailable dependencies.
Operational measures
Latency distributions, tokens, model and tool calls, retries, compute, failure location, recovery time and human-review load.
Human review
Domain or product review for usefulness, ambiguity and unacceptable behaviour that a deterministic oracle cannot represent.
Model-based scoring
A bounded supplement with pinned judge model and rubric, calibration samples, cost ceiling and disagreement analysis—not the sole source of truth.
Freeze the system that produced the result
Agent behaviour can change when the model, prompt, tool description, policy, dataset or provider changes. The record must identify all of them.
- Target
- Source commit, agent framework, model and provider versions, prompts, tool schemas, policies and runtime configuration.
- Dataset
- Task source, licence, synthetic-generation method, labels, splits, adverse cases and contamination risks.
- Execution
- Randomness, concurrency, retries, timeout, cache, environment, number of repetitions and failure-injection schedule.
- Scoring
- Deterministic or human oracle, rubric, judge model where used, aggregation, uncertainty and unacceptable-failure gates.
- Resources
- Wall time, model and tool calls, input/output tokens, CPU, memory, GPU hours and monetary cost where measurable.
Trace decisions without turning prompts into a data leak
The trace needs enough structure to locate failure while preserving the smallest possible content footprint.
- Correlate task, model, tool, approval and recovery spans.
- Record versions, timing, token and error metadata separately from content.
- Disable or redact prompt and tool payload capture by default.
- Use synthetic identifiers and fixtures for the first implementation.
- Set region, access, retention, export and deletion deliberately.
- Keep provider keys, secrets and customer data out of evaluation artefacts.
Interpretation rules
Keep gates visible
A high average cannot cancel one prohibited data access or unsafe action.
Report the distribution
Tail latency, retries and rare failures matter in multi-step workflows.
Separate components
Ablate model, prompt, tool, policy and runtime changes rather than crediting the whole stack.
Bound the claim
Describe the tested versions and tasks, not universal intelligence or safety.
Questions people ask
What is the Neul Agent Quality Harness?
It is a proposed reproducible reference method for evaluating a Neul-owned or customer-authorised agent using synthetic or approved tasks, deterministic checks, traces, adverse cases, resource measurements and explicit limitations. This page describes the research design; it does not claim a completed public benchmark result.
Which metrics matter for an AI agent?
Metrics depend on the decision. Common measures include task completion, invalid tool arguments, prohibited actions, recovery after failure, human intervention, latency, model and tool calls, tokens, compute and cost. A single aggregate score can hide unsafe or expensive behaviour.
Can an LLM judge another LLM?
A model judge can help score open-ended outputs at scale, but it can be unstable, biased toward presentation or share failures with the tested model. It should use a clear rubric, calibration set, pinned version and disagreement review alongside deterministic and human checks.
How do you protect data in evaluation traces?
Start with synthetic data, capture only fields needed for the question, redact secrets and personal data before export, separate identifiers from content, select retention and region deliberately and test deletion. Customer data requires explicit authority and project-specific controls.