# SkillPatch skill: m365-agent-evaluator

This skill enables agents to help users evaluate Microsoft 365 Copilot declarative agents using the `@microsoft/m365-copilot-eval` CLI. It covers the full evaluation lifecycle: environment setup, dataset authoring (schema v1.2.0), running evals via the correct CLI invocation, analyzing results, and recommending fixes. It supports both ATK and non-ATK project shapes and provides detailed guidance on evaluator configuration and thresholds.

You (the agent) were given this URL and asked to use this skill. This is a **public** skill — no SkillPatch account, API key, or prior setup is required. Two ways to use it:

**1. Use it right now** — the skill's complete file tree (SKILL.md + all reference files) is inlined below; read `SKILL.md` first, then follow it, consulting the other files as it directs.

**2. Install the exact package onto disk** (recommended if you can run a shell — this reproduces the skill byte-for-byte, including any binary assets that can't be inlined):

```bash
mkdir -p .claude/skills/m365-agent-evaluator
curl -sSL https://skillpatch.dev/install_skill/m365-agent-evaluator | tar -xz -C .claude/skills/
```

(`.claude/skills/` is Claude Code's convention; use whatever directory your agent loads skills from.)


---

## Skill files (17)

- `SKILL.md`
- `examples/analyze-failures.md`
- `examples/basic-generation.md`
- `examples/iterate-on-changes.md`
- `examples/missing-instructions.md`
- `examples/not-atk-project.md`
- `examples/run-and-analyze.md`
- `references/azure-setup.md`
- `references/eval-templates.md`
- `references/guardrails.md`
- `references/output-schema.json`
- `references/output-schema.md`
- `references/pra-framework.md`
- `references/prompts-schema.json`
- `references/remediation-patterns.md`
- `references/result-analysis.md`
- `references/workflow.md`


### `SKILL.md`

````markdown
---
name: m365-agent-evaluator
description: >
  Use this skill when a user wants to create, run, or analyze evaluation suites for Microsoft 365 Copilot declarative agents with the public @microsoft/m365-copilot-eval CLI. Trigger on intents such as "evaluate my agent", "test my agent", "run my evals", "create eval prompts", "add multi-turn tests", "tune evaluator thresholds", "why is my agent failing", or "set up eval environment variables".
---

# M365 Agent Evaluator

Use this skill to help users evaluate Microsoft 365 Copilot declarative agents with `@microsoft/m365-copilot-eval`. The skill designs schema-compatible eval datasets, runs the public preview CLI, analyzes results, and recommends targeted fixes.

Default to Microsoft 365 Agents Toolkit (ATK) projects when detected, but do not hard-stop solely because the current directory is not ATK. The CLI can also evaluate deployed agents with an explicit `M365_AGENT_ID` or `--m365-agent-id`.

## Always use this CLI invocation

```powershell
npx -y --package @microsoft/m365-copilot-eval@latest runevals
```

Do not recommend the old private `aka.ms` installer, global installs, bare `runevals`, bare `npx runevals`, `--input`, or `--html`.

## Activation workflow

1. Identify the user goal: setup, dataset authoring, running evals, analyzing results, or updating an existing eval suite.
2. Load only the reference needed for the current goal:
   - `references/workflow.md` for the end-to-end operator workflow and CLI commands.
   - `references/azure-setup.md` for prerequisites, env files, and secret handling.
   - `references/eval-templates.md` when creating or editing eval datasets.
   - `references/pra-framework.md` when deciding what scenarios to generate.
   - `references/result-analysis.md` after JSON/CSV/HTML results exist.
   - `references/guardrails.md` before writing files, handling secrets, clearing cache, signing out, or troubleshooting.
3. Detect project shape:
   - ATK: `.env.local`, `.env.local.user`, `env\.env.local.user`, `m365agents.yml`, or `appPackage\declarativeAgent.json`.
   - Non-ATK: an eval dataset plus `M365_AGENT_ID`, `--m365-agent-id`, or a named environment file such as `env\.env.dev`.
4. Verify prerequisites without exposing values:
   - Node.js 24.12.0 or newer.
   - Microsoft 365 Copilot license and a deployed M365 Copilot agent.
   - Tenant admin consent for the WorkIQ Client App.
   - `TENANT_ID`, Azure OpenAI in Foundry Models endpoint/key, and recommended/default `gpt-4o-mini` deployment.
5. Choose the workflow:
   - No dataset: create `evals\evals.json`.
   - Existing dataset: run, analyze prior results, or propose changes.
   - Quick check: use inline prompts.
   - Exploration: use interactive mode.

## Current dataset contract

Generate schema version `1.2.0` documents with a root `items` array. Do not generate the old `PromptsObject` or root `prompts` format.

Minimum shape:

```json
{
  "schemaVersion": "1.2.0",
  "metadata": {
    "name": "Agent evaluation suite",
    "tags": ["starter"]
  },
  "default_evaluators": {
    "Relevance": {},
    "Coherence": {}
  },
  "items": [
    {
      "prompt": "What can this agent help me with?",
      "expected_response": "The agent explains its supported scope without inventing unsupported capabilities."
    }
  ]
}
```

Use `references\prompts-schema.json` as the local schema source and `references\eval-templates.md` for copyable single-turn, multi-turn, evaluator, and threshold examples.

## Public evaluator names

Evaluator names are case-sensitive. Use only the public configurable evaluator names unless a newer authoritative source proves otherwise.

| Evaluator | Semantics |
|---|---|
| `Relevance` | LLM score from 1-5; default threshold 3. |
| `Coherence` | LLM score from 1-5; default threshold 3. |
| `Groundedness` | LLM score from 1-5 against `context`/expected evidence; default threshold 3. |
| `Similarity` | LLM score from 1-5 against `expected_response`; default threshold 3. |
| `Citations` | Count-based citation check; default threshold 1. |
| `ExactMatch` | Boolean exact string match. |
| `PartialMatch` | String similarity from 0.0-1.0; default threshold 0.5. |

Treat `ToolCallAccuracy` as legacy/private for authoring. Do not add it to generated datasets unless current public CLI/schema documentation explicitly reintroduces it.

## Common commands

```powershell
# Version/help checks
npx -y --package @microsoft/m365-copilot-eval@latest runevals --version
npx -y --package @microsoft/m365-copilot-eval@latest runevals --help

# First-time setup / EULA
npx -y --package @microsoft/m365-copilot-eval@latest runevals accept-eula
npx -y --package @microsoft/m365-copilot-eval@latest runevals --init-only

# Batch run with explicit JSON output
npx -y --package @microsoft/m365-copilot-eval@latest runevals --prompts-file evals\evals.json --output .evals\results.json

# Human-review HTML or spreadsheet-friendly CSV
npx -y --package @microsoft/m365-copilot-eval@latest runevals --prompts-file evals\evals.json --output .evals\results.html
npx -y --package @microsoft/m365-copilot-eval@latest runevals --prompts-file evals\evals.json --output .evals\results.csv

# Quick checks
npx -y --package @microsoft/m365-copilot-eval@latest runevals --prompts "What can you help me with?" --expected "The agent describes its supported scope."

# Non-ATK or named environment
npx -y --package @microsoft/m365-copilot-eval@latest runevals --prompts-file evals\evals.json --m365-agent-id <agent-id> --env dev
```

Use `--concurrency` only with values 1-5. Start with `1` for debugging and increase only after setup is stable.

## Version and PATH safety

Before diagnosing agent behavior, confirm which executable is running:

```powershell
Get-Command runevals -All
npm list -g @microsoft/m365-copilot-eval --depth=0
npm view @microsoft/m365-copilot-eval version
npx -y --package @microsoft/m365-copilot-eval@latest runevals --version
npx -y --package @microsoft/m365-copilot-eval@latest where runevals
```

If bare `runevals` prints `This version of the M365 Evals CLI has stopped working and must be updated`, treat it as a stale PATH/global install. Re-run with the `npx --package ...@latest` command above, then ask before removing global shims with `npm uninstall -g @microsoft/m365-copilot-eval`.

## File conventions

| Path | Purpose |
|---|---|
| `.env.local` | Non-secret ATK config such as `M365_TITLE_ID`. |
| `.env.local.user` or `env\.env.local.user` | Local secrets such as tenant ID and Azure OpenAI key. |
| `env\.env.<environment>` | Named environment config for non-ATK or explicit `--env` workflows. |
| `evals\evals.json` | Source-controlled eval dataset if the user wants it committed. |
| `.evals\` | Local run outputs; usually gitignored. |

Never print or commit secrets, prompts containing sensitive data, retrieved content, debug logs, or raw result files unless the user explicitly asks and confirms the data is safe to share.

## Generation guidance

Use PRA as a scenario-design framework:

- Perceive: retrieval, grounding, and source coverage.
- Reason: instruction adherence, synthesis, ambiguity handling, and refusal behavior.
- Act: declared capability/action behavior. Score with public evaluators such as `Relevance`, `Coherence`, `Similarity`, `ExactMatch`, or `PartialMatch`; do not use legacy `ToolCallAccuracy`.

Ask before overwriting an existing dataset. When writing generated evals, write to a temporary file first and rename on success.

## Result analysis guidance

Analyze only evaluator keys that are present. Missing score keys usually mean the evaluator was not configured for that item, not that it failed.

Use current score keys when present: `relevance`, `coherence`, `groundedness`, `similarity`, `citations`, `exactMatch`, and `partialMatch`. Group failures into likely root causes: instruction issue, grounding issue, citation issue, expected-answer mismatch, capability gap, auth/environment issue, or eval-quality issue.

Do not run real tenant-dependent evals unless the user has provided or approved the necessary tenant, agent, and Azure OpenAI configuration.

````


### `examples/analyze-failures.md`

````markdown
# Example: analyze existing failures

User intent: "Here is my eval output. What should I fix?"

## Inputs

Expected files:

```text
.evals\latest.json
evals\evals.json
```

Do not paste raw prompts, responses, retrieved data, or logs into chat if they may contain sensitive content.

## Process

1. Load `references\result-analysis.md` and `references\remediation-patterns.md`.
2. Inspect `metadata` for CLI version, agent ID/name, and evaluated time.
3. Inspect each item under `items`.
4. For each item, read sparse `scores` keys such as `relevance`, `coherence`, `groundedness`, `similarity`, `citations`, `exactMatch`, and `partialMatch`.
5. Treat missing score keys as "not configured", not failed.
6. Summarize the smallest targeted changes.

## Example finding

```text
Primary issue: Citation behavior is inconsistent.
Evidence: The citation evaluator failed on prompts that ask for workplace-source summaries, while relevance and coherence passed.
Recommended change: Add an instruction requiring citations for source-backed summaries and verify the agent path can surface citations.
Expected effect: `citations` should meet the minimum count threshold without changing the response content.
```

## Common false positives

- Expected response is too specific for a legitimately variable answer.
- `ExactMatch` is used for natural language text.
- A prompt assumes source data that is not available to the deployed agent.
- The run failed during auth, schema validation, or evaluator-model setup.

````


### `examples/basic-generation.md`

````markdown
# Example: create a starter eval dataset

User intent: "Create eval prompts for my M365 Copilot agent."

## Steps

1. Confirm the target agent scenario and whether the repo already has `evals\evals.json`.
2. Load `references\eval-templates.md` and `references\pra-framework.md`.
3. Create a schema `1.2.0` dataset with root `items`.
4. Save it under `evals\evals.json` unless the user asks for another path.

## Starter file

```json
{
  "schemaVersion": "1.2.0",
  "metadata": {
    "name": "Starter M365 Copilot agent evals",
    "description": "Core smoke tests for agent scope, grounding, and response quality.",
    "tags": ["starter", "regression"]
  },
  "default_evaluators": {
    "Relevance": {},
    "Coherence": {}
  },
  "items": [
    {
      "prompt": "What can this agent help me with?",
      "expected_response": "The agent explains its supported scope without claiming unsupported capabilities."
    },
    {
      "prompt": "Summarize the latest status for the project using available sources.",
      "expected_response": "The agent summarizes available status, distinguishes known facts from missing data, and avoids unsupported claims.",
      "evaluators": {
        "Groundedness": {
          "threshold": 3
        }
      },
      "evaluators_mode": "extend"
    },
    {
      "prompt": "List the open action items with owners.",
      "expected_response": "The agent lists action items and owners only when source data supports them.",
      "evaluators": {
        "Citations": {
          "threshold": 1
        }
      },
      "evaluators_mode": "extend"
    }
  ]
}
```

## First safe command

```powershell
npx -y --package @microsoft/m365-copilot-eval@latest runevals --init-only
```

Run real evals only after the user confirms tenant, agent, and Azure OpenAI configuration is ready.

````


### `examples/iterate-on-changes.md`

````markdown
# Example: iterate after agent changes

User intent: "I changed my agent instructions. Re-run the evals and compare."

## Baseline run

```powershell
npx -y --package @microsoft/m365-copilot-eval@latest runevals --prompts-file evals\evals.json --concurrency 1 --output .evals\baseline.json
```

## After-change run

```powershell
npx -y --package @microsoft/m365-copilot-eval@latest runevals --prompts-file evals\evals.json --concurrency 1 --output .evals\after-instructions.json
```

Keep the dataset, evaluator thresholds, model deployment, and concurrency stable when comparing. If the user intentionally changed the dataset, report that the comparison is not a strict regression comparison.

## Compare

1. Compare `items` by prompt or conversation name.
2. Compare only score keys that exist in both runs.
3. Look for improvements and regressions by evaluator theme.
4. If a setup/auth/model error appears in only one run, do not call it an agent regression.

## Example summary

```text
The instruction change improved grounding on the project-status prompt from fail to pass, but the action-item prompt still fails citations. The next targeted change should require source citations when listing owners, or the eval should be relaxed if the agent cannot expose citations for that data path.
```

````


### `examples/missing-instructions.md`

````markdown
# Example: missing or weak instructions

User intent: "My agent gives vague answers in evals."

## Symptoms

- `relevance` passes but `coherence` fails.
- `groundedness` fails because the agent invents missing facts.
- `similarity` fails because the answer omits required structure or decisions.
- Follow-up turns fail after a successful first turn.

## Diagnosis

Check whether the agent instructions specify:

1. supported scenarios and boundaries,
2. source-grounding expectations,
3. citation expectations,
4. response format,
5. behavior when data is missing,
6. follow-up context handling.

## Suggested instruction additions

```text
Use only available workplace sources when answering source-backed questions. If the sources do not contain enough evidence, say what is missing instead of guessing.
```

```text
For project-status answers, use this structure: Summary, Evidence, Risks, Next actions. Keep the answer concise and include citations when available.
```

```text
For follow-up questions, preserve the project, customer, and time window from the prior turn unless the user changes them.
```

## Matching eval update

Add or keep regression prompts that test the new instruction:

```json
{
  "prompt": "Summarize the latest status for the project and include citations.",
  "expected_response": "The agent summarizes only source-backed status, cites available sources, and states when evidence is missing.",
  "evaluators": {
    "Groundedness": {
      "threshold": 4
    },
    "Citations": {
      "threshold": 1
    }
  },
  "evaluators_mode": "extend"
}
```

````


### `examples/not-atk-project.md`

````markdown
# Example: evaluate a non-ATK project

User intent: "I do not have an Agents Toolkit project. Can I still evaluate a deployed agent?"

Yes. Use an explicit deployed agent ID through `M365_AGENT_ID` or `--m365-agent-id`.

## Suggested layout

```text
evals\evals.json
env\.env.dev
.evals\
```

Example `env\.env.dev` values:

```text
TENANT_ID=<tenant-guid>
M365_AGENT_ID=<deployed-agent-id>
AZURE_AI_OPENAI_ENDPOINT=<foundry-models-endpoint>
AZURE_AI_API_KEY=<secret>
AZURE_AI_API_VERSION=2024-12-01-preview
AZURE_AI_MODEL_NAME=gpt-4o-mini
```

Do not print or commit this file if it contains secrets.

## Commands

```powershell
npx -y --package @microsoft/m365-copilot-eval@latest runevals --init-only --env dev
```

```powershell
npx -y --package @microsoft/m365-copilot-eval@latest runevals --prompts-file evals\evals.json --env dev --output .evals\non-atk.json
```

Explicit override:

```powershell
npx -y --package @microsoft/m365-copilot-eval@latest runevals --prompts-file evals\evals.json --m365-agent-id <agent-id> --env dev --output .evals\non-atk.json
```

## Minimal dataset

```json
{
  "schemaVersion": "1.2.0",
  "default_evaluators": {
    "Relevance": {},
    "Coherence": {}
  },
  "items": [
    {
      "prompt": "What can this agent help me with?",
      "expected_response": "The agent describes its supported scope."
    }
  ]
}
```

If auth, tenant consent, or model setup fails, resolve that before evaluating agent quality.

````


### `examples/run-and-analyze.md`

````markdown
# Example: run evals and analyze results

User intent: "Run my evals and tell me why the agent is failing."

## Safe preflight

```powershell
node --version
npx -y --package @microsoft/m365-copilot-eval@latest runevals --version
npx -y --package @microsoft/m365-copilot-eval@latest runevals --help
```

Confirm env files exist without printing values. For first-time setup:

```powershell
npx -y --package @microsoft/m365-copilot-eval@latest runevals accept-eula
npx -y --package @microsoft/m365-copilot-eval@latest runevals --init-only
```

## Run with JSON output

```powershell
npx -y --package @microsoft/m365-copilot-eval@latest runevals --prompts-file evals\evals.json --concurrency 1 --output .evals\latest.json
```

Use `--concurrency 1` for debugging. Increase up to `5` only after setup is stable.

## Optional human report

```powershell
npx -y --package @microsoft/m365-copilot-eval@latest runevals --prompts-file evals\evals.json --output .evals\latest.html
```

## Analysis approach

1. Load `references\result-analysis.md`.
2. Parse `items` from the JSON output.
3. Check only score keys that exist.
4. Separate setup/auth/model/schema failures from quality failures.
5. Group quality failures by likely fix: instructions, grounding, citations, expected response, or capability gap.

Example response:

```text
The main issue is grounding: two prompts passed relevance/coherence but failed groundedness. The agent answered with plausible project facts that were not present in the provided sources. Recommended change: add an instruction to answer only from retrieved workplace sources and say what is missing when evidence is insufficient.
```

````


### `references/azure-setup.md`

````markdown
# Environment and Azure setup

Use this reference when the user needs prerequisites, env files, admin consent, authentication, or Azure OpenAI configuration for `@microsoft/m365-copilot-eval`.

## Prerequisites

| Requirement | Notes |
|---|---|
| Node.js | Node.js 24.12.0 or newer. |
| Operating system | Public docs describe authentication as Windows-first. If another OS fails during auth, validate on Windows before diagnosing the agent. |
| Microsoft 365 Copilot | The signed-in user needs a Microsoft 365 Copilot license. |
| Deployed agent | Evaluate a deployed Microsoft 365 Copilot declarative agent, not only a local manifest. |
| Tenant admin consent | Tenant admin consent is required for the WorkIQ Client App before first use. |
| Azure OpenAI in Foundry Models | The evaluator model endpoint and key are required for LLM-based metrics. |

## Recommended CLI checks

```powershell
node --version
npx -y --package @microsoft/m365-copilot-eval@latest runevals --version
npx -y --package @microsoft/m365-copilot-eval@latest runevals --help
npx -y --package @microsoft/m365-copilot-eval@latest runevals accept-eula
npx -y --package @microsoft/m365-copilot-eval@latest runevals --init-only
```

Do not print environment variable values while checking setup.

## Version and PATH checks

The public preview CLI can retire older versions. Check both the package-scoped version and any bare `runevals` shim before troubleshooting:

```powershell
Get-Command runevals -All
npm list -g @microsoft/m365-copilot-eval --depth=0
npm view @microsoft/m365-copilot-eval version
npx -y --package @microsoft/m365-copilot-eval@latest runevals --version
npx -y --package @microsoft/m365-copilot-eval@latest where runevals
```

If `runevals` without `npx --package` fails with `This version of the M365 Evals CLI has stopped working and must be updated`, a stale global install is being resolved from PATH. Continue with the package-scoped `@latest` command and ask before removing global installs.

## Required configuration values

| Variable | Required | Secret | Purpose |
|---|---:|---:|---|
| `TENANT_ID` | Yes | No | Microsoft Entra tenant ID for the evaluation run. |
| `AZURE_AI_OPENAI_ENDPOINT` | Yes | No | Azure OpenAI in Foundry Models endpoint. |
| `AZURE_AI_API_KEY` | Yes | Yes | Key used by the evaluator model client. |
| `M365_TITLE_ID` | ATK path | No | Agents Toolkit title ID auto-detected from `.env.local`. |
| `M365_AGENT_ID` | Non-ATK or override | No | Deployed M365 Copilot agent ID. |
| `AZURE_AI_API_VERSION` | No | No | Defaults to `2024-12-01-preview`. |
| `AZURE_AI_MODEL_NAME` | No | No | Defaults/recommended value: `gpt-4o-mini`. |

## File placement

### Agents Toolkit project

Use `.env.local` for non-secret project configuration:

```text
M365_TITLE_ID=<atk-title-id>
AZURE_AI_API_VERSION=2024-12-01-preview
AZURE_AI_MODEL_NAME=gpt-4o-mini
```

Use `.env.local.user` or `env\.env.local.user` for local secrets:

```text
TENANT_ID=<tenant-guid>
AZURE_AI_OPENAI_ENDPOINT=<foundry-models-endpoint>
AZURE_AI_API_KEY=<secret>
```

### Non-ATK or named environment

Use a named environment file and select it with `--env`:

```text
env\.env.dev
```

Example:

```text
TENANT_ID=<tenant-guid>
M365_AGENT_ID=<deployed-agent-id>
AZURE_AI_OPENAI_ENDPOINT=<foundry-models-endpoint>
AZURE_AI_API_KEY=<secret>
AZURE_AI_API_VERSION=2024-12-01-preview
AZURE_AI_MODEL_NAME=gpt-4o-mini
```

Run with:

```powershell
npx -y --package @microsoft/m365-copilot-eval@latest runevals --prompts-file evals\evals.json --env dev
```

Or override the agent ID directly:

```powershell
npx -y --package @microsoft/m365-copilot-eval@latest runevals --prompts-file evals\evals.json --m365-agent-id <agent-id> --env dev
```

## Environment precedence

When diagnosing surprising values, check likely sources in this order:

1. Project local files such as `.env.local`.
2. Local user secret files such as `.env.local.user` or `env\.env.local.user`.
3. Named environment files such as `env\.env.dev` when selected with `--env dev`.
4. System environment variables.

Live CLI help has shown `--env` defaulting to `local`; some docs have used `dev`. Prefer passing `--env <name>` explicitly when relying on a named file.

## Gitignore checklist

Ensure local secrets and generated run artifacts are not committed unless the user explicitly chooses to commit sanitized outputs:

```text
.env.local.user
env\.env.local.user
env\.env.*.user
.evals\
*.log
```

`env\.env.<environment>` files can contain secrets in non-ATK projects. Treat them as local-only unless the repo has an established convention for checked-in, non-secret environment templates.

## Troubleshooting setup

| Symptom | Likely cause | Action |
|---|---|---|
| CLI prompts for EULA | EULA has not been accepted | Run `npx -y --package @microsoft/m365-copilot-eval@latest runevals accept-eula`. |
| `This version of the M365 Evals CLI has stopped working and must be updated` | Bare `runevals` is resolving a stale global/PATH install | Use `npx -y --package @microsoft/m365-copilot-eval@latest runevals --version`; ask before removing the global package. |
| Auth fails before agent response | Missing consent, license, or sign-in session | Confirm tenant admin consent and signed-in M365 Copilot user. |
| Model/evaluator errors | Missing endpoint, key, deployment, or API version | Validate Azure OpenAI in Foundry Models values without printing secrets. |
| Agent not found | Wrong `M365_TITLE_ID`, `M365_AGENT_ID`, or environment | Use explicit `--m365-agent-id` and `--env`. |
| Schema validation fails | Dataset uses old format or invalid evaluator names | Validate against `references\prompts-schema.json`. |
| Node error | Node version below public requirement | Upgrade Node.js to 24.12.0 or newer. |

Never paste keys, tenant data, raw prompts, retrieved grounding data, or debug logs into chat unless the user confirms the content is safe to share.

````


### `references/eval-templates.md`

````markdown
# Evaluation dataset templates

Use these templates when creating or editing `@microsoft/m365-copilot-eval` datasets. The current public schema uses `schemaVersion: "1.2.0"` and a root `items` array.

Use `references\prompts-schema.json` as the local schema source.

## Minimal single-turn dataset

```json
{
  "schemaVersion": "1.2.0",
  "metadata": {
    "name": "Agent starter evaluation",
    "description": "Smoke tests for core agent behavior.",
    "tags": ["starter", "single-turn"]
  },
  "default_evaluators": {
    "Relevance": {},
    "Coherence": {}
  },
  "items": [
    {
      "prompt": "What can this agent help me with?",
      "expected_response": "The agent explains its supported scope without claiming unsupported capabilities."
    },
    {
      "prompt": "Summarize the latest status for Contoso renewal.",
      "expected_response": "The agent summarizes available status and avoids inventing facts when source data is unavailable."
    }
  ]
}
```

## Single-turn item with context and groundedness

Use `context` when you have source material the answer must stay grounded in.

```json
{
  "prompt": "What is the renewal deadline?",
  "expected_response": "The renewal deadline is May 31.",
  "context": "The Contoso renewal brief says the renewal deadline is May 31.",
  "evaluators": {
    "Groundedness": {
      "threshold": 4
    },
    "Similarity": {
      "threshold": 3
    }
  },
  "evaluators_mode": "extend"
}
```

`evaluators_mode: "extend"` adds item-level evaluators to `default_evaluators`. Use `"replace"` when the item should use only the item-level evaluator set.

## Evaluator threshold examples

Public configurable evaluator names are case-sensitive:

```json
{
  "default_evaluators": {
    "Relevance": {
      "threshold": 3
    },
    "Coherence": {
      "threshold": 3
    },
    "Groundedness": {
      "threshold": 3
    },
    "Similarity": {
      "threshold": 3
    },
    "Citations": {
      "threshold": 1
    },
    "PartialMatch": {
      "threshold": 0.5
    }
  }
}
```

Use `ExactMatch` sparingly. It is best for deterministic values such as IDs, dates, or fixed policy labels:

```json
{
  "prompt": "Return only the ticket ID for the escalation.",
  "expected_response": "INC-12345",
  "evaluators": {
    "ExactMatch": {}
  },
  "evaluators_mode": "replace"
}
```

## Multi-turn dataset

Use multi-turn items when the agent must preserve context across a conversation. The schema allows up to 20 turns.

```json
{
  "schemaVersion": "1.2.0",
  "metadata": {
    "name": "Multi-turn follow-up suite",
    "tags": ["multi-turn", "regression"]
  },
  "default_evaluators": {
    "Relevance": {},
    "Coherence": {}
  },
  "items": [
    {
      "name": "Follow-up retains project context",
      "description": "The agent should remember that the user is discussing the Contoso renewal.",
      "conversation_id": "contoso-renewal-followup",
      "turns": [
        {
          "prompt": "What is the latest status for the Contoso renewal?",
          "expected_response": "The agent gives the available Contoso renewal status without inventing missing details."
        },
        {
          "prompt": "Who owns the next step?",
          "expected_response": "The agent answers in the context of the Contoso renewal and cites or qualifies the source of the owner."
        }
      ]
    }
  ]
}
```

Do not put top-level `prompt` on a multi-turn item. Put prompts inside `turns`.

## PRA scenario design

Use PRA to choose what to test, then express tests with public evaluators.

| PRA area | Test intent | Useful evaluators |
|---|---|---|
| Perceive | Finds the right source, respects available context, uses citations where required | `Groundedness`, `Citations`, `Relevance` |
| Reason | Synthesizes, follows instructions, handles ambiguity, avoids hallucination | `Relevance`, `Coherence`, `Similarity` |
| Act | Performs or describes declared capabilities accurately | `Relevance`, `Coherence`, `ExactMatch`, `PartialMatch`, `Similarity` |

Do not add legacy/private evaluator names to generated datasets unless a current authoritative public schema includes them.

## Tags and custom metadata

Use root `metadata.tags` for suite-level tags. Use `extensions` for local metadata that the CLI should preserve:

```json
{
  "prompt": "List the open actions from the last planning discussion.",
  "expected_response": "The agent lists action items with owners when available.",
  "extensions": {
    "scenario": "action-items",
    "risk": "hallucinated-owner",
    "priority": "high"
  }
}
```

## Authoring checklist

1. Use `schemaVersion: "1.2.0"` and root `items`.
2. Include clear `expected_response` text for every item where comparison matters.
3. Keep prompts realistic but sanitized.
4. Use only public evaluator names: `Relevance`, `Coherence`, `Groundedness`, `Similarity`, `Citations`, `ExactMatch`, `PartialMatch`.
5. Use `Citations` as a minimum citation count, not as a 1-5 LLM score.
6. Prefer `Similarity` or `PartialMatch` for flexible expected answers; use `ExactMatch` only when exact text is intended.
7. Keep generated datasets in `evals\`; write run outputs under `.evals\`.

````


### `references/guardrails.md`

````markdown
# Guardrails and troubleshooting

Use this reference before writing files, handling secrets, running state-changing commands, or diagnosing failures.

## Safety rules

1. Never print, commit, or summarize secret values from `.env.local.user`, `env\.env.local.user`, `env\.env.<environment>`, system environment variables, or terminal output.
2. Treat prompts, agent responses, retrieved grounding data, HTML reports, CSV files, JSON results, and debug logs as potentially sensitive.
3. Ask before overwriting an existing dataset or report.
4. Prefer source datasets under `evals\` and local run outputs under `.evals\`.
5. Keep generated run outputs out of commits unless the user explicitly confirms they are sanitized and should be committed.
6. Do not run real tenant-dependent evaluations without user-approved tenant, deployed-agent, and Azure OpenAI configuration.

## File writes

Safe defaults:

| File | Default behavior |
|---|---|
| `evals\evals.json` | Ask before overwrite; this may be committed as a regression suite. |
| `.evals\*.json` | Generated result output; usually local-only. |
| `.evals\*.csv` | Generated result output; usually local-only. |
| `.evals\*.html` | Generated result output; usually local-only and may contain response content. |
| `.env.local` | Non-secret ATK config only. |
| `.env.local.user`, `env\.env.local.user`, `env\.env.<environment>` | Secrets/local config; never commit or display values. |

When generating a dataset, write to a new file or a temporary file first, then rename after validation.

## Commands that change local state

Warn the user before running:

```powershell
npx -y --package @microsoft/m365-copilot-eval@latest runevals --cache-clear
npx -y --package @microsoft/m365-copilot-eval@latest runevals --signout
```

`--cache-clear` removes local cache data. `--signout` clears the local auth session and may require the user to sign in again.

## Setup troubleshooting

| Symptom | Likely cause | Action |
|---|---|---|
| EULA prompt or refusal | EULA not accepted | Run `npx -y --package @microsoft/m365-copilot-eval@latest runevals accept-eula`. |
| `This version of the M365 Evals CLI has stopped working and must be updated` | The shell resolved an outdated bare/global `runevals` shim | Use `npx -y --package @microsoft/m365-copilot-eval@latest runevals --version`; ask before uninstalling global packages or deleting shims. |
| `node` or package startup error | Node version too old | Upgrade to Node.js 24.12.0 or newer. |
| Agent cannot be resolved | Missing or wrong `M365_TITLE_ID`, `M365_AGENT_ID`, `--m365-agent-id`, or `--env` | Pass `--m365-agent-id` explicitly and verify the selected env file. |
| Authentication failure | Not signed in, unsupported OS auth path, missing M365 Copilot license, or tenant admin consent missing | Validate signed-in account, license, and consent. |
| Azure evaluator failure | Missing endpoint/key/model/API version | Check env keys exist without printing values; prefer `gpt-4o-mini`. |
| Schema validation error | Old dataset format or unsupported evaluator | Use root `items` and public evaluator names only. |
| Network/proxy errors | Enterprise network blocks auth/model calls | Ask the user to validate network/proxy access; do not retry with exposed secrets. |

## Quality troubleshooting

Do not treat every failed run as an agent bug.

| Failure type | Category |
|---|---|
| Missing env, auth, consent, EULA, schema, or network | Setup failure |
| Low relevance, coherence, groundedness, similarity, citations, exact match, or partial match | Agent/eval quality signal |
| Ambiguous prompt or overly strict expected response | Eval dataset issue |
| Missing source data or inaccessible connector | Capability/data issue |

Fix setup failures first. Then analyze quality signals with `references\result-analysis.md`.

## Debug logging

Use debug logging only when needed:

```powershell
npx -y --package @microsoft/m365-copilot-eval@latest runevals --log-level debug --prompts-file evals\evals.json --output .evals\debug.json
```

Debug output may expose prompts, responses, URLs, identifiers, or retrieved content. Do not paste logs into chat or commit them unless the user confirms they are safe.

````


### `references/output-schema.json`

```
{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "M365 Copilot Eval CLI JSON Output",
  "description": "Compact schema for JSON files written by runevals --output. Output is eval-document compatible and contains evaluated items with sparse score objects.",
  "type": "object",
  "required": ["schemaVersion", "items"],
  "additionalProperties": true,
  "properties": {
    "schemaVersion": {
      "type": "string",
      "pattern": "^1\\.\\d+\\.\\d+$"
    },
    "metadata": {
      "type": "object",
      "additionalProperties": true,
      "properties": {
        "evaluatedAt": { "type": "string" },
        "agentId": { "type": "string" },
        "agentName": { "type": "string" },
        "cliVersion": { "type": "string" }
      }
    },
    "default_evaluators": {
      "type": "object",
      "additionalProperties": true
    },
    "items": {
      "type": "array",
      "minItems": 1,
      "items": {
        "oneOf": [
          { "$ref": "#/$defs/singleTurnOutput" },
          { "$ref": "#/$defs/multiTurnOutput" }
        ]
      }
    }
  },
  "$defs": {
    "singleTurnOutput": {
      "type": "object",
      "required": ["prompt"],
      "additionalProperties": true,
      "properties": {
        "prompt": { "type": "string" },
        "response": { "type": "string" },
        "expected_response": { "type": "string" },
        "context": { "type": "string" },
        "evaluators": { "type": "object", "additionalProperties": true },
        "evaluators_mode": { "enum": ["extend", "replace"] },
        "scores": { "$ref": "#/$defs/scoreMap" }
      },
      "not": {
        "required": ["turns"]
      }
    },
    "multiTurnOutput": {
      "type": "object",
      "required": ["turns"],
      "additionalProperties": true,
      "properties": {
        "name": { "type": "string" },
        "description": { "type": "string" },
        "conversation_id": { "type": "string" },
        "turns": {
          "type": "array",
          "minItems": 1,
          "maxItems": 20,
          "items": {
            "type": "object",
            "additionalProperties": true,
            "properties": {
              "prompt": { "type": "string" },
              "response": { "type": "string" },
              "expected_response": { "type": "string" },
              "scores": { "$ref": "#/$defs/scoreMap" }
            }
          }
        },
        "summary": { "type": "object", "additionalProperties": true }
      },
      "not": {
        "required": ["prompt"]
      }
    },
    "scoreMap": {
      "type": "object",
      "additionalProperties": false,
      "properties": {
        "relevance": { "$ref": "#/$defs/llmScore" },
        "coherence": { "$ref": "#/$defs/llmScore" },
        "groundedness": { "$ref": "#/$defs/llmScore" },
        "similarity": { "$ref": "#/$defs/llmScore" },
        "citations": { "$ref": "#/$defs/citationScore" },
        "exactMatch": { "$ref": "#/$defs/matchScore" },
        "partialMatch": { "$ref": "#/$defs/matchScore" }
      }
    },
    "llmScore": {
      "type": "object",
      "required": ["score", "result", "threshold"],
      "additionalProperties": true,
      "properties": {
        "score": { "type": "number", "minimum": 1, "maximum": 5 },
        "result": { "enum": ["pass", "fail"] },
        "threshold": { "type": "number" }
      }
    },
    "citationScore": {
      "type": "object",
      "required": ["count", "result", "threshold"],
      "additionalProperties": true,
      "properties": {
        "count": { "type": "number", "minimum": 0 },
        "result": { "enum": ["pass", "fail"] },
        "threshold": { "type": "number", "minimum": 0 }
      }
    },
    "matchScore": {
      "type": "object",
      "required": ["score", "result", "threshold"],
      "additionalProperties": true,
      "properties": {
        "score": { "type": "number", "minimum": 0, "maximum": 1 },
        "result": { "enum": ["pass", "fail"] },
        "threshold": { "type": "number", "minimum": 0, "maximum": 1 }
      }
    }
  }
}

```


### `references/output-schema.md`

````markdown
# Output schema reference

The current `runevals --output <file>` JSON output is schema-compatible with the eval document format. It is not the older `{ "summary": ..., "results": [...] }` shape.

Use `references\output-schema.json` for a compact validation-oriented schema and `references\prompts-schema.json` for the full package schema.

## JSON output

Typical JSON output:

```json
{
  "schemaVersion": "1.2.0",
  "metadata": {
    "evaluatedAt": "2025-01-01T00:00:00Z",
    "agentId": "00000000-0000-0000-0000-000000000000",
    "agentName": "Contoso Agent",
    "cliVersion": "1.5.0-preview.1"
  },
  "default_evaluators": {},
  "items": [
    {
      "prompt": "What can this agent help me with?",
      "response": "The agent response.",
      "expected_response": "The expected behavior.",
      "scores": {
        "relevance": {
          "score": 4,
          "result": "pass",
          "threshold": 3
        },
        "coherence": {
          "score": 5,
          "result": "pass",
          "threshold": 3
        }
      }
    }
  ]
}
```

Multi-turn output uses an item with `turns` and may include `summary`.

## Score object

Scores are sparse. Keys appear only for evaluators that ran.

| Score key | Value shape |
|---|---|
| `relevance` | `{ "score": 1-5, "result": "pass" | "fail", "threshold": number }` |
| `coherence` | `{ "score": 1-5, "result": "pass" | "fail", "threshold": number }` |
| `groundedness` | `{ "score": 1-5, "result": "pass" | "fail", "threshold": number }` |
| `similarity` | `{ "score": 1-5, "result": "pass" | "fail", "threshold": number }` |
| `citations` | `{ "count": number, "result": "pass" | "fail", "threshold": number }` |
| `exactMatch` | `{ "score": 0 | 1, "result": "pass" | "fail", "threshold": number }` |
| `partialMatch` | `{ "score": 0.0-1.0, "result": "pass" | "fail", "threshold": number }` |

Do not treat missing score keys as failures.

## CSV output

CSV reports include:

- metadata comments at the top,
- aggregate statistics,
- single-turn sections,
- multi-turn sections,
- serialized score JSON.

Use CSV when the user wants spreadsheet review or lightweight automation without parsing full JSON.

## HTML output

HTML reports are for human review and may open in the default browser. Treat them as sensitive because they can contain prompts, retrieved data, and agent responses.

## Automation guidance

When parsing JSON results:

1. Read `items`.
2. For each item, handle either single-turn `prompt` or multi-turn `turns`.
3. Check only `scores` keys that exist.
4. Treat setup/auth/model/schema errors separately from quality scores.
5. Compare runs only when datasets, evaluators, thresholds, and model configuration are stable.

````


### `references/pra-framework.md`

````markdown
# PRA scenario framework

Use PRA to design a balanced evaluation suite. PRA is a test-design taxonomy, not a set of evaluator names.

## Perceive

Perceive scenarios test whether the agent finds, uses, and cites the right information.

Good prompts:

```text
Summarize the latest status for the Contoso renewal and cite the source.
What risks were identified in the most recent planning discussion?
List the open action items with owners, using only available source data.
```

Useful evaluators:

| Evaluator | Why |
|---|---|
| `Groundedness` | Checks support from source/context. |
| `Citations` | Checks that required citations are present. |
| `Relevance` | Checks that the answer addresses the request. |

Common fixes: improve grounding instructions, clarify source priority, require citations, or update the agent's knowledge/action configuration.

## Reason

Reason scenarios test synthesis, instruction following, ambiguity handling, and refusal behavior.

Good prompts:

```text
Compare the two proposed launch plans and recommend the lower-risk option.
The request is ambiguous: ask one clarifying question before answering.
Create a concise executive summary from the available project context.
```

Useful evaluators:

| Evaluator | Why |
|---|---|
| `Relevance` | Checks that reasoning addresses the ask. |
| `Coherence` | Checks clarity and structure. |
| `Similarity` | Checks alignment with an expected conclusion. |
| `PartialMatch` | Checks that key terms or decisions are present without requiring exact text. |

Common fixes: add response-format guidance, examples, ambiguity rules, refusal guidance, or priority rules for conflicting evidence.

## Act

Act scenarios test declared capability behavior, such as whether the agent can produce the expected final artifact, answer, or action-oriented output.

Good prompts:

```text
Draft a follow-up message with the three agreed actions.
Return only the escalation ticket ID.
Create a prioritized list of next steps for the renewal owner.
```

Useful evaluators:

| Evaluator | Why |
|---|---|
| `ExactMatch` | Best for deterministic IDs or labels. |
| `PartialMatch` | Best for key-term coverage. |
| `Similarity` | Best for flexible expected outputs. |
| `Relevance` | Checks that the action output matches the prompt. |
| `Coherence` | Checks structure and readability. |

Do not use legacy/private action-specific evaluator names in authored datasets unless the current public schema explicitly supports them.

## Suite balance

A strong suite usually includes:

1. Happy-path prompts for the most important user jobs.
2. Edge cases where data is missing, ambiguous, stale, or conflicting.
3. Citation/grounding prompts for answers based on workplace data.
4. Deterministic output prompts for IDs, dates, statuses, or labels.
5. Multi-turn prompts that require follow-up context.
6. Regression prompts for bugs found in production feedback.

Use `metadata.tags` or item `extensions` to label PRA coverage.

Example item metadata:

```json
{
  "prompt": "Who owns the next step for the Contoso renewal?",
  "expected_response": "The agent identifies the owner only if source data supports it.",
  "evaluators": {
    "Groundedness": {},
    "Citations": {}
  },
  "extensions": {
    "pra": "perceive",
    "risk": "unsupported-owner"
  }
}
```

````


### `references/prompts-schema.json`

```
{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://raw.githubusercontent.com/microsoft/M365-Copilot-Agent-Evals/refs/heads/main/schema/v1/eval-document.schema.json",
  "title": "M365 Copilot Eval Document",
  "description": "Schema for evaluation documents used by M365 Copilot Agent Evals CLI. Supports single-turn and multi-turn evaluations.",
  "type": "object",
  "required": ["schemaVersion", "items"],
  "additionalProperties": true,
  "properties": {
    "$schema": {
      "type": "string",
      "format": "uri",
      "description": "JSON Schema URI for editor validation support"
    },
    "schemaVersion": {
      "type": "string",
      "pattern": "^1\\.\\d+\\.\\d+$",
      "description": "SemVer string identifying the schema version this document conforms to (e.g., '1.0.0')",
      "examples": ["1.0.0", "1.1.0", "1.2.0"]
    },
    "metadata": {
      "$ref": "#/$defs/DocumentMetadata"
    },
    "default_evaluators": {
      "$ref": "#/$defs/EvaluatorMap",
      "description": "File-level default evaluators (overrides system defaults)"
    },
    "items": {
      "type": "array",
      "minItems": 1,
      "description": "Array of evaluation items: single-turn evaluations or multi-turn threads",
      "items": {
        "oneOf": [
          { "$ref": "#/$defs/SingleTurnEvaluation" },
          { "$ref": "#/$defs/MultiTurnThread" }
        ]
      }
    }
  },
  "$defs": {
    "DocumentMetadata": {
      "type": "object",
      "description": "Optional metadata about the evaluation document",
      "additionalProperties": true,
      "properties": {
        "name": {
          "type": "string",
          "description": "Human-readable name for the evaluation set"
        },
        "description": {
          "type": "string",
          "description": "Description of what this evaluation set tests"
        },
        "createdAt": {
          "type": "string",
          "format": "date-time",
          "description": "ISO 8601 timestamp when the document was created"
        },
        "createdBy": {
          "type": "string",
          "description": "Author or system that created the document"
        },
        "evaluatedAt": {
          "type": "string",
          "format": "date-time",
          "description": "ISO 8601 timestamp when evaluation was performed"
        },
        "tags": {
          "type": "array",
          "items": {
            "type": "string"
          },
          "description": "Tags for categorization and filtering"
        },
        "agentId": {
          "type": "string",
          "description": "M365 Agent ID this evaluation targets"
        },
        "agentName": {
          "type": "string",
          "description": "Name of the M365 agent this evaluation targets"
        },
        "cliVersion": {
          "type": "string",
          "description": "Version of the M365 Copilot Agent Evals CLI that produced this document"
        },
        "extensions": {
          "type": "object",
          "additionalProperties": true,
          "description": "Extension point for custom metadata. Use reverse-domain notation for field names."
        }
      }
    },
    "SingleTurnEvaluation": {
      "type": "object",
      "description": "A standalone single-turn prompt-response evaluation",
      "required": ["prompt"],
      "additionalProperties": false,
      "properties": {
        "prompt": {
          "type": "string",
          "minLength": 1,
          "description": "The input prompt to evaluate"
        },
        "expected_response": {
          "type": "string",
          "description": "Expected or ideal response for comparison during evaluation"
        },
        "response": {
          "type": "string",
          "description": "Actual response from the agent"
        },
        "context": {
          "type": "string",
          "description": "Additional context for grounding evaluation"
        },
        "evaluators": {
          "$ref": "#/$defs/EvaluatorMap",
          "description": "Per-prompt evaluator overrides"
        },
        "evaluators_mode": {
          "type": "string",
          "enum": ["extend", "replace"],
          "default": "extend",
          "description": "How per-prompt evaluators combine with defaults"
        },
        "citations": {
          "type": "array",
          "items": {
            "$ref": "#/$defs/Citation"
          },
          "description": "Citations included in the response"
        },
        "scores": {
          "$ref": "#/$defs/ScoreCollection"
        },
        "extensions": {
          "type": "object",
          "additionalProperties": true,
          "description": "Extension point for custom item-level fields"
        }
      },
      "not": {
        "required": ["turns"]
      }
    },
    "MultiTurnThread": {
      "type": "object",
      "description": "A multi-turn conversation thread with ordered turns sharing conversation context",
      "required": ["turns"],
      "additionalProperties": false,
      "properties": {
        "name": {
          "type": "string",
          "description": "Human-readable name for the thread"
        },
        "description": {
          "type": "string",
          "description": "Description of what this thread tests"
        },
        "turns": {
          "type": "array",
          "minItems": 1,
          "maxItems": 20,
          "items": { "$ref": "#/$defs/Turn" },
          "description": "Ordered array of conversation turns"
        },
        "conversation_id": {
          "type": "string",
          "description": "Unique identifier for this conversation thread"
        },
        "summary": {
          "$ref": "#/$defs/ThreadSummary",
          "description": "Aggregate statistics for the thread"
        },
        "extensions": {
          "type": "object",
          "additionalProperties": true,
          "description": "Extension point for custom thread-level fields"
        }
      },
      "not": {
        "required": ["prompt"]
      }
    },
    "Turn": {
      "type": "object",
      "description": "A single turn within a multi-turn thread",
      "required": ["prompt"],
      "additionalProperties": false,
      "properties": {
        "prompt": {
          "type": "string",
          "minLength": 1,
          "description": "The user message for this turn"
        },
        "expected_response": {
          "type": "string",
          "description": "Expected agent response for this turn"
        },
        "response": {
          "type": "string",
          "description": "Actual agent response"
        },
        "context": {
          "type": "string",
          "description": "Additional context for grounding evaluation"
        },
        "evaluators": {
          "$ref": "#/$defs/EvaluatorMap",
          "description": "Per-turn evaluator overrides"
        },
        "evaluators_mode": {
          "type": "string",
          "enum": ["extend", "replace"],
          "default": "extend",
          "description": "How per-turn evaluators combine with defaults"
        },
        "citations": {
          "type": "array",
          "items": {
            "$ref": "#/$defs/Citation"
          },
          "description": "Citations included in the response"
        },
        "scores": {
          "$ref": "#/$defs/ScoreCollection"
        },
        "status": {
          "type": "string",
          "enum": ["pass", "fail", "error"],
          "description": "Overall status of this turn"
        },
        "error": {
          "type": "string",
          "description": "Error message if status is 'error'"
        },
        "extensions": {
          "type": "object",
          "additionalProperties": true,
          "description": "Extension point for custom turn-level fields"
        }
      }
    },
    "ThreadSummary": {
      "type": "object",
      "description": "Aggregate statistics for a thread",
      "required": ["turns_total", "turns_passed", "turns_failed", "overall_status"],
      "additionalProperties": false,
      "properties": {
        "turns_total": {
          "type": "integer",
          "minimum": 1,
          "description": "Total number of turns executed"
        },
        "turns_passed": {
          "type": "integer",
          "minimum": 0,
          "description": "Number of turns where all evaluators passed"
        },
        "turns_failed": {
          "type": "integer",
          "minimum": 0,
          "description": "Number of turns where any evaluator failed"
        },
        "overall_status": {
          "type": "string",
          "enum": ["pass", "partial", "fail"],
          "description": "pass: all turns passed, partial: some failed, fail: all failed or error"
        }
      }
    },
    "ScoreCollection": {
      "type": "object",
      "description": "Collection of evaluation scores for an item",
      "additionalProperties": true,
      "properties": {
        "relevance": {
          "$ref": "#/$defs/EvalScore",
          "description": "Relevance score (1-5)"
        },
        "coherence": {
          "$ref": "#/$defs/EvalScore",
          "description": "Coherence score (1-5)"
        },
        "groundedness": {
          "$ref": "#/$defs/EvalScore",
          "description": "Groundedness score (1-5)"
        },
        "similarity": {
          "$ref": "#/$defs/EvalScore",
          "description": "Similarity score (1-5)"
        },
        "citations": {
          "$ref": "#/$defs/CitationScore",
          "description": "Citation evaluation results"
        },
        "exactMatch": {
          "$ref": "#/$defs/ExactMatchScore",
          "description": "Exact match evaluation result"
        },
        "partialMatch": {
          "$ref": "#/$defs/PartialMatchScore",
          "description": "Partial match evaluation result"
        }
      }
    },
    "EvalScore": {
      "type": "object",
      "description": "Standard evaluation score (1-5 scale)",
      "required": ["score", "result", "threshold"],
      "additionalProperties": true,
      "properties": {
        "score": {
          "type": "number",
          "minimum": 1,
          "maximum": 5,
          "description": "Numeric score from 1.0 (worst) to 5.0 (best)"
        },
        "result": {
          "type": "string",
          "enum": ["pass", "fail"],
          "description": "Pass/fail result based on threshold comparison"
        },
        "threshold": {
          "type": "number",
          "minimum": 1,
          "maximum": 5,
          "description": "Threshold used for pass/fail determination"
        },
        "reason": {
          "type": "string",
          "description": "Explanation of why this score was assigned"
        },
        "evaluator": {
          "type": "string",
          "description": "Name or identifier of the evaluator that produced this score"
        }
      }
    },
    "CitationScore": {
      "type": "object",
      "description": "Citation-specific evaluation score",
      "required": ["count", "result", "threshold"],
      "additionalProperties": true,
      "properties": {
        "count": {
          "type": "integer",
          "minimum": 0,
          "description": "Number of citations found in the response"
        },
        "result": {
          "type": "string",
          "enum": ["pass", "fail"],
          "description": "Pass/fail result based on citation count vs threshold"
        },
        "threshold": {
          "type": "integer",
          "minimum": 0,
          "description": "Minimum required number of citations for pass"
        },
        "format": {
          "type": "string",
          "description": "Citation format detected. Known values: 'oai_unicode', 'bracket', 'mixed'. Additional formats may be added.",
          "examples": ["oai_unicode", "bracket", "mixed"]
        },
        "citations": {
          "type": "array",
          "items": {
            "$ref": "#/$defs/Citation"
          },
          "description": "Parsed citation objects"
        }
      }
    },
    "ExactMatchScore": {
      "type": "object",
      "description": "Exact match evaluation result",
      "required": ["match", "result"],
      "additionalProperties": true,
      "properties": {
        "match": {
          "type": "boolean",
          "description": "Whether response exactly matches expected_response (trimmed; case-insensitive by default)"
        },
        "result": {
          "type": "string",
          "enum": ["pass", "fail"],
          "description": "Pass when match is true, fail otherwise"
        },
        "reason": {
          "type": "string",
          "description": "Explanation of the match result"
        }
      }
    },
    "PartialMatchScore": {
      "type": "object",
      "description": "Partial match evaluation result",
      "required": ["score", "result", "threshold"],
      "additionalProperties": true,
      "properties": {
        "score": {
          "type": "number",
          "minimum": 0,
          "maximum": 1,
          "description": "Match score from 0.0 (no match) to 1.0 (full match)"
        },
        "result": {
          "type": "string",
          "enum": ["pass", "fail"],
          "description": "Pass/fail based on score vs threshold"
        },
        "threshold": {
          "type": "number",
          "minimum": 0,
          "maximum": 1,
          "description": "Minimum score required for pass (default: 0.5)"
        },
        "reason": {
          "type": "string",
          "description": "Explanation of the match result"
        }
      }
    },
    "EvaluatorMap": {
      "type": "object",
      "description": "Map of evaluator names to their configuration options",
      "propertyNames": {
        "enum": ["Relevance", "Coherence", "Groundedness", "Similarity", "Citations", "ExactMatch", "PartialMatch"]
      },
      "additionalProperties": {
        "$ref": "#/$defs/EvaluatorOptions"
      }
    },
    "EvaluatorOptions": {
      "type": "object",
      "description": "Evaluator configuration options. Use empty object {} for defaults.",
      "additionalProperties": false,
      "properties": {
        "threshold": {
          "type": "number",
          "description": "Pass/fail threshold. Range depends on evaluator type: 1-5 for LLM evaluators (default: 3), >= 1 integer for Citations (min citation count, default: 1), 0.0-1.0 for PartialMatch (min match ratio, default: 0.5). Validated per-evaluator at runtime."
        },
        "citation_format": {
          "type": "string",
          "examples": ["oai_unicode", "bracket", "mixed"],
          "description": "Citation format for detection. 'oai_unicode': new OAI unicode format, 'bracket': legacy [^i^] bracket format, 'mixed': auto-detect both formats. Default: oai_unicode."
        },
        "case_sensitive": {
          "type": "boolean",
          "default": false,
          "description": "Case-sensitive matching for ExactMatch/PartialMatch"
        },
        "options": {
          "type": "object",
          "additionalProperties": true,
          "description": "Evaluator-specific configuration"
        }
      }
    },
    "Citation": {
      "type": "object",
      "description": "A single citation reference",
      "required": ["index"],
      "additionalProperties": true,
      "properties": {
        "index": {
          "type": "integer",
          "minimum": 1,
          "description": "Citation index (1-based)"
        },
        "text": {
          "type": "string",
          "description": "The cited text"
        },
        "source": {
          "type": "string",
          "description": "Source reference (URL, document name, etc.)"
        }
      }
    }
  }
}

```


### `references/remediation-patterns.md`

````markdown
# Remediation patterns

Use these patterns when converting eval failures into concrete changes to agent instructions, manifest configuration, knowledge sources, or the eval dataset.

## Before recommending changes

1. Confirm the run reached the agent and evaluator model successfully.
2. Separate setup failures from quality failures.
3. Check whether the failed evaluator was configured for that item.
4. Inspect prompt, expected response, actual response, and context for ambiguity or sensitive content.
5. Recommend the smallest targeted change.

## Failure-to-fix map

| Failure | Likely cause | Targeted fix |
|---|---|---|
| Low relevance | Agent answers adjacent intent or ignores constraints | Strengthen instructions about supported scope and task routing; add examples for common intents. |
| Low coherence | Answer is hard to scan or mixes unrelated content | Add response format requirements, length limits, headings, or ordered steps. |
| Low groundedness | Answer includes unsupported facts | Require source-backed answers, clarify source priority, and instruct the agent to say when evidence is missing. |
| Low similarity | Actual content differs from expected behavior | Update instructions/data access, or relax the expected response if multiple valid answers exist. |
| Failed citations | No citation when one is required | Add citation requirements and verify the underlying agent capability can surface citations. |
| Failed exact match | Formatting or deterministic value differs | Add strict output-only instructions, or switch to `PartialMatch` if exact text is too brittle. |
| Low partial match | Key terms missing | Add expected terms to instructions/examples or improve retrieval for the missing concepts. |
| Multi-turn follow-up failure | Agent loses conversation context | Add follow-up handling examples and clarify how to resolve pronouns or references. |

## Instruction remediation examples

Grounding:

```text
Answer using only information available from the retrieved workplace sources. If the sources do not contain enough evidence, say what is missing instead of guessing.
```

Citations:

```text
When summarizing workplace information, include citations for the source messages, meetings, or documents whenever citations are available.
```

Scope:

```text
If the user asks for work outside this agent's supported scenarios, briefly explain the supported scope and offer a related prompt the agent can answer.
```

Format:

```text
Use this response shape: Summary, Key details, Open risks, Next actions. Keep each section concise.
```

Multi-turn:

```text
For follow-up questions, preserve the active project, customer, and time window from the prior turn unless the user changes them.
```

## Eval dataset remediation

Sometimes the eval is the problem. Update the dataset when:

- the prompt is ambiguous outside hidden context,
- the expected response demands exact phrasing when flexible phrasing is acceptable,
- the evaluator threshold is stricter than the user requirement,
- the prompt contains stale or unavailable source data,
- a setup failure was recorded as if it were an agent-quality failure.

Prefer adding a new regression item for each distinct production issue instead of overloading one prompt with many requirements.

## Reporting format

When handing back recommendations, use this shape:

```text
Primary issue: <one-sentence theme>
Evidence: <sanitized score/prompt/response observation>
Recommended change: <specific instruction, manifest, data, or eval update>
Expected effect: <which evaluator should improve and why>
```

Do not include raw sensitive prompts, responses, retrieved data, or debug logs.

````


### `references/result-analysis.md`

````markdown
# Result analysis

Use this reference after an evaluation run has produced JSON, CSV, or HTML output.

## Current output shape

JSON output is an eval-document-style object:

```json
{
  "schemaVersion": "1.2.0",
  "metadata": {
    "evaluatedAt": "2025-01-01T00:00:00Z",
    "agentId": "<agent-id>",
    "agentName": "<agent-name>",
    "cliVersion": "<version>"
  },
  "default_evaluators": {},
  "items": [
    {
      "prompt": "What can this agent help me with?",
      "response": "The agent response.",
      "expected_response": "The expected behavior.",
      "scores": {
        "relevance": {
          "score": 4,
          "result": "pass",
          "threshold": 3
        }
      }
    }
  ]
}
```

The `scores` object is sparse. A missing score key usually means that evaluator was not configured for that item.

CSV output includes metadata comments, aggregate statistics, single-turn and multi-turn sections, and serialized score JSON. HTML output is best for human review and may open in a browser.

## Score keys and semantics

| Score key | Authoring evaluator | Interpretation |
|---|---|---|
| `relevance` | `Relevance` | 1-5 LLM score for whether the answer addresses the prompt. |
| `coherence` | `Coherence` | 1-5 LLM score for clarity and structure. |
| `groundedness` | `Groundedness` | 1-5 LLM score for support from provided or retrieved context. |
| `similarity` | `Similarity` | 1-5 LLM comparison to `expected_response`. |
| `citations` | `Citations` | Count-based citation result against the configured threshold. |
| `exactMatch` | `ExactMatch` | Boolean exact match result. |
| `partialMatch` | `PartialMatch` | 0.0-1.0 partial string match result. |

Analyze only keys that are present. Do not assume a missing evaluator failed.

## Triage patterns

| Signal | Likely root cause | Recommended remediation |
|---|---|---|
| Low `relevance` | Wrong intent, vague prompt, or agent ignored the ask | Clarify instructions, add examples, improve scenario routing, or update the eval prompt if it is ambiguous. |
| Low `coherence` | Response is disorganized, contradictory, or too verbose | Tighten response-format instructions and add expected structure. |
| Low `groundedness` | Response uses unsupported facts or ignores context | Improve grounding instructions, retrieval configuration, source selection, or refusal behavior when evidence is missing. |
| Low `similarity` | Actual response diverges from expected answer | Check whether expected response is too rigid; otherwise update instructions or data access. |
| Failed `citations` | Response lacks required source citations | Add citation requirements to instructions and verify source-citation support in the agent. |
| Failed `exactMatch` | Deterministic output is not exact | Use stricter formatting instructions or replace with `PartialMatch` if exact text is not necessary. |
| Low `partialMatch` | Key expected terms are missing | Add expected terminology to instructions or adjust expected response to match acceptable variants. |
| Auth, consent, schema, or model errors | Environment failure | Fix setup before judging agent quality. |

## Multi-turn analysis

For multi-turn items, check whether failure starts on the first failing turn or compounds across turns:

1. First-turn failure usually means prompt handling, grounding, or data access is broken.
2. Later-turn failure usually means the agent loses conversation context or mishandles follow-up references.
3. Mixed pass/fail patterns can indicate evaluator thresholds are too strict for some turns.

Keep conversation IDs and turn order stable when comparing runs.

## Comparing runs over time

For regression checks:

1. Keep the same dataset and expected responses.
2. Keep evaluator names and thresholds stable.
3. Use the same Azure OpenAI model deployment when comparing score movement.
4. Save outputs with explicit names, for example `.evals\2025-01-01-baseline.json` and `.evals\2025-01-02-after-instructions.json`.
5. Separate environment failures from quality failures before reporting a pass rate.

## Reporting recommendations

When summarizing results to the user:

- Lead with pass/fail themes, not raw JSON.
- Group failures by likely fix area: instructions, grounding/retrieval, citations, expected-answer quality, capability gap, or setup.
- Include the smallest concrete manifest/instruction change that is likely to address the issue.
- Do not paste raw prompts, retrieved data, responses, or logs if they may contain sensitive content.

````


### `references/workflow.md`

````markdown
# M365 Copilot eval workflow

Use this workflow when the user wants to set up, author, run, or analyze evaluations with the public preview `@microsoft/m365-copilot-eval` CLI.

## Canonical command

Always invoke the CLI through the public npm package:

```powershell
npx -y --package @microsoft/m365-copilot-eval@latest runevals
```

Do not use private-preview installers, global installs, bare `runevals`, bare `npx runevals`, or retired flags such as `--input` and `--html`.

## 1. Detect project shape

Default to the Agents Toolkit path, but support explicit agent IDs for non-ATK projects.

| Project shape | Signals | Agent ID source |
|---|---|---|
| ATK / Teams Toolkit | `.env.local`, `.env.local.user`, `env\.env.local.user`, `m365agents.yml`, `appPackage\declarativeAgent.json` | `M365_TITLE_ID` from `.env.local`, or `M365_AGENT_ID` |
| Non-ATK | Eval dataset plus named env files or explicit CLI args | `M365_AGENT_ID` or `--m365-agent-id` |

If no agent ID can be found, ask the user for the deployed M365 Copilot agent ID or tell them to add it to a local env file.

## 2. Verify local prerequisites

Run safe checks that do not reveal secret values:

```powershell
node --version
npx -y --package @microsoft/m365-copilot-eval@latest runevals --version
npx -y --package @microsoft/m365-copilot-eval@latest runevals --help
```

Current public docs require Node.js 24.12.0 or newer and describe authentication support as Windows-first. The user also needs a Microsoft 365 Copilot license, a deployed M365 Copilot agent, tenant admin consent for the WorkIQ Client App, and Azure OpenAI in Foundry Models configuration.

Also check for stale global/PATH installs before troubleshooting the agent:

```powershell
Get-Command runevals -All
npm list -g @microsoft/m365-copilot-eval --depth=0
npm view @microsoft/m365-copilot-eval version
npx -y --package @microsoft/m365-copilot-eval@latest where runevals
```

If bare `runevals` reports `This version of the M365 Evals CLI has stopped working and must be updated`, the shell is resolving an outdated global shim. Use the package-scoped `npx --package @microsoft/m365-copilot-eval@latest` command, and only remove the global package after user confirmation.

## 3. Accept the EULA and initialize

Use these commands for first-time setup:

```powershell
npx -y --package @microsoft/m365-copilot-eval@latest runevals accept-eula
npx -y --package @microsoft/m365-copilot-eval@latest runevals --init-only
```

`--init-only` validates setup and can create starter files without running a full tenant-dependent evaluation.

## 4. Prepare env files

Use `references\azure-setup.md` for full setup details. Keep secrets out of `.env.local`.

Minimal ATK layout:

```text
.env.local              # non-secret ATK values, for example M365_TITLE_ID
.env.local.user         # local secrets
env\.env.local.user     # alternate local secrets path
```

Minimal non-ATK layout:

```text
env\.env.dev            # named environment selected with --env dev
evals\evals.json        # evaluation dataset
```

Required values are:

```text
TENANT_ID=<tenant-guid>
AZURE_AI_OPENAI_ENDPOINT=<foundry-models-endpoint>
AZURE_AI_API_KEY=<secret>
```

Recommended/default model values:

```text
AZURE_AI_API_VERSION=2024-12-01-preview
AZURE_AI_MODEL_NAME=gpt-4o-mini
```

Set one of:

```text
M365_TITLE_ID=<atk-title-id>
M365_AGENT_ID=<deployed-agent-id>
```

## 5. Create or validate the dataset

The CLI auto-discovers `prompts.json`, `evals.json`, or `tests.json` in the current directory or in `evals\`. Prefer `evals\evals.json` for new work.

Use schema version `1.2.0` with root `items`:

```json
{
  "schemaVersion": "1.2.0",
  "metadata": {
    "name": "Agent regression suite",
    "tags": ["regression"]
  },
  "default_evaluators": {
    "Relevance": {},
    "Coherence": {}
  },
  "items": [
    {
      "prompt": "What can this agent help me with?",
      "expected_response": "The agent describes only supported capabilities."
    }
  ]
}
```

Use `references\eval-templates.md` for copyable single-turn, multi-turn, and evaluator-threshold examples.

## 6. Run evaluations

Batch run with explicit output:

```powershell
npx -y --package @microsoft/m365-copilot-eval@latest runevals --prompts-file evals\evals.json --output .evals\results.json
```

Human-review HTML:

```powershell
npx -y --package @microsoft/m365-copilot-eval@latest runevals --prompts-file evals\evals.json --output .evals\results.html
```

Spreadsheet-friendly CSV:

```powershell
npx -y --package @microsoft/m365-copilot-eval@latest runevals --prompts-file evals\evals.json --output .evals\results.csv
```

Inline smoke test:

```powershell
npx -y --package @microsoft/m365-copilot-eval@latest runevals --prompts "What can you help me with?" --expected "The agent explains its supported scope."
```

Interactive exploration:

```powershell
npx -y --package @microsoft/m365-copilot-eval@latest runevals --interactive
```

Named environment:

```powershell
npx -y --package @microsoft/m365-copilot-eval@latest runevals --prompts-file evals\evals.json --env dev
```

Explicit non-ATK agent:

```powershell
npx -y --package @microsoft/m365-copilot-eval@latest runevals --prompts-file evals\evals.json --m365-agent-id <agent-id> --env dev
```

Controlled concurrency:

```powershell
npx -y --package @microsoft/m365-copilot-eval@latest runevals --prompts-file evals\evals.json --concurrency 1 --output .evals\debug.json
npx -y --package @microsoft/m365-copilot-eval@latest runevals --prompts-file evals\evals.json --concurrency 5 --output .evals\batch.json
```

Use values from 1 to 5 only. Start with 1 while debugging auth, schema, or agent behavior.

## 7. Manage local state

Use these only when needed, and warn the user first because they affect local state:

```powershell
npx -y --package @microsoft/m365-copilot-eval@latest runevals --cache-info
npx -y --package @microsoft/m365-copilot-eval@latest runevals --cache-dir
npx -y --package @microsoft/m365-copilot-eval@latest runevals --cache-clear
npx -y --package @microsoft/m365-copilot-eval@latest runevals --signout
```

`--cache-clear` can remove cached run data. `--signout` resets the local authentication session.

## 8. Analyze and iterate

Load `references\result-analysis.md` after a run. Treat setup/auth/schema failures separately from agent-quality failures. For regression comparisons, keep the same dataset, expected responses, evaluator set, thresholds, model deployment, and concurrency where possible.

Do not run real evaluations unless the user has provided or approved the tenant, deployed agent, and Azure OpenAI configuration.

````