Files

T

Marshall Bowers 5c0b161563 Handle new refusal stop reason from Claude 4 models (#31217 )

This PR adds support for handling the new [`refusal` stop
reason](https://docs.anthropic.com/en/docs/test-and-evaluate/strengthen-guardrails/handle-streaming-refusals)
from Claude 4 models.

<img width="409" alt="Screenshot 2025-05-22 at 4 31 56 PM"
src="https://github.com/user-attachments/assets/707b04f5-5a52-4a19-95d9-cbd2be2dd86f"
/>

Release Notes:

- Added handling for `"stop_reason": "refusal"` from Claude 4 models.

2025-05-22 16:56:59 -04:00

docs

eval: Add HTML overview for evaluation runs (#29413 )

2025-04-25 17:49:05 +03:00

src

Handle new refusal stop reason from Claude 4 models (#31217 )

2025-05-22 16:56:59 -04:00

.gitignore

Add judge to new eval + provide LSP diagnostics (#28713 )

2025-04-14 20:18:47 +00:00

Cargo.toml

chore: Make terminal_view own the TerminalSlashCommand (#31070 )

2025-05-21 09:27:54 +00:00

LICENSE-GPL

…

README.md

eval: Add support for reading from a .env file (#29426 )

2025-04-25 15:53:02 +00:00

runner_settings.json

Introduce a new StreamingEditFileTool (#29733 )

2025-05-01 17:37:43 +02:00

README.md

Eval

This eval assumes the working directory is the root of the repository. Run it with:

cargo run -p eval

The eval will optionally read a .env file in crates/eval if you need it to set environment variables, such as API keys.

Explorer Tool

The explorer tool generates a self-contained HTML view from one or more thread JSON file. It provides a visual interface to explore the agent thread, including tool calls and results. See ./docs/explorer.md for more details.

Usage

cargo run -p eval --bin explorer -- --input <path-to-json-files> --output <output-html-path>

Example:

cargo run -p eval --bin explorer -- --input ./runs/2025-04-23_15-53-30/fastmcp_bugifx/*/last.messages.json --output /tmp/explorer.html