This format is enabled for Google models as they seem to prefer it.
A relevant unit eval's pass rate has increased from 0.77 to 0.98.
Diff-fenced format looks like this (markdown fences and a line hint are
optional):
```diff
<<<<<<< SEARCH line=42
...
=======
...
>>>>>>> REPLACE
```
Release Notes:
- Agent: Gemini models now use the diff-fenced format when making edits
These changes help the agent edit files when `<old_text>` matches more
than one location.
First, the agent can specify an optional `<old_text line=XX>` parameter.
When this is provided and multiple matches exist, we use this hint to
identify the best match.
Second, when there is ambiguity in matches, we now return the agent a
more helpful message listing the line numbers of all possible matches.
Together, these changes should reduce the number of misplaced edits and
agent confusion.
I have ensured the LLM Worker works with these prompt changes.
Release Notes:
- Agent: Improved locating edits
This change instructs models to wrap new file content in Markdown fences
and introduces a parser for this format. The reasons are:
1. This is the format we put a lot of effort into explaining in the
system prompt.
2. Gemini really prefers to do it.
3. It adds an option for a model to think before writing the content
The `eval_zode` pass rate for GEmini models goes from 0% to 100%. Other
models were already at 100%, this hasn't changed.
Release Notes:
- N/A
This change improves `eval_extract_handle_command_output` results for
all models:
Model | Pass rate before | Pass rate after
----------------------------|------------------|----------------
claude-3.7-sonnet | 0.96 | 0.98
gemini-2.5-pro | 0.35 | 0.86
gpt-4.1 | 0.81 | 1.00
Part of this improvement comes from more robust evaluation, which now
accepts multiple possible outcomes. Another part is from the prompt
adaptation: addressing common Gemini failure modes, adding a few-shot
example, and, in the final commit, auto-rewriting instructions for
clarity and conciseness.
This change still needs validation from larger end-to-end evals.
Release Notes:
- N/A
1. Add system prompt: this is how it's called from threads. Previously,
we were sending
2. Fix an issue with writing agent thought into a newly created empty
file.
Release Notes:
- N/A
---------
Co-authored-by: Ben Brandt <benjamin.j.brandt@gmail.com>
Co-authored-by: Antonio Scandurra <me@as-cii.com>
This improves the new eval scenario by ~80% (`0.29` vs `0.525`) without
decreasing performance in the other evals.
Release Notes:
- Improved the performance of the `edit_file` tool.
Refs #29733
This pull request introduces a new field to the `StreamingEditFileTool`
that lets the model create or overwrite a file in a streaming way. When
one of the `assistant.stream_edits` setting / `agent-stream-edits`
feature flag is enabled, we are going to disable the `CreateFileTool` so
that the agent model can only use `StreamingEditFileTool` for file
creation.
Release Notes:
- N/A
---------
Co-authored-by: Ben Brandt <benjamin.j.brandt@gmail.com>
Co-authored-by: Oleksiy Syvokon <oleksiy.syvokon@gmail.com>
This pull request introduces a new tool for streaming edits. The
short-term goal is for this tool to replace the existing `EditFileTool`,
but we want to get this out the door as soon as possible so that we can
start testing it.
`StreamingEditFileTool` is mutually exclusive with `EditFileTool`. It
will be enabled by default for anyone who has the `agent-stream-edits`
feature flag, as well as people that set `assistant.stream_edits` to
`true` in their settings.
### Implementation
Streaming is achieved by requesting a completion while the `edit_file`
tool gets called. We invoke the model by taking the existing
conversation with the agent and appending a prompt specifically tailored
for editing. In that prompt, we ask the model to produce a stream of
`<old_text>`/`<new_text>` tags. As the model streams text in, we
incrementally parse it and start editing as soon as we can.
### Evals
Note that, as part of this pull request, I also defined some new evals
that I used to drive the behavior of the recursive LLM call. To run
them, use this command:
```bash
cargo test --package=assistant_tools --features eval -- eval_extract_handle_command_output
```
Or comment out the `#[cfg_attr(not(feature = "eval"), ignore)]` macro.
I recommend running them one at a time, because right now we don't
really have a way of orchestrating of all these evals. I think we should
invest into that effort once the new agent panel goes live.
Release Notes:
- N/A
---------
Co-authored-by: Nathan Sobo <nathan@zed.dev>
Co-authored-by: Bennet Bo Fenner <bennetbo@gmx.de>
Co-authored-by: Oleksiy Syvokon <oleksiy.syvokon@gmail.com>
Oleksiy Syvokon
