Previously, the vision request header was only set if the last message
in a thread contained an image. This caused 400 errors from the Copilot
API when sending follow-up messages in a thread that contained images in
earlier messages.
Modified the `is_vision_request` check to scan all messages in a thread
for image content instead of just the last one, ensuring the proper
header is set for the entire conversation.
Added a unit test to verify all cases function correctly.
Release Notes:
- Fix GitHub Copilot chat provider error when sending follow-up messages
in threads containing images
Issues: #30994
I've implemented an important optimisation in response to GitHub
Copilot's recent rate limit on concurrent Vision API calls. Previously,
our system was defaulting to vision header: true for all API calls. To
prevent unnecessary calls and adhere to the new limits, I've updated our
logic: the vision header is now only sent if the current message is a
vision message, specifically when the preceding message includes an
image.
Prompt used to reproduce and verify the fix: `Give me a context for my
agent crate about. Browse my repo.`
Release Notes:
- copilot: Set Copilot-Vision-Request header based on message content
https://github.com/zed-industries/zed/issues/30972 brought up another
case where our context is not enough to track the actual source of the
issue: we get a general top-level error without inner error.
The reason for this was `.ok_or_else(|| anyhow!("failed to read HEAD
SHA"))?; ` on the top level.
The PR finally reworks the way we use anyhow to reduce such issues (or
at least make it simpler to bubble them up later in a fix).
On top of that, uses a few more anyhow methods for better readability.
* `.ok_or_else(|| anyhow!("..."))`, `map_err` and other similar error
conversion/option reporting cases are replaced with `context` and
`with_context` calls
* in addition to that, various `anyhow!("failed to do ...")` are
stripped with `.context("Doing ...")` messages instead to remove the
parasitic `failed to` text
* `anyhow::ensure!` is used instead of `if ... { return Err(...); }`
calls
* `anyhow::bail!` is used instead of `return Err(anyhow!(...));`
Release Notes:
- N/A
This is very basic support for them. There are a number of other TODOs
before this is really a first-class supported feature, so not adding any
release notes for it; for now, this PR just makes it so that if
read_file tries to read a PNG (which has come up in practice), it at
least correctly sends it to Anthropic instead of messing up.
This also lays the groundwork for future PRs for more first-class
support for images in tool calls across more image file formats and LLM
providers.
Release Notes:
- N/A
---------
Co-authored-by: Agus Zubiaga <hi@aguz.me>
Co-authored-by: Agus Zubiaga <agus@zed.dev>
Problem Statement:
Support for image analysis (vision) is currently restricted to Anthropic
and Gemini models. This limits users who wish to leverage vision
capabilities available in other models, such as Copilot, for tasks like
attaching image context within the agent message editor.
Proposed Change:
This PR extends vision support to include Copilot models that are
already equipped with vision capabilities. This integration will allow
users within VS Code to attach and analyze images using supported
Copilot models via the agent message editor.
Scope Limitation:
This PR does not implement controls within the message editor to ensure
that image context (e.g., through copy-paste or attachment) is
exclusively enabled or prompted only when a vision-supported model is
active. Long term the message editor should have access to each models
vision capability and stop the users from attaching images by either
greying out the context saying it's not support or not work through both
copy paste and file/directory search.
Closes#30076
Release Notes:
- Add vision support for Copilot Chat models
---------
Co-authored-by: Bennet Bo Fenner <bennet@zed.dev>
I noticed the discussion in #28881, and had thought of exactly the same
a few days prior.
This implementation should preserve existing functionality fairly well.
I've added a dependency (serde_with) to allow the deserializer to skip
models which cannot be deserialized, which could occur if a future
provider, for instance, is added. Without this modification, such a
change could break all models. If extra dependencies aren't desired, a
manual implementation could be used instead.
- Closes#29369
Release Notes:
- Dynamically detect available Copilot Chat models, including all models
with tool support
---------
Co-authored-by: AidanV <aidanvanduyne@gmail.com>
Co-authored-by: imumesh18 <umesh4257@gmail.com>
Co-authored-by: Bennet Bo Fenner <bennet@zed.dev>
Co-authored-by: Agus Zubiaga <hi@aguz.me>
* Adds a fast / cheaper model to providers and defaults thread
summarization to this model. Initial motivation for this was that
https://github.com/zed-industries/zed/pull/29099 would cause these
requests to fail when used with a thinking model. It doesn't seem
correct to use a thinking model for summarization.
* Skips system prompt, context, and thinking segments.
* If tool use is happening, allows 2 tool uses + one more agent response
before summarizing.
Downside of this is that there was potential for some prefix cache reuse
before, especially for title summarization (thread summarization omitted
tool results and so would not share a prefix for those). This seems fine
as these requests should typically be fairly small. Even for full thread
summarization, skipping all tool use / context should greatly reduce the
token use.
Release Notes:
- N/A
Release Notes:
- Add support for OpenAI o3 and o4-mini models via OpenAI API and
Copilot Chat providers.
---------
Co-authored-by: Peter Tripp <peter@zed.dev>
Release Notes:
- Add support for OpenAI GPT-4.1 via Copilot Chat and OpenAI API
---------
Co-authored-by: Danilo Leal <daniloleal09@gmail.com>
Co-authored-by: Bennet Bo Fenner <bennetbo@gmx.de>
This PR adds tool calling support for GitHub Copilot Chat models.
Currently only supports the Claude family of models.
Release Notes:
- agent: Added tool calling support for Claude models in GitHub Copilot
Chat.
---------
Co-authored-by: Marshall Bowers <git@maxdeviant.com>
This is the core change:
https://github.com/zed-industries/zed/pull/26758/files#diff-044302c0d57147af17e68a0009fee3e8dcdfb4f32c27a915e70cfa80e987f765R1052
TODO:
- [x] Use AsyncFn instead of Fn() -> Future in GPUI spawn methods
- [x] Implement it in the whole app
- [x] Implement it in the debugger
- [x] Glance at the RPC crate, and see if those box future methods can
be switched over. Answer: It can't directly, as you can't make an
AsyncFn* into a trait object. There's ways around that, but they're all
more complex than just keeping the code as is.
- [ ] Fix platform specific code
Release Notes:
- N/A
Closes#25594
This PR fixes an issue where signing into Copilot required restarting
Zed.
Copilot depends on an OAuth token that comes from either `hosts.json` or
`apps.json`. Initially, both files don't exist. If neither file is
found, we fallback to watching `hosts.json` for updates. However, if the
auth process creates `apps.json`, we won't receive updates from it,
causing the UI to remain outdated.
This PR fixes that by watching the parent `github-copilot` directory
instead, which will always contain one of those files along with an
additional version file.
I have tested this on macOS and Linux Wayland.
Release Notes:
- Fixed an issue where signing into Copilot required restarting Zed.
Fix a bug where a GPUI macro still used `ModelContext`
Rename `AsyncAppContext` -> `AsyncApp`
Rename update_model, read_model, insert_model, and reserve_model to
update_entity, read_entity, insert_entity, and reserve_entity
Release Notes:
- N/A
There's still a bit more work to do on this, but this PR is compiling
(with warnings) after eliminating the key types. When the tasks below
are complete, this will be the new narrative for GPUI:
- `Entity<T>` - This replaces `View<T>`/`Model<T>`. It represents a unit
of state, and if `T` implements `Render`, then `Entity<T>` implements
`Element`.
- `&mut App` This replaces `AppContext` and represents the app.
- `&mut Context<T>` This replaces `ModelContext` and derefs to `App`. It
is provided by the framework when updating an entity.
- `&mut Window` Broken out of `&mut WindowContext` which no longer
exists. Every method that once took `&mut WindowContext` now takes `&mut
Window, &mut App` and every method that took `&mut ViewContext<T>` now
takes `&mut Window, &mut Context<T>`
Not pictured here are the two other failed attempts. It's been quite a
month!
Tasks:
- [x] Remove `View`, `ViewContext`, `WindowContext` and thread through
`Window`
- [x] [@cole-miller @mikayla-maki] Redraw window when entities change
- [x] [@cole-miller @mikayla-maki] Get examples and Zed running
- [x] [@cole-miller @mikayla-maki] Fix Zed rendering
- [x] [@mikayla-maki] Fix todo! macros and comments
- [x] Fix a bug where the editor would not be redrawn because of view
caching
- [x] remove publicness window.notify() and replace with
`AppContext::notify`
- [x] remove `observe_new_window_models`, replace with
`observe_new_models` with an optional window
- [x] Fix a bug where the project panel would not be redrawn because of
the wrong refresh() call being used
- [x] Fix the tests
- [x] Fix warnings by eliminating `Window` params or using `_`
- [x] Fix conflicts
- [x] Simplify generic code where possible
- [x] Rename types
- [ ] Update docs
### issues post merge
- [x] Issues switching between normal and insert mode
- [x] Assistant re-rendering failure
- [x] Vim test failures
- [x] Mac build issue
Release Notes:
- N/A
---------
Co-authored-by: Antonio Scandurra <me@as-cii.com>
Co-authored-by: Cole Miller <cole@zed.dev>
Co-authored-by: Mikayla <mikayla@zed.dev>
Co-authored-by: Joseph <joseph@zed.dev>
Co-authored-by: max <max@zed.dev>
Co-authored-by: Michael Sloan <michael@zed.dev>
Co-authored-by: Mikayla Maki <mikaylamaki@Mikaylas-MacBook-Pro.local>
Co-authored-by: Mikayla <mikayla.c.maki@gmail.com>
Co-authored-by: joão <joao@zed.dev>
This PR updates the `gpui::prelude` to not export the `Context` trait
named.
This prevents some naming clashes in downstream consumers.
Release Notes:
- N/A
This removes the `low_speed_timeout` setting from all providers as a
response to issue #19509.
Reason being that the original `low_speed_timeout` was only as part of
#9913 because users wanted to _get rid of timeouts_. They wanted to bump
the default timeout from 5sec to a lot more.
Then, in the meantime, the meaning of `low_speed_timeout` changed in
#19055 and was changed to a normal `timeout`, which is a different thing
and breaks slower LLMs that don't reply with a complete response in the
configured timeout.
So we figured: let's remove the whole thing and replace it with a
default _connect_ timeout to make sure that we can connect to a server
in 10s, but then give the server as long as it wants to complete its
response.
Closes#19509
Release Notes:
- Removed the `low_speed_timeout` setting from LLM provider settings,
since it was only used to _increase_ the timeout to give LLMs more time,
but since we don't have any other use for it, we simply remove the
setting to give LLMs as long as they need.
---------
Co-authored-by: Antonio <antonio@zed.dev>
Co-authored-by: Peter Tripp <peter@zed.dev>
Users of our http_client crate knew they were interacting with isahc as
they set its extensions on the request. This change adds our own
equivalents for their APIs in preparation for changing the default http
client.
Release Notes:
- N/A
This PR moves the `copilot_chat_config_path` out of `paths` and into
`copilot` with the rest of the Copilot code.
Since this doesn't actually return a Zed path, it doesn't belong in
`paths`.
Release Notes:
- N/A
We also eliminate the `completion` crate and moved its logic into
`LanguageModelRegistry`.
Release Notes:
- N/A
---------
Co-authored-by: Nathan <nathan@zed.dev>
# Summary
This commit implements Github Copilot Chat support within the existing
Assistant panel/framework. It required a little bit of trickery and
internal API modification, as Copilot doesn't use the same
authentication-style as all of the existing providers, opting to use
OAuth and a short lived API key instead of a straight API key. All
existing Assistant features should work.
Release Notes:
- Added Github Copilot Chat support
([#4673](https://github.com/zed-industries/zed/issues/4673)).
## Screenshots
<img width="1552" alt="A screenshot showing a conversation between a
user and Github Copilot Chat within the Zed editor."
src="https://github.com/user-attachments/assets/73eaf6a2-792b-4c40-a7fe-f763bd6417d7">
---------
Co-authored-by: Bennet Bo Fenner <bennet@zed.dev>
Umesh Yadav
Stephen Murray
Kirill Bulatov
Agus Zubiaga
Bennet Bo Fenner
Antonio Scandurra
Michael Sloan
Richard Hao
Piotr Osiewicz
isomo
Mikayla Maki
Smit Barmase
zhaopengandPeter Tripp
Jacob ChapelandMarshall Bowers
Marcel KerstenandPeter Tripp
fred-schandBennet Bo Fenner
dhaus67andPeter Tripp
Jonathan ToledoandBennet Bo Fenner
Conrad Irwin
Ryan Hawkins