smit
5b1b3c51d4
Closes #31108 The `num_tokens_from_messages` method we use from `tiktoken-rs` creates new BPE every time that method is called. This creation of BPE is expensive as well as has some underlying issue that keeps memory from releasing once the method is finished, specifically noticeable on Linux. This leads to a gradual increase in memory every time that method is called in my case around +50MB on each call. We call this method with debounce every time user types in Agent Panel to calculate tokens. This can add up really fast. This PR lands quick fix, while I/maintainers figure out underlying issue. See upstream discussion: https://github.com/zurawiki/tiktoken-rs/issues/39. Here on fork https://github.com/zed-industries/tiktoken-rs/pull/1, instead of creating BPE instances every time that method is called, we use singleton BPE instances instead. So, whatever memory it is holding on to, at least that is only once per model. Before: Increase of 700MB+ on extensive use On init: <img width="500" alt="prev-init" src="https://github.com/user-attachments/assets/70da7c44-60cb-477b-84aa-7dd579baa3da" /> First message: <img width="500" alt="prev-first-call" src="https://github.com/user-attachments/assets/599ffc48-3ad3-4729-b94c-6d88493afdbf" /> Extensive use: <img width="500" alt="prev-extensive-use" src="https://github.com/user-attachments/assets/e0e6b688-6412-486d-8b2e-7216c6b62470" /> After: Increase of 50MB+ on extensive use On init: <img width="500" alt="now-init" src="https://github.com/user-attachments/assets/11a2cd9c-20b0-47ae-be02-07ff876e68ad" /> First message: <img width="500" alt="now-first-call" src="https://github.com/user-attachments/assets/ef505f8d-cd31-49cd-b6bb-7da3f0838fa7" /> Extensive use: <img width="500" alt="now-extensive-use" src="https://github.com/user-attachments/assets/513cb85a-a00b-4f11-8666-69103a9eb2b8" /> Release Notes: - Fixed issue where Agent Panel would cause high memory consumption over prolonged use.
Zed
Welcome to Zed, a high-performance, multiplayer code editor from the creators of Atom and Tree-sitter.
Installation
On macOS and Linux you can download Zed directly or install Zed via your local package manager.
Other platforms are not yet available:
- Windows (tracking issue)
- Web (tracking issue)
Developing Zed
- Building Zed for macOS
- Building Zed for Linux
- Building Zed for Windows
- Running Collaboration Locally
Contributing
See CONTRIBUTING.md for ways you can contribute to Zed.
Also... we're hiring! Check out our jobs page for open roles.
Licensing
License information for third party dependencies must be correctly provided for CI to pass.
We use cargo-about to automatically comply with open source licenses. If CI is failing, check the following:
- Is it showing a
no license specifiederror for a crate you've created? If so, addpublish = falseunder[package]in your crate's Cargo.toml. - Is the error
failed to satisfy license requirementsfor a dependency? If so, first determine what license the project has and whether this system is sufficient to comply with this license's requirements. If you're unsure, ask a lawyer. Once you've verified that this system is acceptable add the license's SPDX identifier to theacceptedarray inscript/licenses/zed-licenses.toml. - Is
cargo-aboutunable to find the license for a dependency? If so, add a clarification field at the end ofscript/licenses/zed-licenses.toml, as specified in the cargo-about book.