Commit Graph
70 Commits
Author SHA1 Message Date
458ffaa134 Add new action to run agent eval (#29158)
The old one wasn't linking, and
https://github.com/zed-industries/zed/pull/29081 has a bunch of merge
conflicts. Wanted to start simple/small.

## Todo

* [x] Remove low-signal examples
* [x] Make the eval run on a cron, on main, and on any PR with the
`run-eval` label
* [x] Noise in logs about failure to write settings
    ```
[2025-04-21T20:45:04Z ERROR settings] Failed to write settings to file
"/home/runner/.config/zed/settings.json"
    
       Caused by:
No such file or directory (os error 2) at path
"/home/runner/.config/zed/.tmpLewFEs"
    ```
* [x] `Agentic loop stalled`
(https://github.com/zed-industries/zed/actions/runs/14581044243/job/40897622894)
* [x] Make sure that events are recorded in snowflake
* [ ] Change judge criteria to be more explicit about meanings of scores

Release Notes:

- N/A

---------

Co-authored-by: Antonio Scandurra <me@as-cii.com>
Co-authored-by: Agus Zubiaga <hi@aguz.me>
Co-authored-by: Max Brunsfeld <maxbrunsfeld@gmail.com>
Co-authored-by: Thomas Mickley-Doyle <tmickleydoyle@gmail.com>
2025-04-21 21:30:21 -07:00
Max Brunsfeld 0dc3dffe38 Use insert_id as partition key for crash events (#28293)
Release Notes:

- N/A
2025-04-07 17:24:31 -07:00
Max Brunsfeld d385a60ed1 Fix phrasing of crash/panic event names 2025-04-07 16:31:29 -07:00
Max Brunsfeld 6220b86f94 Write panics and crashes to snowflake (#28284)
This will let us create a better crashes dashboard, using Hex.

Release Notes:

- N/A
2025-04-07 15:50:16 -07:00
Piotr Osiewicz dc64ec9cc8 chore: Bump Rust edition to 2024 (#27800)
Follow-up to https://github.com/zed-industries/zed/pull/27791

Release Notes:

- N/A
2025-03-31 20:55:27 +02:00
Joseph T. Lyons ed510b5e93 Remove unused AssistantThreadFeedback event (#27021)
It looks like:

- https://github.com/zed-industries/zed/pull/26780

accidentally added a new event type, `AssistantThreadFeedback`, using
the old event system, that it didn't end up actually using, as the code
actually relies on using the newer (preferred) `telemetry::event!()`.

Release Notes:

- N/A
2025-03-18 20:39:54 +00:00
f68a475eca Introduce rating for assistant threads (#26780)
Release Notes:

- N/A

---------

Co-authored-by: Richard Feldman <oss@rtfeldman.com>
Co-authored-by: Agus Zubiaga <hi@aguz.me>
2025-03-14 14:41:50 +00:00
Richard Feldman 667396c44b Use the term "edit prediction" over "inline completion" (#24211)
Note that this does *not* involve any breaking code changes.

cc @0xtimsb - I didn't change any settings or anything here. That can
happen separately!

Release Notes:

- N/A
2025-02-04 10:33:01 -08:00
Joseph T. Lyons a8741dc310 Migrate more events to telemetry::event! macro (#24102)
I believe this takes care of the remaining events running through the
old flow that requires transformation at the collab server level. I
think all events are now going through `telemetry::event!()`.

For anyone curious where the new telemetry names are coming from, you
can check the `for_snowflake` function within
`crates/collab/src/api/events.rs`, to see how collab is currently
transforming the events going through the old flow.

Release Notes:

- N/A
2025-02-03 16:38:45 +00:00
Cole MillerandConrad a2022d7da3 Improve Linux panic reporting (#22202)
- [x] Upload separate debug symbols for Linux binaries to DigitalOcean
- [x] Send raw offsets with panic report JSON on Linux
- [x] Update `symbolicate` script to handle Linux crashes
- [x] Demangle backtraces 🎉 
- [x] Check that it works
- [x] Improve deduplication (?)
 
Release Notes:

- N/A

---------

Co-authored-by: Conrad <conrad@zed.dev>
2024-12-22 08:20:17 +00:00
Conrad Irwin 7425d242bc Add telemetry::event! (#22146)
CC @JosephTLyons

Release Notes:

- N/A
2024-12-17 11:39:18 -07:00
Joseph T. Lyons f80eb73213 Update event type to conform to standard (#21827)
Release Notes:

- N/A
2024-12-10 16:14:31 -05:00
Conrad Irwin 03efd0d1d9 Stop sending data to Clickhouse (#21763)
Release Notes:

- N/A
2024-12-10 08:47:29 -07:00
Marshall Bowers 8a85d6ef96 collab: Make metrics_id required in LlmTokenClaims (#21771)
This PR makes the `metrics_id` field on the `LlmTokenClaims` required,
as we always have one in practice.

Release Notes:

- N/A
2024-12-09 17:58:14 -05:00
77b8296fbb Introduce staff-only inline completion provider (#21739)
Release Notes:

- N/A

---------

Co-authored-by: Thorsten Ball <mrnugget@gmail.com>
Co-authored-by: Bennet <bennet@zed.dev>
Co-authored-by: Thorsten <thorsten@zed.dev>
2024-12-09 14:26:36 +01:00
Joseph T. Lyons 3dcb94c204 Correct more app event inconsistencies (#21129)
- Unify welcome page event type string structure
- Differentiate between metric telemetry event and diagnostic telemetry
event

Release Notes:

- N/A
2024-11-24 00:34:02 -05:00
Joseph T. Lyons 0395d1b037 Clean up app event transformations (#21115)
This needs scrutinized. Detailed breakdown of what events I kept and
threw out here:

https://zed.dev/channel/app-events-17178/notes

I also removed a few fake events and tossed out json properties that
were being inserted for things we don't have logic to track. See PR
review comments below.

I think the only bad data we have are that we were identifying all node,
pnpm, and yarn projects as 'node' in the `project_type` property, so a
few days of lost data there.

Release Notes:

- N/A
2024-11-23 23:11:45 -05:00
Joseph T. Lyons 43f0ea759b Remove non-existent call event types (#21093)
These are not real (from Clickhouse):

```
    ┌─operation────────────┬──────c─┐
 1. │ join channel         │ 136221 │
 2. │ open channel notes   │  95529 │
 3. │ hang up              │  66264 │
 4. │ disable microphone   │  34116 │
 5. │ enable microphone    │  25090 │
 6. │ enable screen share  │  20751 │
 7. │ invite               │  15827 │
 8. │ share project        │  14580 │
 9. │ accept incoming      │  13708 │
10. │ disable screen share │  10440 │
11. │ unshare project      │   9556 │
12. │ decline incoming     │    455 │
13. │ enable camera        │      6 │
14. │ disable camera       │      4 │
    └──────────────────────┴────────┘
```

Release Notes:

- N/A
2024-11-22 23:49:53 -05:00
Conrad Irwin 984bb192ba Send llm events to snowflake too (#21091)
Closes #ISSUE

Release Notes:

- N/A
2024-11-22 20:40:39 -07:00
Joseph T. LyonsandNathan Sobo 933c11a9b2 Remove dead snowflake code (#21041)
Co-authored-by: Nathan Sobo <1789+nathansobo@users.noreply.github.com>

Release Notes:

- N/A
2024-11-21 23:06:03 -05:00
Conrad Irwin ebca6a8f3d Send os_version and country to amplitude (#20936)
Release Notes:

- N/A
2024-11-20 15:34:24 -07:00
Conrad Irwin 705a06c3dd Send Country/OS/Version amplitude style (#20884)
Release Notes:

- N/A
2024-11-19 16:38:14 -07:00
Conrad IrwinandNathan Sobo ea5131ce0a Country Code To Snowflake (#20875)
Release Notes:

- N/A

---------

Co-authored-by: Nathan Sobo <nathan@zed.dev>
2024-11-19 12:52:00 -07:00
Conrad Irwin 496dae968b Remove old CPU/Memory events (#20865)
Release Notes:

- Telemetry: stop reporting CPU/RAM on a timer
2024-11-19 12:25:16 -07:00
Nathan Sobo ee207ab77e Map "feature upsell" events to the new "Noun Verbed" format (#20787)
Release Notes:

- N/A
2024-11-17 07:38:30 -07:00
Nathan Sobo f9990b42fa Send events to Snowflake in the format they're expected by Amplitude (#20765)
This will allow us to use the events table directly in Amplitude, which
lets us use the newer event ingestion flow that detects changes to the
table. Otherwise we'll need a transformation.

I think Amplitude's API is probably a pretty good example to follow for
the raw event schema, even if we don't end up using their product. They
also recommend a "Noun Verbed" format for naming events, so I think we
should go with this. This will help us be consistent and encourage the
author of events to think more clearly about what event they're
reporting.

cc @ConradIrwin 

Release Notes:

- N/A
2024-11-16 09:58:19 -07:00
Conrad Irwin ac5ecf5487 Don't log every value (#20744)
Release Notes:

- N/A
2024-11-15 14:36:58 -07:00
Conrad IrwinandNathan 6ff69faf37 Start to send data to Snowflake too (#20698)
This PR adds support for sending telemetry events to AWS Kinesis.

In our AWS account we now have three new things:
* The [Kinesis data
stream](https://us-east-1.console.aws.amazon.com/kinesis/home?region=us-east-1#/streams/details/zed-telemetry/monitoring)
that we will actually write to.
* A [Firehose for
Axiom](https://us-east-1.console.aws.amazon.com/firehose/home?region=us-east-1#/details/telemetry-to-axiom/monitoring)
that sends events from that stream to Axiom for ad-hoc queries over
recent data.
* A [Firehose for
Snowflake](https://us-east-1.console.aws.amazon.com/firehose/home?region=us-east-1#/details/telemetry-to-snowflake/monitoring)
that sends events from that stream to Snowflake for long-term retention.
This Firehose also backs up data into an S3 bucket in case we want to
change how the system works in the future.

In a follow-up PR, we'll add support for ad-hoc telemetry events; and
slowly move away from the current Clickhouse defined schemas; though we
won't move off click house until we have what we need in Snowflake.

Co-Authored-By: Nathan <nathan@zed.dev>

Release Notes:

- N/A
2024-11-15 12:58:00 -07:00
Joseph T. Lyons 1cbaca667f Remove historical_event column in editor events (#18932)
We have a lot of data in Clickhouse. This column was used when migrating
the events dataset between analytics databases and has no purpose today.

Naive maths: 257,170,993 editor event rows * 1 byte per boolean =
257,170,993 bytes, or ~0.24 GB

I'll drop the column after deploying a new collab.

Going forward, I'd like to remove more data that we never touch, to try
to keep things more focused. We should discuss some TTL at some point.

Release Notes:

- N/A
2024-10-09 13:19:13 -04:00
Joseph T. Lyons 77bf2ad0f1 Add is_via_ssh field to edit events (#18867)
Release Notes:

- N/A
2024-10-08 13:13:40 -04:00
Joseph T. Lyons d295c46433 Remove deprecated copilot event (#18862)
`CopilotEvent` was succeeded by `InlineCompletionEvent` 5 months ago.

Release Notes:

- N/A
2024-10-08 11:10:20 -04:00
Joseph T. Lyons 4c7a6f5e7f Add is_via_ssh field to editor events (#18837)
Release Notes:

- N/A
2024-10-08 10:30:04 -04:00
Marshall Bowers 9565a90528 collab: Revert changes to Clickhouse event rows (#18654)
This PR reverts the changes to the Clickhouse event rows that were
included in https://github.com/zed-industries/zed/pull/18414.

The changes don't seem to be correct, as they make the row structs
differ from the underlying table schema.

Release Notes:

- N/A
2024-10-02 16:10:25 -04:00
Conrad IrwinandMikayla 3a5deb5c6f Replace isahc with async ureq (#18414)
REplace isahc with ureq everywhere gpui is used.

This should allow us to make http requests without libssl; and avoid a
long-tail of panics caused by ishac.

Release Notes:

- (potentially breaking change) updated our http client

---------

Co-authored-by: Mikayla <mikayla@zed.dev>
2024-10-02 12:30:48 -07:00
Marshall Bowers 167af4bc1d Use const over static for string literals (#18635)
I noticed a few places where we were storing `&'static str`s in
`static`s instead of `const`s.

This PR updates them to use `const`.

Release Notes:

- N/A
2024-10-02 12:33:13 -04:00
Conrad Irwin bbf5ed2ba1 Fix collab filtering panics better (#18344)
Release Notes:

- N/A
2024-09-25 09:42:07 -06:00
Conrad Irwin b69c6ee7df Exclude initialization failed errors from slack (#18232)
Release Notes:

- N/A
2024-09-24 10:17:43 -06:00
Joseph T. Lyons 601090511b Remove system_id from all events but editor_events (#18154)
Release Notes:

- N/A
2024-09-20 13:25:06 -04:00
edf2c19250 Hide GPU problems from Slack (#18087)
Release Notes:

- N/A

---------

Co-authored-by: Marshall <marshall@zed.dev>
Co-authored-by: Marshall Bowers <elliott.codes@gmail.com>
2024-09-19 17:28:30 -04:00
Joseph T. Lyons ca4980df02 Add system_id (#18040)
This PR adds `system_id` to telemetry, which is contained within a new
`global` database (accessible by any release channel of Zed on a single
system). This will help us get a more accurate understanding of user
count, instead of relying on `installationd_id`, which is different per
release channel. This doesn't solve the problem of a user with multiple
machines, but it gets us closer.

Release Notes:

- N/A
2024-09-19 07:20:27 -04:00
Conrad Irwin b43b800a54 More assistant events (#18032)
Release Notes:

- N/A
2024-09-18 18:07:39 -06:00
Piotr Osiewicz 095a08d9c8 chore: Another round of style lints fixes (#17519)
Closes #ISSUE

Release Notes:

- N/A
2024-09-07 02:36:55 +02:00
Piotr Osiewicz e6c1c51b37 chore: Fix several style lints (#17488)
It's not comprehensive enough to start linting on `style` group, but
hey, it's a start.

Release Notes:

- N/A
2024-09-06 11:58:39 +02:00
Marshall Bowers a9441879c3 collab: Fix writing LLM rate limit events to Clickhouse (#16367)
This PR fixes the writing of LLM rate limit events to Clickhouse.

We had a table in the table name: `llm_rate_limits` instead of
`llm_rate_limit_events`.

I also extracted a helper function to write to Clickhouse so we can use
it anywhere we need to.

Release Notes:

- N/A
2024-08-16 14:03:34 -04:00
8e9c2b1125 Introduce a separate backend service for LLM calls (#15831)
This PR introduces a separate backend service for making LLM calls.

It exposes an HTTP interface that can be called by Zed clients. To call
these endpoints, the client must provide a `Bearer` token. These tokens
are issued/refreshed by the collab service over RPC.

We're adding this in a backwards-compatible way. Right now the access
tokens can only be minted for Zed staff, and calling this separate LLM
service is behind the `llm-service` feature flag (which is not
automatically enabled for Zed staff).

Release Notes:

- N/A

---------

Co-authored-by: Marshall <marshall@zed.dev>
Co-authored-by: Marshall Bowers <elliott.codes@gmail.com>
2024-08-05 20:26:21 -04:00
Marshall Bowers f11f3f2599 collab: Attach GeoIP country code to RPC sessions (#15814)
This PR updates collab to attach the user's GeoIP country code to their
RPC session.

We source the country code from the
[`CF-IPCountry`](https://developers.cloudflare.com/fundamentals/reference/http-request-headers/#cf-ipcountry)
header.

Release Notes:

- N/A
2024-08-05 11:11:49 -04:00
Joseph T. LyonsandKyle Kelley 856d9632e4 Add repl events (#15259)
Release Notes:

- N/A

---------

Co-authored-by: Kyle Kelley <rgbkrk@gmail.com>
2024-07-26 03:31:41 -04:00
Joseph T. Lyons 5b7e31c075 Add metrics_id to editor_events (#13525)
Release Notes:

- N/A
2024-06-25 16:47:55 -04:00
Conrad Irwin 4f364d6d09 Hide old linux panics (#13221)
Remove noise from the #panics channel by excluding any linux build
before
0.139.x. We filter on the os_version and os_name because evern older
versions
of linux set app_version = 1.0.0.

Release Notes:

- N/A
2024-06-18 13:48:26 -06:00
Joseph T. Lyons f489c8b79f Allow for non-official builds to report telemetry (#13175)
Release Notes:

- N/A
2024-06-17 20:24:18 -04:00