Kyle Caverly
37568ccbf0
Vector store ( #2658 )
...
This PR includes a new crate, aimed at maintaining a consistent semantic
embedding database, for any project opened with Zed. At a high level,
for each file in a project, we parse the file with treesitter, embed the
symbol "document" objects with OpenAI, and maintain a consistent
database of these embeddings and offset locations in a sqlite database.
Once stored, we have built a simple modal interface for querying on
these symbols embeddings using natural language, offering the
opportunity to navigate to the selected symbol.
This initial PR is intended to provide this functionality only in preview,
as we explore, evaluate and iterate on the vector store.
- Full task details are provided in the [Semantic Search Linear
Project](https://linear.app/zed-industries/project/semantic-search-7c787d198ebe/Z )
2023-07-12 13:26:17 -04:00
KCaverly
c141519dba
merged with main
2023-07-12 13:15:23 -04:00
KCaverly
33e2b52a01
added test registration for project settings
2023-07-11 20:12:43 -04:00
KCaverly
297fa029e3
Merge branch 'main' of github.com:zed-industries/zed into vector_store
2023-07-11 20:05:29 -04:00
KCaverly
b68cd58a3b
updated vector store settings to remove batch embeddings size
2023-07-11 19:54:03 -04:00
KCaverly and maxbrunsfeld
af7b2f17ae
added initial keymap for toggle semantic search
...
Co-authored-by: maxbrunsfeld <max@zed.dev >
2023-07-11 17:13:58 -04:00
KCaverly
2ca4b3f4cc
cleaned up warnings and added javascript
2023-07-11 16:41:08 -04:00
KCaverly
debe6f107e
updated embedding queries for tsx and typescript
2023-07-11 16:22:40 -04:00
KCaverly
02f523094b
expanded embeddable context to accomodate for struct context and file paths
2023-07-11 15:58:33 -04:00
KCaverly
dd0dbdc5bd
brought up to speed with main
2023-07-11 14:50:48 -04:00
KCaverly
1649cf81de
added versioning to files table
2023-07-11 14:42:03 -04:00
KCaverly
298c2213a0
added opt-in default settings for vector store
2023-07-11 12:03:56 -04:00
KCaverly
f5fec55930
updated vector_store to handle for removed files
2023-07-11 10:03:53 -04:00
KCaverly
dce72a1ce7
updated tests to accomodate for new dot location
2023-07-10 18:19:29 -04:00
KCaverly and maxbrunsfeld
307d8d9c8d
Reduced redundant database connections on each worktree change.
...
Co-authored-by: maxbrunsfeld <max@zed.dev >
2023-07-10 17:50:19 -04:00
KCaverly and maxbrunsfeld
82079dd422
Updated batching to accomodate for full flushes, and cleaned up reindexing.
...
Co-authored-by: maxbrunsfeld <max@zed.dev >
2023-07-10 16:33:14 -04:00
KCaverly
0189742497
pulled treesitter parsing to own file for ease of testing and management
2023-07-10 10:06:07 -04:00
KCaverly
3f5667b101
merged main
2023-07-07 14:24:29 -04:00
KCaverly
c03dda1a0c
fixed bug on absolute vs relative path
2023-07-06 17:15:41 -04:00
KCaverly
6f1e988cb9
updated embedding treesitter query for python
2023-07-06 16:36:28 -04:00
KCaverly
7d634f66e2
updated vector_store to include extra context for semantic search modal
2023-07-06 16:33:54 -04:00
KCaverly
e57f6f21fe
reindexing update to appropriately accomodate for buffer delay and persistent pending files list
2023-07-06 15:26:43 -04:00
KCaverly
a86b6c42c7
corrected batching order and managed for open ai embedding errors
2023-07-06 11:11:39 -04:00
KCaverly
afccf608f4
updated both embed and parsing tasks to be multi-threaded.
2023-07-05 12:39:08 -04:00
KCaverly
eff0ee3b60
enabled batching for embedding calls
2023-07-05 10:02:42 -04:00
KCaverly
b6520a8f1d
updated vector_store to reindex on save after timed delay
2023-07-04 14:42:12 -04:00
KCaverly
e45d3a0a63
WIP: initial reindexing logic worked out
2023-07-04 11:46:09 -04:00
KCaverly and maxbrunsfeld
18a5a47f8a
moved semantic search model to dev and preview only.
...
moved db update tasks to long lived persistent task.
Co-authored-by: maxbrunsfeld <max@zed.dev >
2023-06-30 18:41:19 -04:00
KCaverly and maxbrunsfeld
3408b98167
updated file compare in the semantic indexing engine, to work off of modified system times as opposed to file hashes
...
Co-authored-by: maxbrunsfeld <max@zed.dev >
2023-06-30 16:53:23 -04:00
KCaverly and maxbrunsfeld
36907bb4dc
updated vector store indexing to only use languages with an embedding.scm treesitter query
...
Co-authored-by: maxbrunsfeld <max@zed.dev >
2023-06-30 16:14:11 -04:00
KCaverly
0db0876289
implemented file deletes on project indexing
2023-06-30 11:01:35 -04:00
KCaverly
e3ab54942e
removed sleep from directory scanning as fixes upstream appear to be scanning correctly
2023-06-30 10:17:31 -04:00
KCaverly
1d737e490b
Merge branch 'main' of github.com:zed-industries/zed into vector_store
2023-06-30 09:58:13 -04:00
KCaverly
39137fc19f
updated vector_store db to leverage EMBEDDINGS_DIR path
2023-06-29 15:18:32 -04:00
KCaverly
0a7245a583
updated semantic search modal to manage for duplicate queries
2023-06-29 13:50:49 -04:00
KCaverly
a08d60fc61
added navigation on confirm to semantic search modal
2023-06-29 11:58:47 -04:00
KCaverly and maxbrunsfeld
85e71415fe
updated embedding database calls to maintain project consistency
...
Co-authored-by: maxbrunsfeld <max@zed.dev >
2023-06-28 16:25:05 -04:00
KCaverly and maxbrunsfeld
400d39740c
updated both indexing and search method for vector store, to maintain both zed worktree ids and db worktree ids
...
Co-authored-by: maxbrunsfeld <max@zed.dev >
2023-06-28 16:21:03 -04:00
KCaverly
3ca3de807c
Merge branch 'main' of github.com:zed-industries/zed into vector_store
2023-06-28 14:42:24 -04:00
KCaverly
40ff7779bb
WIP: Working modal, without navigation and search on every keystroke
2023-06-28 13:27:26 -04:00
KCaverly
9d19dea7dd
updated vector_store to remove parsing module
2023-06-28 08:58:50 -04:00
KCaverly and maxbrunsfeld
d1bdfa0be6
Added a dummy action for testing the semantic search functionality in the command palette.
...
Co-authored-by: maxbrunsfeld <max@zed.dev >
2023-06-27 15:53:07 -04:00
KCaverly and maxbrunsfeld
4bfe3de1f2
Working incremental index engine, with streaming similarity search!
...
Co-authored-by: maxbrunsfeld <max@zed.dev >
2023-06-27 15:31:21 -04:00
KCaverly and maxbrunsfeld
953e928bdb
WIP: Got the streaming matrix multiplication working, and started work on file hashing.
...
Co-authored-by: maxbrunsfeld <max@zed.dev >
2023-06-26 19:01:19 -04:00
KCaverly and maxbrunsfeld
74b693d6b9
Updated database calls to share single connection, and simplified top_k_search sorting.
...
Co-authored-by: maxbrunsfeld <max@zed.dev >
2023-06-26 14:57:57 -04:00
KCaverly
0f232e0ce2
added file metadata retrieval from db
2023-06-26 10:35:56 -04:00
KCaverly
7937a16002
added brute force search and VectorSearch trait
2023-06-26 10:34:12 -04:00
KCaverly
65bbb7c57b
added proper blob serialization for embeddings and vector search trait
2023-06-25 20:02:56 -04:00
KCaverly
c071b271be
removed tokio and sqlx dependency, added dummy embeddings provider to save on open ai costs when testing
2023-06-23 10:25:12 -04:00
KCaverly
dd309070eb
open ai indexing on open for rust files
2023-06-22 16:50:07 -04:00
KCaverly
d4a4db42aa
WIP: started DB creating and naive inserts
2023-06-22 13:25:33 -04:00
Kyle Caverly
