oak-team/oak-gpui
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
11,058 Commits 1 Branch 0 Tags
80c779b95e610ade8258cf5dcf28fa4f8ac7cd2c
Commit Graph
99 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Kyle Caverly ce258dfeb9 Expanded Semantic language support (#2818) 
Expand Language Support within Semantic Search

Release Notes (Preview-only)

- Added semantic search support for lua, ruby, php, svelte, erb, heex,
html and markdown.
- Full details are included here:
https://linear.app/zed-industries/issue/Z-2611/allow-semantic-search-for-all-supported-languages
 2023-08-01 11:51:51 -04:00
KCaverly 300c693d55 catchup with main 2023-08-01 10:40:38 -04:00
KCaverly e221f23018 add support for markdown files to semantic search 2023-08-01 10:30:34 -04:00
KCaverly 9a50b43eaa add templating languages html, erb, heex, svelte as entire parseable file types 2023-07-31 21:03:02 -04:00
KCaverly 599f674827 add php support for semantic search 2023-07-31 16:36:09 -04:00
KCaverly 89edb3d1b5 fix templating bug for parseable entire files 2023-07-31 11:41:18 -04:00
KCaverly ca4e21881e add ruby support for semantic search 2023-07-31 10:54:30 -04:00
KCaverly a5dd8dd0a9 add lua embedding query for semantic search 2023-07-31 10:02:28 -04:00
Kyle Caverly ee66f99ce6 Parallel vector db (#2792) 
Parallelize Vector Database calls for project semantic search.

Release Notes: (Preview-only)

- Parallelize Vector database calls for project semantic search. Cuts
query time by 2/3rds.
- Removed default keymap for old semantic search modal.
 2023-07-26 17:17:59 -04:00
KCaverly fbede4a5a3 removed old code 2023-07-26 17:11:30 -04:00
KCaverly e2b38f7a31 remove unused imports 2023-07-26 17:01:44 -04:00
KCaverly 06167889c3 remove default keymap for semantic search modal 2023-07-26 16:59:19 -04:00
KCaverly 5c48729c7c managed for small batch size edge case in semantic search 2023-07-26 16:56:41 -04:00
KCaverly 89bbcdfa4f remove debug logging for project_search semantic search 2023-07-26 16:51:25 -04:00
KCaverly 98fde36834 batch search queries in the vector database 2023-07-26 16:36:39 -04:00
Kyle Caverly fc1844d684 Semantic search v2 (#2789) 
Move semantic search from navigation modal, to project search option.
This PR is intended to be released in Preview only, and requires an
opt-in semantic_index option to enable. Without this opt-in setting
enable, the user should perceive no differences between previous project
search.

Release Notes: (Preview-only)

- Added Semantic Search as a opt-in feature within Project Search
- Show indexing feedback on indexing process within project search view
 2023-07-26 10:34:58 -04:00
KCaverly 0b61c93a25 ensure semantic search is not enabled on stable 2023-07-26 10:22:33 -04:00
KCaverly 394a105639 fix warnings 2023-07-26 10:03:30 -04:00
KCaverly 0ac919f6e0 catchup with main 2023-07-26 09:50:38 -04:00
KCaverly ca6f7d8a80 add worktree previously indexed functionality to vector db 2023-07-26 09:17:04 -04:00
KCaverly 75999204ad update project search to only show semantic button visible with semantic_index enabled 2023-07-25 16:26:37 -04:00
KCaverly e8210b827d move visible text to just start anchor with context lines for semantic search 2023-07-25 15:24:27 -04:00
KCaverly cdceddd2cc update semantic index tests for elixir 2023-07-25 15:20:35 -04:00
KCaverly 97c3d97792 update semantic index tests for cpp 2023-07-25 13:30:38 -04:00
KCaverly c86096a886 update semantic index tests for javascript 2023-07-25 10:38:37 -04:00
KCaverly e02d6bc0d4 add glob filtering functionality to semantic search 2023-07-20 13:46:27 -04:00
KCaverlyandmaxbrunsfeld efe973ebe2 add embedding query for json with nested arrays and strings 
Co-authored-by: maxbrunsfeld <max@zed.dev>
 2023-07-19 16:52:44 -04:00
KCaverlyandmaxbrunsfeld 9809ec3d70 update treesitter parsing to accomodate for collapsed nested functions 
Co-authored-by: maxbrunsfeld <max@zed.dev>
 2023-07-19 15:47:05 -04:00
KCaverly 0e071919a0 parellelize embedding api calls 2023-07-18 16:09:44 -04:00
KCaverly 80ef92a3e1 fix db schema update process to ensure all tables are dropped 2023-07-18 11:14:13 -04:00
KCaverly ed1b1a5ccd update logging for open ai embedding and remove redundant truncation 2023-07-18 11:00:21 -04:00
KCaverly b9fdfd60f0 catch up with main 2023-07-18 10:26:28 -04:00
KCaverly d83c4ffb07 remove debug logging for enabled settings 2023-07-17 17:09:51 -04:00
KCaverly 8b42f5b1b3 rename vector_store crate to semantic_index 2023-07-17 17:06:10 -04:00
KCaverly e630ff38c4 add embedding treesitter queries for elixir 2023-07-17 16:29:25 -04:00
KCaverly f0bf60fded add css as a embeddable file type in which the entire file is embedded individually 2023-07-17 14:53:57 -04:00
KCaverly 1362c5a3d9 add embedding treesitter query for cpp 2023-07-17 14:43:29 -04:00
KCaverly cf0dd09b5c update vector_store to accomodate for full file parsing for JSON, TOML and YAML files 2023-07-17 10:04:32 -04:00
KCaverly 4bece54655 update jsx family of languages for preceeding comments and nested exports 2023-07-17 09:22:37 -04:00
KCaverly 2dae42b1ba update embedding query for tsx to accomodate for leading comments 2023-07-14 14:25:08 -04:00
KCaverly d4971e9ead update typescript parsing to manage for leading tsdoc comments 2023-07-14 13:47:10 -04:00
KCaverly 3a625d15d3 update c embedding query for preceding comments 2023-07-14 11:33:49 -04:00
KCaverlyandmaxbrunsfeld b38e3b804c remove reindexing subscription, and add status methods for vector store 
Co-authored-by: maxbrunsfeld <max@zed.dev>
 2023-07-13 18:14:44 -04:00
KCaverly d8fd0be598 update vector store to remove dummy embeddings 2023-07-13 17:01:56 -04:00
KCaverlyandmaxbrunsfeld 623cb9833c add tests for rust context parsing, and update rust embedding query 
Co-authored-by: maxbrunsfeld <max@zed.dev>
 2023-07-13 16:58:42 -04:00
KCaverlyandmaxbrunsfeld 0a0e40fb24 refactored code context retrieval and standardized database migration 
Co-authored-by: maxbrunsfeld <max@zed.dev>
 2023-07-13 16:34:32 -04:00
KCaverlyAlexmaxbrunsfeld
 		5eab628580 Added go parsing for semantic search, and added preceeding comments on go and rust. 
Co-authored-by: Alex <alexviscreanu@gmail.com>
Co-authored-by: maxbrunsfeld <max@zed.dev>
 2023-07-13 14:33:31 -04:00
KCaverly a56d454a07 added semantic search support for c 2023-07-13 10:10:24 -04:00
Kyle Caverly 37568ccbf0 Vector store (#2658) 
This PR includes a new crate, aimed at maintaining a consistent semantic
embedding database, for any project opened with Zed. At a high level,
for each file in a project, we parse the file with treesitter, embed the
symbol "document" objects with OpenAI, and maintain a consistent
database of these embeddings and offset locations in a sqlite database.
Once stored, we have built a simple modal interface for querying on
these symbols embeddings using natural language, offering the
opportunity to navigate to the selected symbol.

This initial PR is intended to provide this functionality only in preview,
as we explore, evaluate and iterate on the vector store.

- Full task details are provided in the [Semantic Search Linear
Project](https://linear.app/zed-industries/project/semantic-search-7c787d198ebe/Z)
 2023-07-12 13:26:17 -04:00
KCaverly c141519dba merged with main 2023-07-12 13:15:23 -04:00
KCaverly 33e2b52a01 added test registration for project settings 2023-07-11 20:12:43 -04:00
KCaverly 297fa029e3 Merge branch 'main' of github.com:zed-industries/zed into vector_store 2023-07-11 20:05:29 -04:00
KCaverly b68cd58a3b updated vector store settings to remove batch embeddings size 2023-07-11 19:54:03 -04:00
KCaverlyandmaxbrunsfeld af7b2f17ae added initial keymap for toggle semantic search 
Co-authored-by: maxbrunsfeld <max@zed.dev>
 2023-07-11 17:13:58 -04:00
KCaverly 2ca4b3f4cc cleaned up warnings and added javascript 2023-07-11 16:41:08 -04:00
KCaverly debe6f107e updated embedding queries for tsx and typescript 2023-07-11 16:22:40 -04:00
KCaverly 02f523094b expanded embeddable context to accomodate for struct context and file paths 2023-07-11 15:58:33 -04:00
KCaverly dd0dbdc5bd brought up to speed with main 2023-07-11 14:50:48 -04:00
KCaverly 1649cf81de added versioning to files table 2023-07-11 14:42:03 -04:00
KCaverly 298c2213a0 added opt-in default settings for vector store 2023-07-11 12:03:56 -04:00
KCaverly f5fec55930 updated vector_store to handle for removed files 2023-07-11 10:03:53 -04:00
KCaverly dce72a1ce7 updated tests to accomodate for new dot location 2023-07-10 18:19:29 -04:00
KCaverlyandmaxbrunsfeld 307d8d9c8d Reduced redundant database connections on each worktree change. 
Co-authored-by: maxbrunsfeld <max@zed.dev>
 2023-07-10 17:50:19 -04:00
KCaverlyandmaxbrunsfeld 82079dd422 Updated batching to accomodate for full flushes, and cleaned up reindexing. 
Co-authored-by: maxbrunsfeld <max@zed.dev>
 2023-07-10 16:33:14 -04:00
KCaverly 0189742497 pulled treesitter parsing to own file for ease of testing and management 2023-07-10 10:06:07 -04:00
KCaverly 3f5667b101 merged main 2023-07-07 14:24:29 -04:00
KCaverly c03dda1a0c fixed bug on absolute vs relative path 2023-07-06 17:15:41 -04:00
KCaverly 6f1e988cb9 updated embedding treesitter query for python 2023-07-06 16:36:28 -04:00
KCaverly 7d634f66e2 updated vector_store to include extra context for semantic search modal 2023-07-06 16:33:54 -04:00
KCaverly e57f6f21fe reindexing update to appropriately accomodate for buffer delay and persistent pending files list 2023-07-06 15:26:43 -04:00
KCaverly a86b6c42c7 corrected batching order and managed for open ai embedding errors 2023-07-06 11:11:39 -04:00
KCaverly afccf608f4 updated both embed and parsing tasks to be multi-threaded. 2023-07-05 12:39:08 -04:00
KCaverly eff0ee3b60 enabled batching for embedding calls 2023-07-05 10:02:42 -04:00
KCaverly b6520a8f1d updated vector_store to reindex on save after timed delay 2023-07-04 14:42:12 -04:00
KCaverly e45d3a0a63 WIP: initial reindexing logic worked out 2023-07-04 11:46:09 -04:00
KCaverlyandmaxbrunsfeld 18a5a47f8a moved semantic search model to dev and preview only. 
moved db update tasks to long lived persistent task.

Co-authored-by: maxbrunsfeld <max@zed.dev>
 2023-06-30 18:41:19 -04:00
KCaverlyandmaxbrunsfeld 3408b98167 updated file compare in the semantic indexing engine, to work off of modified system times as opposed to file hashes 
Co-authored-by: maxbrunsfeld <max@zed.dev>
 2023-06-30 16:53:23 -04:00
KCaverlyandmaxbrunsfeld 36907bb4dc updated vector store indexing to only use languages with an embedding.scm treesitter query 
Co-authored-by: maxbrunsfeld <max@zed.dev>
 2023-06-30 16:14:11 -04:00
KCaverly 0db0876289 implemented file deletes on project indexing 2023-06-30 11:01:35 -04:00
KCaverly e3ab54942e removed sleep from directory scanning as fixes upstream appear to be scanning correctly 2023-06-30 10:17:31 -04:00
KCaverly 1d737e490b Merge branch 'main' of github.com:zed-industries/zed into vector_store 2023-06-30 09:58:13 -04:00
KCaverly 39137fc19f updated vector_store db to leverage EMBEDDINGS_DIR path 2023-06-29 15:18:32 -04:00
KCaverly 0a7245a583 updated semantic search modal to manage for duplicate queries 2023-06-29 13:50:49 -04:00
KCaverly a08d60fc61 added navigation on confirm to semantic search modal 2023-06-29 11:58:47 -04:00
KCaverlyandmaxbrunsfeld 85e71415fe updated embedding database calls to maintain project consistency 
Co-authored-by: maxbrunsfeld <max@zed.dev>
 2023-06-28 16:25:05 -04:00
KCaverlyandmaxbrunsfeld 400d39740c updated both indexing and search method for vector store, to maintain both zed worktree ids and db worktree ids 
Co-authored-by: maxbrunsfeld <max@zed.dev>
 2023-06-28 16:21:03 -04:00
KCaverly 3ca3de807c Merge branch 'main' of github.com:zed-industries/zed into vector_store 2023-06-28 14:42:24 -04:00
KCaverly 40ff7779bb WIP: Working modal, without navigation and search on every keystroke 2023-06-28 13:27:26 -04:00
KCaverly 9d19dea7dd updated vector_store to remove parsing module 2023-06-28 08:58:50 -04:00
KCaverlyandmaxbrunsfeld d1bdfa0be6 Added a dummy action for testing the semantic search functionality in the command palette. 
Co-authored-by: maxbrunsfeld <max@zed.dev>
 2023-06-27 15:53:07 -04:00
KCaverlyandmaxbrunsfeld 4bfe3de1f2 Working incremental index engine, with streaming similarity search! 
Co-authored-by: maxbrunsfeld <max@zed.dev>
 2023-06-27 15:31:21 -04:00
KCaverlyandmaxbrunsfeld 953e928bdb WIP: Got the streaming matrix multiplication working, and started work on file hashing. 
Co-authored-by: maxbrunsfeld <max@zed.dev>
 2023-06-26 19:01:19 -04:00
KCaverlyandmaxbrunsfeld 74b693d6b9 Updated database calls to share single connection, and simplified top_k_search sorting. 
Co-authored-by: maxbrunsfeld <max@zed.dev>
 2023-06-26 14:57:57 -04:00
KCaverly 0f232e0ce2 added file metadata retrieval from db 2023-06-26 10:35:56 -04:00
KCaverly 7937a16002 added brute force search and VectorSearch trait 2023-06-26 10:34:12 -04:00
KCaverly 65bbb7c57b added proper blob serialization for embeddings and vector search trait 2023-06-25 20:02:56 -04:00
KCaverly c071b271be removed tokio and sqlx dependency, added dummy embeddings provider to save on open ai costs when testing 2023-06-23 10:25:12 -04:00
KCaverly dd309070eb open ai indexing on open for rust files 2023-06-22 16:50:07 -04:00
KCaverly d4a4db42aa WIP: started DB creating and naive inserts 2023-06-22 13:25:33 -04:00