oak-team/oak-gpui
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
11,831 Commits 1 Branch 0 Tags
a2e91e45d9c4399ced604cc3400a0ddee3f56761
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Kyle Caverly 49af2874bb Eager background indexing (#2928) 
This PR ships a series of optimizations for the semantic search engine.
Mostly focused on removing invalid states, optimizing requests to
OpenAI, and reducing token usage.

Release Notes (Preview-Only):

- Added eager incremental indexing in the background on a debounce.
- Added a local embeddings cache for reducing redundant calls to OpenAI.
- Moved to an Embeddings Queue model which ensures optimal batch sizes
at the token level, and atomic file & document writes.
- Adjusted OpenAI Embedding API requests to use provided backoff delays
during Rate Limiting.
- Removed flush races between parsing files step and embedding queue
steps.
- Moved truncation to parsing step reducing the probability that OpenAI
encounters bad data.
 2023-09-05 13:15:54 -04:00
KCaverly 8dbc0fe033 update pragma settings for improved database performance 2023-09-01 17:07:20 -04:00
KCaverlyandMax 54235f4fb1 updated embeddings background delay to 5 minutes 
Co-authored-by: Max <max@zed.dev>
 2023-09-01 13:04:09 -04:00
KCaverlyandMax e86964eb5d optimize insert file in vector database 
Co-authored-by: Max <max@zed.dev>
 2023-09-01 13:01:37 -04:00
KCaverlyandAntonio 524533cfb2 flush embeddings queue when no files are parsed for 250 milliseconds 
Co-authored-by: Antonio <antonio@zed.dev>
 2023-09-01 11:24:08 -04:00
KCaverlyandAntonio c4db914f0a move embeddings queue to use single hashmap for all changed paths 
Co-authored-by: Antonio <me@as-cii.com>
 2023-09-01 08:59:25 -04:00
KCaverly afa59abbcd WIP: work towards wiring up a embeddings_for_digest hashmap that is stored for all indexed files 2023-08-31 16:42:39 -04:00
KCaverly 50cfb067e7 fill embeddings with database values and skip during embeddings queue 2023-08-31 13:19:17 -04:00
KCaverly 7d4d6c871b fix bug for truncation ensuring no valid inputs are sent to openai 2023-08-30 17:42:16 -04:00
KCaverlyandMax 5abad58b0d moved semantic index to use embeddings queue to batch and managed for atomic database writes 
Co-authored-by: Max <max@zed.dev>
 2023-08-30 16:58:45 -04:00
KCaverlyandMax 76ce52df4e move queuing to embedding_queue functionality and update embedding provider to include trait items for max tokens per batch" 
Co-authored-by: Max <max@zed.dev>
 2023-08-30 16:01:28 -04:00
KCaverly 9781047156 move truncation to parsing step leveraging the EmbeddingProvider trait 2023-08-30 12:13:26 -04:00
KCaverly 76caea80f7 add should_truncate to embedding providers 2023-08-30 11:58:45 -04:00
KCaverly e377ada1a9 added token count to documents during parsing 2023-08-30 11:05:46 -04:00
KCaverlyandMax a7e6a65deb reindex files in the background after they have not been edited for 10 minutes 
Co-authored-by: Max <max@zed.dev>
 2023-08-29 17:14:44 -04:00
KCaverly 4f8b95cf0d add proper handling for open ai rate limit delays 2023-08-29 15:44:51 -04:00
Kyle Caverly bc7e9088fe Eager Semantic Indexing Queue (#2886) 
Optimization to the Semantic Indexing Engine.

We've transitioned from a framework in which the entire project tree is
walked at each index command, to an eager queuing method, in which an
initial queue of outstanding indexing work is initialized upon workspace
creation, and then subscriptions are leveraged for file change events to
continually keep an updated view on outstanding work.

This optimization contributes towards quicker user feedback, when
initializing or using Semantic Search functionality. It also opens the
doors towards better transparency across the system on outstanding
indexing work.

Release Notes:

- Refactored index operation queue to an eager queuing framework.
- Moved semantic search initialization to workspace creation.
- Adjusted rate limiting strategy on api delays to reduce time spent
waiting for rate limits.
 2023-08-25 11:03:05 +02:00
KCaverly ee97bc54cf cleaned up warnings 2023-08-25 10:38:01 +02:00
KCaverly 131950f670 add handling for Added file events to semantic index 2023-08-24 18:40:08 +02:00
KCaverlyandPiotr a892a51ec3 update initialize project call to accomodate for test scenarios 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-24 13:46:43 +02:00
KCaverlyandPiotr 0b204bfdc8 reindex semantic index when search project pane is reactivated in semantic mode 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-24 13:40:04 +02:00
KCaverlyandPiotr a1519e4c38 move semantic search project intialization to a subscribe event for workspace created 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-24 13:14:19 +02:00
KCaverlyandPiotr e8e7b294d8 add delete files operation for remaining files in database not included in current worktree 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-24 12:49:20 +02:00
KCaverlyandPiotr afe0e74868 remove worktree_file_mtimes in state as it is no longer used 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-24 12:42:41 +02:00
KCaverlyandPiotr aa07872a24 accomodate for duplicate entries in indexing queue 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-24 12:36:43 +02:00
KCaverly 3f9f742530 update rate limiting embeddings strategy to delay less 2023-08-24 11:45:52 +02:00
KCaverly b72c4c576b catchup with main 2023-08-23 22:31:39 +02:00
KCaverly e42b9e910e fix async calls on project updated entries to ensure that all files are updating appropriately 2023-08-23 22:28:30 +02:00
KCaverly 09fd99b1e3 moved semantic_index project intialization to queue and channel method 2023-08-23 15:09:15 +02:00
KCaverly 328b7e523c reorganized to stop the race 2023-08-22 15:01:21 +02:00
KCaverly aabdfa210f working on initialization + index breakup 2023-08-22 14:45:27 +02:00
KCaverly ced2b2aec3 reworked ProjectState to include additional context 2023-08-22 11:58:48 +02:00
KCaverly 3d89cd10a4 added sha1 encoding for each document 2023-08-21 16:35:57 +02:00
Kyle Caverly bbe6d3b261 Semantic index rough edges (#2871) 
Small PR aimed at improving a few edge cases for semantic indexing large
projects.

Release Notes (Preview-only).

- Batched large files with a number of documents greater than
EMBEDDINGS_BATCH_SIZE.
- Ensured that the job handle counting mechanism is consistent with
inner file batching.
- Updated tab content names for semantic search, to match text/regex
searches.
 2023-08-21 13:43:08 +02:00
KCaverlyandPiotr def215af9f update job handle to ensure file count is consistent 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-21 12:47:43 +02:00
KCaverly 1cae4758cc manage for edge case in which file documents are larger than the allowable limit 2023-08-21 11:29:45 +02:00
Kyle Caverly 8451e7eb7e Project search design (#2834) 
TODO before merging: 
- [x] Re-run project search when options (case, word, regex) change

/cc @PixelJanitor 
Release Notes:
- Revamped project & buffer search UI.
- Added "Cycle Mode" command for search
 2023-08-18 14:38:01 +02:00
KCaverly b7dd12e53e ensured search results are cleared appropriately while cycling modes 2023-08-17 11:11:09 +01:00
KCaverlyandPiotr c99b530968 remove kill search, and restart search immediately upon cycle mode 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-16 16:13:21 +01:00
KCaverlyandPiotr aeda5d9842 fix semantic search panic which is created via incompatible build_search_query path 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-16 15:50:54 +01:00
KCaverly 6e3e61ec95 removed commented out code 2023-08-16 14:01:53 +01:00
KCaverlyandPiotr 2a7df106e1 adjusted icon sizes downwards 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-16 14:01:06 +01:00
KCaverlyandPiotr 6f78a1633d fix editor height in buffer search, but the dancing is back 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-16 13:51:41 +01:00
KCaverlyandPiotr a59535efa1 remove redundant and unneeded styling 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-16 12:55:32 +01:00
KCaverly d727ba18f2 Merge branch 'main' of github.com:zed-industries/zed into project_search_design 2023-08-16 12:43:04 +01:00
KCaverlyandPiotr 9bf227b884 remove regex keymap, and made spacing consistent between search objects 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-16 12:17:59 +01:00
KCaverlyandPiotr 11ecb7b604 reorganize search bar, enable filters and disable select all during invalid states 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-16 11:35:09 +01:00
KCaverly 32bec2e401 left align buffer search with new structure 2023-08-15 23:18:03 +01:00
KCaverlyandmaxbrunsfeld 1e8a7c7caa refactor buffer_search to reduce redundancy and simplying height management 
Co-authored-by: maxbrunsfeld <max@zed.dev>
 2023-08-15 20:04:14 +01:00
KCaverly 695e6d2f2e fix spacing on editor magnifying glass icon 2023-08-15 15:05:39 +01:00
KCaverly c14a99d8fa updated project_search text 2023-08-03 17:02:46 -04:00
KCaverlyandPiotr 8dd330a160 add semantic search prompt for the indexing permission on first search 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-03 12:52:20 -04:00
KCaverly 358e4e5ccf update minor text when actively searching 2023-08-03 10:22:31 -04:00
KCaverly b4f6d6eadc update search text for no results 2023-08-02 17:14:15 -04:00
KCaverly a125e318fe catch up with main 2023-08-02 16:48:11 -04:00
KCaverly 71bbd5f2f6 update keymaps for cycle search mode and toggle filters 2023-08-02 15:08:54 -04:00
KCaverly acf78f5fb4 add kill_search function to stop searching on mode change 2023-08-02 14:18:28 -04:00
KCaverlyandPiotr 7b43b0d4f1 refactored search mode to ensure state is consistent 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-02 12:29:19 -04:00
KCaverlyandPiotr caaa4b1618 add theme for search mode option buttons, and adjust padding 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-02 11:23:55 -04:00
KCaverlyandPiotr cf060f0011 added major and minor display text to project search, and fixed icon padding 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-01 13:28:21 -04:00
KCaverlyandPiotr 7d5ff60ff0 added svg right margin in search bar 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-01 12:34:02 -04:00
Kyle Caverly ce258dfeb9 Expanded Semantic language support (#2818) 
Expand Language Support within Semantic Search

Release Notes (Preview-only)

- Added semantic search support for lua, ruby, php, svelte, erb, heex,
html and markdown.
- Full details are included here:
https://linear.app/zed-industries/issue/Z-2611/allow-semantic-search-for-all-supported-languages
 2023-08-01 11:51:51 -04:00
KCaverlyandPiotrek c28ba3a11a add cycle mode to project search 
Co-authored-by: Piotrek <piotr@zed.dev>
 2023-08-01 11:47:30 -04:00
KCaverly 300c693d55 catchup with main 2023-08-01 10:40:38 -04:00
KCaverly e221f23018 add support for markdown files to semantic search 2023-08-01 10:30:34 -04:00
KCaverly 9a50b43eaa add templating languages html, erb, heex, svelte as entire parseable file types 2023-07-31 21:03:02 -04:00
KCaverly 599f674827 add php support for semantic search 2023-07-31 16:36:09 -04:00
KCaverly 89edb3d1b5 fix templating bug for parseable entire files 2023-07-31 11:41:18 -04:00
KCaverly ca4e21881e add ruby support for semantic search 2023-07-31 10:54:30 -04:00
KCaverly a5dd8dd0a9 add lua embedding query for semantic search 2023-07-31 10:02:28 -04:00
Kyle Caverly ee66f99ce6 Parallel vector db (#2792) 
Parallelize Vector Database calls for project semantic search.

Release Notes: (Preview-only)

- Parallelize Vector database calls for project semantic search. Cuts
query time by 2/3rds.
- Removed default keymap for old semantic search modal.
 2023-07-26 17:17:59 -04:00
KCaverly fbede4a5a3 removed old code 2023-07-26 17:11:30 -04:00
KCaverly e2b38f7a31 remove unused imports 2023-07-26 17:01:44 -04:00
KCaverly 06167889c3 remove default keymap for semantic search modal 2023-07-26 16:59:19 -04:00
KCaverly 5c48729c7c managed for small batch size edge case in semantic search 2023-07-26 16:56:41 -04:00
KCaverly 89bbcdfa4f remove debug logging for project_search semantic search 2023-07-26 16:51:25 -04:00
KCaverly 98fde36834 batch search queries in the vector database 2023-07-26 16:36:39 -04:00
Kyle Caverly fc1844d684 Semantic search v2 (#2789) 
Move semantic search from navigation modal, to project search option.
This PR is intended to be released in Preview only, and requires an
opt-in semantic_index option to enable. Without this opt-in setting
enable, the user should perceive no differences between previous project
search.

Release Notes: (Preview-only)

- Added Semantic Search as a opt-in feature within Project Search
- Show indexing feedback on indexing process within project search view
 2023-07-26 10:34:58 -04:00
KCaverly 0b61c93a25 ensure semantic search is not enabled on stable 2023-07-26 10:22:33 -04:00
KCaverly 394a105639 fix warnings 2023-07-26 10:03:30 -04:00
KCaverly 0ac919f6e0 catchup with main 2023-07-26 09:50:38 -04:00
KCaverly ca6f7d8a80 add worktree previously indexed functionality to vector db 2023-07-26 09:17:04 -04:00
KCaverly 75999204ad update project search to only show semantic button visible with semantic_index enabled 2023-07-25 16:26:37 -04:00
KCaverly e8210b827d move visible text to just start anchor with context lines for semantic search 2023-07-25 15:24:27 -04:00
KCaverly cdceddd2cc update semantic index tests for elixir 2023-07-25 15:20:35 -04:00
KCaverly 97c3d97792 update semantic index tests for cpp 2023-07-25 13:30:38 -04:00
KCaverly c86096a886 update semantic index tests for javascript 2023-07-25 10:38:37 -04:00
KCaverly e02d6bc0d4 add glob filtering functionality to semantic search 2023-07-20 13:46:27 -04:00
KCaverlyandmaxbrunsfeld efe973ebe2 add embedding query for json with nested arrays and strings 
Co-authored-by: maxbrunsfeld <max@zed.dev>
 2023-07-19 16:52:44 -04:00
KCaverlyandmaxbrunsfeld 9809ec3d70 update treesitter parsing to accomodate for collapsed nested functions 
Co-authored-by: maxbrunsfeld <max@zed.dev>
 2023-07-19 15:47:05 -04:00
KCaverly 0e071919a0 parellelize embedding api calls 2023-07-18 16:09:44 -04:00
KCaverly 80ef92a3e1 fix db schema update process to ensure all tables are dropped 2023-07-18 11:14:13 -04:00
KCaverly ed1b1a5ccd update logging for open ai embedding and remove redundant truncation 2023-07-18 11:00:21 -04:00
KCaverly b9fdfd60f0 catch up with main 2023-07-18 10:26:28 -04:00
KCaverly d83c4ffb07 remove debug logging for enabled settings 2023-07-17 17:09:51 -04:00
KCaverly 8b42f5b1b3 rename vector_store crate to semantic_index 2023-07-17 17:06:10 -04:00
KCaverly e630ff38c4 add embedding treesitter queries for elixir 2023-07-17 16:29:25 -04:00
KCaverly f0bf60fded add css as a embeddable file type in which the entire file is embedded individually 2023-07-17 14:53:57 -04:00
KCaverly 1362c5a3d9 add embedding treesitter query for cpp 2023-07-17 14:43:29 -04:00
KCaverly cf0dd09b5c update vector_store to accomodate for full file parsing for JSON, TOML and YAML files 2023-07-17 10:04:32 -04:00