oak-team/oak-gpui
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
12,217 Commits 1 Branch 0 Tags
fe10875285449cf01e24dff2f8ca3520a8a5be92
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Kyle Caverly 5fe8aa064f Keyboard navigation and setting persistence for project search (#2996) 
Enable keyboard shortcuts for Project Search modes, and ensure project
search settings are persisted search to search.

Release Notes:

- Added alt-cmd-s to Toggle Semantic Search Mode
- Added alt-cmd-g to Toggle Regex Search Mode
- Added alt-cmd-x to Toggle Text Search Mode
- Defaulted new project searches to using last used search mode and
settings.
 2023-09-20 12:44:30 -04:00
KCaverly 37a0c6615f remove conditional action registration for activate semantic search mode 2023-09-20 12:38:29 -04:00
KCaverly 22d095bd35 enable project search states to persist between workspace project searches 2023-09-20 12:12:01 -04:00
KCaverly 28ed406d54 add keymaps for activating text, regex and semantic modes 2023-09-20 10:04:43 -04:00
KCaverly 912e6e8091 create initial action for Semantic search mode 2023-09-20 09:48:27 -04:00
Kyle Caverly 2f44055079 Semantic index eval (#2988) 
v0 of the Semantic Index evaluate test suite

Release Notes:

- Added eval.rs as an example to the semantic-index crates
- Generates test metrics for two small projects, as a starting point to
systematically evaluate retrieval quality
 2023-09-19 19:17:06 -04:00
KCaverly 11b3bfdc99 fix warnings 2023-09-19 19:05:26 -04:00
KCaverly 25cb79e475 remove git2 dependency for repository cloning in semantic_index eval 2023-09-19 18:55:15 -04:00
KCaverly b57b5c0b33 updated git2 to use ssl 2023-09-19 16:36:51 -04:00
KCaverly d85acceeec move git2 to workspace dependency globally 2023-09-19 16:13:47 -04:00
Kyle Caverly ae019fb7d3 remove release channel flags in semantic_index (#2989) 
Remove Release channel filters from semantic_index allowing the index to
be released to stable.

Release Notes:

- Remove Release channel filters in enabled and semantic_index
initialization
 2023-09-19 12:39:26 -04:00
KCaverly 4f1a59ebf5 formatting 2023-09-19 12:27:33 -04:00
KCaverly fc8dd8433c remove release channel flags in semantic_index 2023-09-19 12:20:59 -04:00
KCaverly 183758a7c5 fix Cargo.lock for merge 2023-09-19 11:44:51 -04:00
KCaverly 25bd357426 add recall and precision to semantic index 2023-09-18 18:25:02 -04:00
KCaverly 566bb9f71b add map to evaluation suite for semantic_index 2023-09-18 09:57:52 -04:00
Kyle Caverly b9c1f3d558 enable includes filtering for open and modified buffers (#2979) 
enable include file filtering for semantic searching in open and
modified buffers

Release Notes:

- Added include based filtering for modified buffers
 2023-09-15 15:23:24 -04:00
KCaverly 1433160a08 enable include based filtering for search inside open and modified buffers 2023-09-15 15:16:20 -04:00
KCaverly 04bd107ada add ndcg@k to evaluate metrics 2023-09-15 10:36:21 -04:00
KCaverly 3a661c5977 catchup with main 2023-09-15 09:31:33 -04:00
KCaverly 796bdd3da7 update searching in modified buffers to accomodate for excluded paths 2023-09-14 19:42:06 -04:00
Kyle Caverly 9b901df256 small fix to rate status update (#2967) 
Small fix to update code for rate limiting status.

Release Notes (Preview only)
- Fixed update to only stop updating status, when the rate limit is
reset to None
 2023-09-14 15:43:30 -04:00
KCaverly 8ff3e37044 small fix to rate status update 2023-09-14 15:42:21 -04:00
KCaverly c19c8899fe add initial search inside modified buffers 2023-09-14 14:58:34 -04:00
KCaverly 137dda3ee6 wip eval framework for semantic index 2023-09-14 09:30:19 -04:00
KCaverly 0c1b2e5aa6 cleaned up warnings 2023-09-13 20:04:53 -04:00
KCaverly eff44f9aa4 semantic index eval, indexing appropriately 2023-09-13 20:02:15 -04:00
KCaverly 6f29582fb0 progress on eval 2023-09-13 10:32:36 -04:00
KCaverly d4fbe99052 add eval for gpt-engineer 2023-09-12 21:27:35 -04:00
KCaverly 0d14bbbf5b add eval values for tree-sitter 2023-09-12 20:36:06 -04:00
KCaverly 66c967da88 start work on eval script for semantic_index 2023-09-12 16:25:31 -04:00
Kyle Caverly d4da5540ac Rate limiting status (#2955) 
Add a rate limit remaining status to Project Search Semantic Search
minor text

Release Notes (Preview-Only):

- Added tracking functionality within EmbeddingProvider, to track rate
limit expiry
- Update minor text within Project Search to show countdown remaining
before rate limit expiry
 2023-09-11 10:40:11 -04:00
KCaverly 74ab41000b Merge branch 'main' of github.com:zed-industries/zed into rate_limiting_status 2023-09-11 10:26:22 -04:00
KCaverly e678c7d9ee swap 
SystemTime for Instant throughout rate_limit_expiry tracking
 2023-09-11 10:26:14 -04:00
KCaverlyandAntonio 7df21f86dd move cx notify observe for rate_limit_expiry into ProjectState in the semantic index 
Co-authored-by: Antonio <antonio@zed.dev>
 2023-09-11 10:11:40 -04:00
KCaverly 37915ec4f2 updated notify to accomodate for updated countdown 2023-09-08 16:53:16 -04:00
KCaverly bf43f93197 updated semantic_index reset status to leverage target reset system time as opposed to duration 2023-09-08 15:04:50 -04:00
KCaverly a5ee8fc805 initial outline for rate limiting status updates 2023-09-08 12:35:15 -04:00
Kyle Caverly 560d6b1644 update semantic search to show no results if search query is blank (#2942) 
Update semantic search to show no search results if search query is
blank
 2023-09-07 14:58:00 -04:00
KCaverly cf5d1d91a4 update semantic search to go to no results if search query is blank 2023-09-07 14:43:41 -04:00
Kyle Caverly bbeb82f884 Token count fix (#2935) 
Fix token count for OpenAIEmbeddings

Release Notes (Preview Only)

- update token count calculation for truncated OpenAIEmbeddings
- increased request timeout for OpenAI
 2023-09-06 15:15:02 -04:00
KCaverly 265d02a583 update request timeout for open ai embeddings 2023-09-06 15:09:46 -04:00
KCaverly 17237f748c update token_count for OpenAIEmbeddings to accomodate for truncation 2023-09-06 15:09:15 -04:00
Kyle Caverly 49af2874bb Eager background indexing (#2928) 
This PR ships a series of optimizations for the semantic search engine.
Mostly focused on removing invalid states, optimizing requests to
OpenAI, and reducing token usage.

Release Notes (Preview-Only):

- Added eager incremental indexing in the background on a debounce.
- Added a local embeddings cache for reducing redundant calls to OpenAI.
- Moved to an Embeddings Queue model which ensures optimal batch sizes
at the token level, and atomic file & document writes.
- Adjusted OpenAI Embedding API requests to use provided backoff delays
during Rate Limiting.
- Removed flush races between parsing files step and embedding queue
steps.
- Moved truncation to parsing step reducing the probability that OpenAI
encounters bad data.
 2023-09-05 13:15:54 -04:00
KCaverly 8dbc0fe033 update pragma settings for improved database performance 2023-09-01 17:07:20 -04:00
KCaverlyandMax 54235f4fb1 updated embeddings background delay to 5 minutes 
Co-authored-by: Max <max@zed.dev>
 2023-09-01 13:04:09 -04:00
KCaverlyandMax e86964eb5d optimize insert file in vector database 
Co-authored-by: Max <max@zed.dev>
 2023-09-01 13:01:37 -04:00
KCaverlyandAntonio 524533cfb2 flush embeddings queue when no files are parsed for 250 milliseconds 
Co-authored-by: Antonio <antonio@zed.dev>
 2023-09-01 11:24:08 -04:00
KCaverlyandAntonio c4db914f0a move embeddings queue to use single hashmap for all changed paths 
Co-authored-by: Antonio <me@as-cii.com>
 2023-09-01 08:59:25 -04:00
KCaverly afa59abbcd WIP: work towards wiring up a embeddings_for_digest hashmap that is stored for all indexed files 2023-08-31 16:42:39 -04:00
KCaverly 50cfb067e7 fill embeddings with database values and skip during embeddings queue 2023-08-31 13:19:17 -04:00
KCaverly 7d4d6c871b fix bug for truncation ensuring no valid inputs are sent to openai 2023-08-30 17:42:16 -04:00
KCaverlyandMax 5abad58b0d moved semantic index to use embeddings queue to batch and managed for atomic database writes 
Co-authored-by: Max <max@zed.dev>
 2023-08-30 16:58:45 -04:00
KCaverlyandMax 76ce52df4e move queuing to embedding_queue functionality and update embedding provider to include trait items for max tokens per batch" 
Co-authored-by: Max <max@zed.dev>
 2023-08-30 16:01:28 -04:00
KCaverly 9781047156 move truncation to parsing step leveraging the EmbeddingProvider trait 2023-08-30 12:13:26 -04:00
KCaverly 76caea80f7 add should_truncate to embedding providers 2023-08-30 11:58:45 -04:00
KCaverly e377ada1a9 added token count to documents during parsing 2023-08-30 11:05:46 -04:00
KCaverlyandMax a7e6a65deb reindex files in the background after they have not been edited for 10 minutes 
Co-authored-by: Max <max@zed.dev>
 2023-08-29 17:14:44 -04:00
KCaverly 4f8b95cf0d add proper handling for open ai rate limit delays 2023-08-29 15:44:51 -04:00
Kyle Caverly bc7e9088fe Eager Semantic Indexing Queue (#2886) 
Optimization to the Semantic Indexing Engine.

We've transitioned from a framework in which the entire project tree is
walked at each index command, to an eager queuing method, in which an
initial queue of outstanding indexing work is initialized upon workspace
creation, and then subscriptions are leveraged for file change events to
continually keep an updated view on outstanding work.

This optimization contributes towards quicker user feedback, when
initializing or using Semantic Search functionality. It also opens the
doors towards better transparency across the system on outstanding
indexing work.

Release Notes:

- Refactored index operation queue to an eager queuing framework.
- Moved semantic search initialization to workspace creation.
- Adjusted rate limiting strategy on api delays to reduce time spent
waiting for rate limits.
 2023-08-25 11:03:05 +02:00
KCaverly ee97bc54cf cleaned up warnings 2023-08-25 10:38:01 +02:00
KCaverly 131950f670 add handling for Added file events to semantic index 2023-08-24 18:40:08 +02:00
KCaverlyandPiotr a892a51ec3 update initialize project call to accomodate for test scenarios 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-24 13:46:43 +02:00
KCaverlyandPiotr 0b204bfdc8 reindex semantic index when search project pane is reactivated in semantic mode 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-24 13:40:04 +02:00
KCaverlyandPiotr a1519e4c38 move semantic search project intialization to a subscribe event for workspace created 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-24 13:14:19 +02:00
KCaverlyandPiotr e8e7b294d8 add delete files operation for remaining files in database not included in current worktree 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-24 12:49:20 +02:00
KCaverlyandPiotr afe0e74868 remove worktree_file_mtimes in state as it is no longer used 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-24 12:42:41 +02:00
KCaverlyandPiotr aa07872a24 accomodate for duplicate entries in indexing queue 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-24 12:36:43 +02:00
KCaverly 3f9f742530 update rate limiting embeddings strategy to delay less 2023-08-24 11:45:52 +02:00
KCaverly b72c4c576b catchup with main 2023-08-23 22:31:39 +02:00
KCaverly e42b9e910e fix async calls on project updated entries to ensure that all files are updating appropriately 2023-08-23 22:28:30 +02:00
KCaverly 09fd99b1e3 moved semantic_index project intialization to queue and channel method 2023-08-23 15:09:15 +02:00
KCaverly 328b7e523c reorganized to stop the race 2023-08-22 15:01:21 +02:00
KCaverly aabdfa210f working on initialization + index breakup 2023-08-22 14:45:27 +02:00
KCaverly ced2b2aec3 reworked ProjectState to include additional context 2023-08-22 11:58:48 +02:00
KCaverly 3d89cd10a4 added sha1 encoding for each document 2023-08-21 16:35:57 +02:00
Kyle Caverly bbe6d3b261 Semantic index rough edges (#2871) 
Small PR aimed at improving a few edge cases for semantic indexing large
projects.

Release Notes (Preview-only).

- Batched large files with a number of documents greater than
EMBEDDINGS_BATCH_SIZE.
- Ensured that the job handle counting mechanism is consistent with
inner file batching.
- Updated tab content names for semantic search, to match text/regex
searches.
 2023-08-21 13:43:08 +02:00
KCaverlyandPiotr def215af9f update job handle to ensure file count is consistent 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-21 12:47:43 +02:00
KCaverly 1cae4758cc manage for edge case in which file documents are larger than the allowable limit 2023-08-21 11:29:45 +02:00
Kyle Caverly 8451e7eb7e Project search design (#2834) 
TODO before merging: 
- [x] Re-run project search when options (case, word, regex) change

/cc @PixelJanitor 
Release Notes:
- Revamped project & buffer search UI.
- Added "Cycle Mode" command for search
 2023-08-18 14:38:01 +02:00
KCaverly b7dd12e53e ensured search results are cleared appropriately while cycling modes 2023-08-17 11:11:09 +01:00
KCaverlyandPiotr c99b530968 remove kill search, and restart search immediately upon cycle mode 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-16 16:13:21 +01:00
KCaverlyandPiotr aeda5d9842 fix semantic search panic which is created via incompatible build_search_query path 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-16 15:50:54 +01:00
KCaverly 6e3e61ec95 removed commented out code 2023-08-16 14:01:53 +01:00
KCaverlyandPiotr 2a7df106e1 adjusted icon sizes downwards 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-16 14:01:06 +01:00
KCaverlyandPiotr 6f78a1633d fix editor height in buffer search, but the dancing is back 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-16 13:51:41 +01:00
KCaverlyandPiotr a59535efa1 remove redundant and unneeded styling 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-16 12:55:32 +01:00
KCaverly d727ba18f2 Merge branch 'main' of github.com:zed-industries/zed into project_search_design 2023-08-16 12:43:04 +01:00
KCaverlyandPiotr 9bf227b884 remove regex keymap, and made spacing consistent between search objects 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-16 12:17:59 +01:00
KCaverlyandPiotr 11ecb7b604 reorganize search bar, enable filters and disable select all during invalid states 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-16 11:35:09 +01:00
KCaverly 32bec2e401 left align buffer search with new structure 2023-08-15 23:18:03 +01:00
KCaverlyandmaxbrunsfeld 1e8a7c7caa refactor buffer_search to reduce redundancy and simplying height management 
Co-authored-by: maxbrunsfeld <max@zed.dev>
 2023-08-15 20:04:14 +01:00
KCaverly 695e6d2f2e fix spacing on editor magnifying glass icon 2023-08-15 15:05:39 +01:00
KCaverly c14a99d8fa updated project_search text 2023-08-03 17:02:46 -04:00
KCaverlyandPiotr 8dd330a160 add semantic search prompt for the indexing permission on first search 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-03 12:52:20 -04:00
KCaverly 358e4e5ccf update minor text when actively searching 2023-08-03 10:22:31 -04:00
KCaverly b4f6d6eadc update search text for no results 2023-08-02 17:14:15 -04:00
KCaverly a125e318fe catch up with main 2023-08-02 16:48:11 -04:00
KCaverly 71bbd5f2f6 update keymaps for cycle search mode and toggle filters 2023-08-02 15:08:54 -04:00
KCaverly acf78f5fb4 add kill_search function to stop searching on mode change 2023-08-02 14:18:28 -04:00