oak-team/oak-gpui
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
12,765 Commits 1 Branch 0 Tags
95ef61bc45efbcbfed13a040ec8da75d5c6f398e
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Kyle Caverly cf5d89d13c Leverage embeddings query to collapse syntax nodes if not selected (#3067) 
Reverts zed-industries/zed#3049
 2023-10-03 12:02:47 +03:00
KCaverly 9f160537ef move collapsed only matches outside item parent in embedding.scm 2023-10-03 11:56:45 +03:00
Kyle Caverly dbaaf4216d add scheme for full parseable files in semantic index (#3069) 
add scheme as a parseable file type in semantic index.
Each file will operate as a single embedding, in which no real scheme
syntax or tree-sitter level data is stored.

Release Notes:

- Added scheme to Semantic Index
 2023-09-29 14:42:15 -04:00
KCaverly 3c12e711a4 add scheme for full parseable files in semantic index 2023-09-29 14:35:02 -04:00
Kyle Caverly 2c0547079a Revert "leverage file outline and selection as opposed to entire file" (#3049) 
Reverts zed-industries/zed#3040
 2023-09-27 12:21:11 -04:00
Kyle Caverly 4242b45646 Revert "removed stale dbg in assistant from main" (#3048) 
Reverts zed-industries/zed#3046
 2023-09-27 12:19:54 -04:00
Kyle Caverly edf29aa67d implement new search strategy (#3029) 
Augment current search strategy in semantic search, reducing search
times by ~60%

Release Notes:

- Implemented minimum batch sizes for concurrent database reads.
- Batch embedding matrix multiplication.
- Calculate matmul with ndarray
 2023-09-27 10:37:48 -04:00
KCaverly 0e6fd645fd leverage embeddings len returned in construction matrix multiplication 2023-09-27 10:33:04 -04:00
KCaverly 3682751455 Merge branch 'main' of github.com:zed-industries/zed into faster_semantic_search 2023-09-27 09:43:39 -04:00
KCaverly abefa2738b removed blas and increase batch size for vector search 2023-09-27 09:43:23 -04:00
Kyle Caverly 4ccd69350b removed stale dbg in assistant from main (#3046) 
remove small dbg! statement in main
 2023-09-27 09:13:41 -04:00
KCaverly 0d6880adb3 removed stale dbg in assistant from main 2023-09-27 09:13:00 -04:00
Kyle Caverly 2f368de397 leverage file outline and selection as opposed to entire file (#3040) 
Transition generate prompt for inline assist to leverage outline as
opposed to full file.
This enables, us to leverage the inline assist for large files.

Release Notes:

- Change inline assist to use tree-sitter based outlines for code
generation instead of full files
 2023-09-27 09:10:18 -04:00
KCaverly 650a160f04 update test outline for prompt tests for new cursor span 2023-09-27 09:06:53 -04:00
KCaverly 90f17d4a28 updated codegen match to leverage unused values 2023-09-26 17:11:20 -04:00
KCaverly e8dd412ac1 update inline generate prompt to leverage more explicit <|START| and |END|> spans 2023-09-26 17:10:31 -04:00
KCaverlyandAntonio 54c63063e4 changed inline assist generate prompt to leverage outline as opposed to entire prior file 
Co-Authored-by: Antonio <antonio@zed.dev>
 2023-09-26 16:23:48 -04:00
KCaverly e75f56a0f2 move to system blas 2023-09-26 12:39:22 -04:00
KCaverly 330a71d28b fixed bug limiting number of results returned 2023-09-26 10:29:55 -04:00
KCaverly ea278b5b12 ensure desc sort and cleanup unused imports 2023-09-26 09:53:49 -04:00
KCaverly 86ec0b1d9f implement new search strategy 2023-09-25 13:44:19 -04:00
Kyle Caverly e84339ef4a reorganize AI crates to structure future development (#3015) 
Reorganized assistant/semantic_index crates and introduced AI crate to
include shared functionality.

Release Notes:

- Moved most of the Assistant functionality from ai crate to assistant
crate
- Moved interaction with embedding providers from semantic_index to ai
crate
 2023-09-22 09:54:46 -04:00
KCaverly fbd6b5b434 cargo fmt 2023-09-22 09:46:06 -04:00
KCaverly dc49dec4f0 catchup with main 2023-09-22 09:43:39 -04:00
KCaverly 68c37ca2a4 move embedding provider to ai crate 2023-09-22 09:33:59 -04:00
KCaverly 48e151495f introduce ai crate with completion providers 2023-09-21 22:44:56 -04:00
KCaverly 5f6334696a rename ai crate to assistant crate 2023-09-21 21:54:59 -04:00
Kyle Caverly 3c2b05be90 add semantic index status, for non authenticated users (#3005) 
Update project search semantic ui to accommodate for users who have not
set the OPENAI_API_KEY in their environment variables.

Release Notes:

- Expand Semantic Index status to include status for non authenticated
users
- Update Search UI to illustrate this status.
 2023-09-21 14:18:58 -04:00
KCaverly 7b63369df2 move api authentication to embedding provider 2023-09-21 14:00:00 -04:00
KCaverly 997f362cc2 add semantic index status, for non authenticated users 2023-09-21 13:40:01 -04:00
Kyle Caverly 5fe8aa064f Keyboard navigation and setting persistence for project search (#2996) 
Enable keyboard shortcuts for Project Search modes, and ensure project
search settings are persisted search to search.

Release Notes:

- Added alt-cmd-s to Toggle Semantic Search Mode
- Added alt-cmd-g to Toggle Regex Search Mode
- Added alt-cmd-x to Toggle Text Search Mode
- Defaulted new project searches to using last used search mode and
settings.
 2023-09-20 12:44:30 -04:00
KCaverly 37a0c6615f remove conditional action registration for activate semantic search mode 2023-09-20 12:38:29 -04:00
KCaverly 22d095bd35 enable project search states to persist between workspace project searches 2023-09-20 12:12:01 -04:00
KCaverly 28ed406d54 add keymaps for activating text, regex and semantic modes 2023-09-20 10:04:43 -04:00
KCaverly 912e6e8091 create initial action for Semantic search mode 2023-09-20 09:48:27 -04:00
Kyle Caverly 2f44055079 Semantic index eval (#2988) 
v0 of the Semantic Index evaluate test suite

Release Notes:

- Added eval.rs as an example to the semantic-index crates
- Generates test metrics for two small projects, as a starting point to
systematically evaluate retrieval quality
 2023-09-19 19:17:06 -04:00
KCaverly 11b3bfdc99 fix warnings 2023-09-19 19:05:26 -04:00
KCaverly 25cb79e475 remove git2 dependency for repository cloning in semantic_index eval 2023-09-19 18:55:15 -04:00
KCaverly b57b5c0b33 updated git2 to use ssl 2023-09-19 16:36:51 -04:00
KCaverly d85acceeec move git2 to workspace dependency globally 2023-09-19 16:13:47 -04:00
Kyle Caverly ae019fb7d3 remove release channel flags in semantic_index (#2989) 
Remove Release channel filters from semantic_index allowing the index to
be released to stable.

Release Notes:

- Remove Release channel filters in enabled and semantic_index
initialization
 2023-09-19 12:39:26 -04:00
KCaverly 4f1a59ebf5 formatting 2023-09-19 12:27:33 -04:00
KCaverly fc8dd8433c remove release channel flags in semantic_index 2023-09-19 12:20:59 -04:00
KCaverly 183758a7c5 fix Cargo.lock for merge 2023-09-19 11:44:51 -04:00
KCaverly 25bd357426 add recall and precision to semantic index 2023-09-18 18:25:02 -04:00
KCaverly 566bb9f71b add map to evaluation suite for semantic_index 2023-09-18 09:57:52 -04:00
Kyle Caverly b9c1f3d558 enable includes filtering for open and modified buffers (#2979) 
enable include file filtering for semantic searching in open and
modified buffers

Release Notes:

- Added include based filtering for modified buffers
 2023-09-15 15:23:24 -04:00
KCaverly 1433160a08 enable include based filtering for search inside open and modified buffers 2023-09-15 15:16:20 -04:00
KCaverly 04bd107ada add ndcg@k to evaluate metrics 2023-09-15 10:36:21 -04:00
KCaverly 3a661c5977 catchup with main 2023-09-15 09:31:33 -04:00
KCaverly 796bdd3da7 update searching in modified buffers to accomodate for excluded paths 2023-09-14 19:42:06 -04:00
Kyle Caverly 9b901df256 small fix to rate status update (#2967) 
Small fix to update code for rate limiting status.

Release Notes (Preview only)
- Fixed update to only stop updating status, when the rate limit is
reset to None
 2023-09-14 15:43:30 -04:00
KCaverly 8ff3e37044 small fix to rate status update 2023-09-14 15:42:21 -04:00
KCaverly c19c8899fe add initial search inside modified buffers 2023-09-14 14:58:34 -04:00
KCaverly 137dda3ee6 wip eval framework for semantic index 2023-09-14 09:30:19 -04:00
KCaverly 0c1b2e5aa6 cleaned up warnings 2023-09-13 20:04:53 -04:00
KCaverly eff44f9aa4 semantic index eval, indexing appropriately 2023-09-13 20:02:15 -04:00
KCaverly 6f29582fb0 progress on eval 2023-09-13 10:32:36 -04:00
KCaverly d4fbe99052 add eval for gpt-engineer 2023-09-12 21:27:35 -04:00
KCaverly 0d14bbbf5b add eval values for tree-sitter 2023-09-12 20:36:06 -04:00
KCaverly 66c967da88 start work on eval script for semantic_index 2023-09-12 16:25:31 -04:00
Kyle Caverly d4da5540ac Rate limiting status (#2955) 
Add a rate limit remaining status to Project Search Semantic Search
minor text

Release Notes (Preview-Only):

- Added tracking functionality within EmbeddingProvider, to track rate
limit expiry
- Update minor text within Project Search to show countdown remaining
before rate limit expiry
 2023-09-11 10:40:11 -04:00
KCaverly 74ab41000b Merge branch 'main' of github.com:zed-industries/zed into rate_limiting_status 2023-09-11 10:26:22 -04:00
KCaverly e678c7d9ee swap 
SystemTime for Instant throughout rate_limit_expiry tracking
 2023-09-11 10:26:14 -04:00
KCaverlyandAntonio 7df21f86dd move cx notify observe for rate_limit_expiry into ProjectState in the semantic index 
Co-authored-by: Antonio <antonio@zed.dev>
 2023-09-11 10:11:40 -04:00
KCaverly 37915ec4f2 updated notify to accomodate for updated countdown 2023-09-08 16:53:16 -04:00
KCaverly bf43f93197 updated semantic_index reset status to leverage target reset system time as opposed to duration 2023-09-08 15:04:50 -04:00
KCaverly a5ee8fc805 initial outline for rate limiting status updates 2023-09-08 12:35:15 -04:00
Kyle Caverly 560d6b1644 update semantic search to show no results if search query is blank (#2942) 
Update semantic search to show no search results if search query is
blank
 2023-09-07 14:58:00 -04:00
KCaverly cf5d1d91a4 update semantic search to go to no results if search query is blank 2023-09-07 14:43:41 -04:00
Kyle Caverly bbeb82f884 Token count fix (#2935) 
Fix token count for OpenAIEmbeddings

Release Notes (Preview Only)

- update token count calculation for truncated OpenAIEmbeddings
- increased request timeout for OpenAI
 2023-09-06 15:15:02 -04:00
KCaverly 265d02a583 update request timeout for open ai embeddings 2023-09-06 15:09:46 -04:00
KCaverly 17237f748c update token_count for OpenAIEmbeddings to accomodate for truncation 2023-09-06 15:09:15 -04:00
Kyle Caverly 49af2874bb Eager background indexing (#2928) 
This PR ships a series of optimizations for the semantic search engine.
Mostly focused on removing invalid states, optimizing requests to
OpenAI, and reducing token usage.

Release Notes (Preview-Only):

- Added eager incremental indexing in the background on a debounce.
- Added a local embeddings cache for reducing redundant calls to OpenAI.
- Moved to an Embeddings Queue model which ensures optimal batch sizes
at the token level, and atomic file & document writes.
- Adjusted OpenAI Embedding API requests to use provided backoff delays
during Rate Limiting.
- Removed flush races between parsing files step and embedding queue
steps.
- Moved truncation to parsing step reducing the probability that OpenAI
encounters bad data.
 2023-09-05 13:15:54 -04:00
KCaverly 8dbc0fe033 update pragma settings for improved database performance 2023-09-01 17:07:20 -04:00
KCaverlyandMax 54235f4fb1 updated embeddings background delay to 5 minutes 
Co-authored-by: Max <max@zed.dev>
 2023-09-01 13:04:09 -04:00
KCaverlyandMax e86964eb5d optimize insert file in vector database 
Co-authored-by: Max <max@zed.dev>
 2023-09-01 13:01:37 -04:00
KCaverlyandAntonio 524533cfb2 flush embeddings queue when no files are parsed for 250 milliseconds 
Co-authored-by: Antonio <antonio@zed.dev>
 2023-09-01 11:24:08 -04:00
KCaverlyandAntonio c4db914f0a move embeddings queue to use single hashmap for all changed paths 
Co-authored-by: Antonio <me@as-cii.com>
 2023-09-01 08:59:25 -04:00
KCaverly afa59abbcd WIP: work towards wiring up a embeddings_for_digest hashmap that is stored for all indexed files 2023-08-31 16:42:39 -04:00
KCaverly 50cfb067e7 fill embeddings with database values and skip during embeddings queue 2023-08-31 13:19:17 -04:00
KCaverly 7d4d6c871b fix bug for truncation ensuring no valid inputs are sent to openai 2023-08-30 17:42:16 -04:00
KCaverlyandMax 5abad58b0d moved semantic index to use embeddings queue to batch and managed for atomic database writes 
Co-authored-by: Max <max@zed.dev>
 2023-08-30 16:58:45 -04:00
KCaverlyandMax 76ce52df4e move queuing to embedding_queue functionality and update embedding provider to include trait items for max tokens per batch" 
Co-authored-by: Max <max@zed.dev>
 2023-08-30 16:01:28 -04:00
KCaverly 9781047156 move truncation to parsing step leveraging the EmbeddingProvider trait 2023-08-30 12:13:26 -04:00
KCaverly 76caea80f7 add should_truncate to embedding providers 2023-08-30 11:58:45 -04:00
KCaverly e377ada1a9 added token count to documents during parsing 2023-08-30 11:05:46 -04:00
KCaverlyandMax a7e6a65deb reindex files in the background after they have not been edited for 10 minutes 
Co-authored-by: Max <max@zed.dev>
 2023-08-29 17:14:44 -04:00
KCaverly 4f8b95cf0d add proper handling for open ai rate limit delays 2023-08-29 15:44:51 -04:00
Kyle Caverly bc7e9088fe Eager Semantic Indexing Queue (#2886) 
Optimization to the Semantic Indexing Engine.

We've transitioned from a framework in which the entire project tree is
walked at each index command, to an eager queuing method, in which an
initial queue of outstanding indexing work is initialized upon workspace
creation, and then subscriptions are leveraged for file change events to
continually keep an updated view on outstanding work.

This optimization contributes towards quicker user feedback, when
initializing or using Semantic Search functionality. It also opens the
doors towards better transparency across the system on outstanding
indexing work.

Release Notes:

- Refactored index operation queue to an eager queuing framework.
- Moved semantic search initialization to workspace creation.
- Adjusted rate limiting strategy on api delays to reduce time spent
waiting for rate limits.
 2023-08-25 11:03:05 +02:00
KCaverly ee97bc54cf cleaned up warnings 2023-08-25 10:38:01 +02:00
KCaverly 131950f670 add handling for Added file events to semantic index 2023-08-24 18:40:08 +02:00
KCaverlyandPiotr a892a51ec3 update initialize project call to accomodate for test scenarios 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-24 13:46:43 +02:00
KCaverlyandPiotr 0b204bfdc8 reindex semantic index when search project pane is reactivated in semantic mode 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-24 13:40:04 +02:00
KCaverlyandPiotr a1519e4c38 move semantic search project intialization to a subscribe event for workspace created 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-24 13:14:19 +02:00
KCaverlyandPiotr e8e7b294d8 add delete files operation for remaining files in database not included in current worktree 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-24 12:49:20 +02:00
KCaverlyandPiotr afe0e74868 remove worktree_file_mtimes in state as it is no longer used 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-24 12:42:41 +02:00
KCaverlyandPiotr aa07872a24 accomodate for duplicate entries in indexing queue 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-24 12:36:43 +02:00
KCaverly 3f9f742530 update rate limiting embeddings strategy to delay less 2023-08-24 11:45:52 +02:00
KCaverly b72c4c576b catchup with main 2023-08-23 22:31:39 +02:00