Optimistic Next Edit Suggestions #222
Conversation
avarayr
changed the title
avarayr
changed the title
|
hey @ulugbekna, i'd appreciate your [& other code owners] insight on this. There are some kinks to iron out, but in my testing, this massively improves the UX of NES. some points for discussion: Optimistic NES will multiply the number of NES requests by 3, which may affect the rate limit/user allowance. |
avarayr
force-pushed
the
feature/optimistic-next-edit-fetching
branch
4 times, most recently
from
ef9c8d9 to
24edf6a
Compare
|
@avarayr Thanks for the PR! We'll take a detailed look. We are also working on additional approaches to improve the experience while keeping the number of requests in check. (E.g., microsoft/vscode#255727 and microsoft/vscode#255728.) |
avarayr
force-pushed
the
feature/optimistic-next-edit-fetching
branch
3 times, most recently
from
69b2edb to
89154be
Compare
- Add prefetch chain mechanism that triggers when edits are shown - Prefetch up to 3 edits deep to reduce latency for sequential edits - Integrate with existing NextEditCache infrastructure - Track active prefetches to avoid duplicates - Clean up resources on disposal
avarayr
force-pushed
the
feature/optimistic-next-edit-fetching
branch
from
89154be to
f21a024
Compare
There was a problem hiding this comment.
@avarayr Thanks again for the PR! I have left a few questions. Could you rebase/merge it with the latest from main?
Could you also add a setting to enable/disable it? We would keep this off by default at first, so we can get it well tested and estimate the number of additional LLM calls.
| } | ||
|
|
||
| // Create a unique key for this prefetch based on document state | ||
| const prefetchKey = `${docId}:${documentState.value.length}:${depth}`; |
There was a problem hiding this comment.
Since we only show one suggestion at a time, maybe there should be at most one active prefetch at a time? (Avoiding the need for a key here.)
| try { | ||
|
|
||
| // Create a synthetic document that represents the future state | ||
| const syntheticDoc = this._workspace.getDocument(docId); |
|
|
||
| // Create a synthetic document wrapper that returns our future state | ||
| const syntheticDocWrapper = { | ||
| ...doc, |
There was a problem hiding this comment.
Not sure this is safe to copy like that, will need to take closer look.
| // Optimistic fetching already triggered in handleShown | ||
| } | ||
|
|
||
| public handleRejection(docId: DocumentId, suggestion: NextEditResult) { |
There was a problem hiding this comment.
Should the prefetch be canceled when the current suggestion is rejected? Similarly: What if the current suggestion is ignored (callback below).
4 participants
Basis: Tab, Tab, Tab — Objects in motion want to stay in motion.
Summary
This PR implements optimistic prefetching for next edit suggestions, eliminating delay between accepting an edit and receiving the next suggestion by assuming that user will keep pressing tab to accept sequential edits.
Problem
Currently, when users accept an inline edit suggestion, there's a noticeable delay
before the next suggestion appears. This breaks the flow of rapid sequential edits.
Solution
Implement predictive fetching that anticipates user behavior based on the "objects in
motion stay in motion" principle. When a NES is shown, the system
immediately fetches the next edit in the background, assuming the user will continue
accepting suggestions.
All existing tests continue to pass.
Performance
Discussion
Optimistic NES will multiply the number of NES requests by 3, which may affect the rate limit/user allowance.