Github Copilot like Ghost text autocomplete

[shkreios]

Description

It would be nice instead of the adding the AI text directly to the document, it to be shown as a ghost text while you write something, similar to how GitHub copilot works and require tab to accept the generated text.

Implementation

I don't know how feasible this is using tiptap, I checked if there is any plugin for it that allows you to add ghost text i.e. text that is shown, but no selectable & disappears if you don't accept it via tab, but couldn't find any. So, this may require creating one.

Consideration

Since similar to GitHub copilot I would expect the autocomplete to be rerun if I don't accept the text and continue writing, this could result in a higher number of calls to openai. Therefore, it would probably be good to limit the output tokens

[steven-tey]

Great idea! I actually had this as part of v1:

CleanShot.2023-03-21.at.16.36.49.mp4

Tiptap doesn't support this natively so I had to build a custom Placeholder component: https://github.com/steven-tey/novel/blob/44878691b9acf237476675c5c6ac8aa9105c8c3b/lib/tiptap/placeholder.ts

There were 2 main issues though:

1. I couldn't figure out how to position the placeholder ghost text properly so I had to use a duplicate ghost text that was hidden by the overlying text – which won't work with bolded text & inline code
2. This has to be super fast for it to be useful – as you can see in the demo, it's a bit slow. It has to be fast enough that you won't need to stream the responses in (it'll just show up)

So I ended up scraping it 😅

If anyone can figure out a stellar implementation for this based on the work I've done before, I'm all ears 😄

[shkreios]

@steven-tey Thank you for the fast response

This has to be super fast for it to be useful – as you can see in the demo, it's a bit slow. It has to be fast enough that you won't need to stream the responses in (it'll just show up)

Have you already tried using gpt-3.5-turbo via the azure openai service. From what I have understood of the many chats on the openai forum, is that many are experiencing much slower responses from the openai api than chatgpt(plus) using the same model.

The current answer to these threads are that the openai api is using a shared engine ref and chatgpt is using an isolated one/separate env. The reason I am bringing up azure openai is that many posts mention that they are seeing 2x maybe faster response times for the same model and since the pricing, as far as i can tell, is the same, it's perhaps worth giving it a try.

[AeroXi]

Great idea! I actually had this as part of v1:

CleanShot.2023-03-21.at.16.36.49.mp4

Tiptap doesn't support this natively so I had to build a custom Placeholder component: https://github.com/steven-tey/novel/blob/44878691b9acf237476675c5c6ac8aa9105c8c3b/lib/tiptap/placeholder.ts

There were 2 main issues though:

1. I couldn't figure out how to position the placeholder ghost text properly so I had to use a duplicate ghost text that was hidden by the overlying text – which won't work with bolded text & inline code
2. This has to be super fast for it to be useful – as you can see in the demo, it's a bit slow. It has to be fast enough that you won't need to stream the responses in (it'll just show up)

So I ended up scraping it 😅

If anyone can figure out a stellar implementation for this based on the work I've done before, I'm all ears 😄

I really love this feature!
According to https://stackoverflow.com/a/9325686 the google search box may use similar implementation.
And I tried in Gmail, I believe they determine the ghost text style by the last user input character, like this:

Also, ChatGPT API are getting faster, I think it would be a good idea to add this feature first. Here's a video for Azure openai gpt-3.5-turbo, really fast:
https://twitter.com/gaodengming/status/1674972718704521216?s=20

[steven-tey]

Gonna have to try Azure OpenAI soon! Any of y'all wanna whip up a PR for this? 👀

[emil14]

Would be awesome to have API for this so I can use my own backend for that!