Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Anthropic (Claude) provider #22

Merged
merged 7 commits into from
Jan 24, 2025
Merged

Conversation

brichet
Copy link
Collaborator

@brichet brichet commented Dec 9, 2024

This PR adds the Anthropic provider as a chat model.
It does not include completion.

@brichet brichet added the enhancement New feature or request label Dec 9, 2024
@brichet brichet marked this pull request as ready for review December 16, 2024 12:09
@brichet brichet marked this pull request as draft December 17, 2024 12:36
@brichet
Copy link
Collaborator Author

brichet commented Dec 18, 2024

I added a first version of code completion.
The idea is to give a context to the AI, and to send the current prompt as a message to complete.
The 2 issues I have so far are:

  • it is really slow on my side
  • anthropic raises if there is a space at the end of the prompt. Currently this PR is removing it, but it can lead to wrong suggestions

@jtpio
Copy link
Member

jtpio commented Dec 19, 2024

Thanks @brichet for looking into this 👍

I just tried this PR locally and there seems to be many requests made to the anthropic API, so maybe there is indeed something not expected happening:

jupyterlite-anthropic.webm

@jtpio
Copy link
Member

jtpio commented Dec 19, 2024

Interestingly when testing with Jupyter AI, completions don't seem to work and trigger some warnings in the dev tools console:

jupyter-ai-lab-434-anthropic.webm

The Jupyter Server logs also show some long response time for the inline completer:

image

@brichet
Copy link
Collaborator Author

brichet commented Dec 20, 2024

Thanks for testing it @jtpio

Using Anthropic completion with jupyter-ai do not trigger error on my side, but the suggestions are not really relevant (most of the time):

Peek 2024-12-20 13-59

Indeed, you're right about the extra requests, I'll take a closer look.

@jtpio
Copy link
Member

jtpio commented Dec 20, 2024

Using Anthropic completion with jupyter-ai do not trigger error on my side, but the suggestions are not really relevant (most of the time):

Indeed according to your screenshot there seems to be an issue with how Jupyter AI handles the inline completer suggestions. So likely an issue in Jupyter AI (we could open one if non exists already)

@brichet
Copy link
Collaborator Author

brichet commented Jan 21, 2025

Looks like the default model is claude-2.1, which seems to be slower and less accurate than claude-3-5-sonnet-20241022. Setting it manually improve the performances of the code completion.

Maybe we should revive #14 to get all the settings for each providers.

@jtpio
Copy link
Member

jtpio commented Jan 22, 2025

Maybe we should revive #14 to get all the settings for each providers.

Yeah the JupyterLab related change should now be available in 4.4.0a2.

@jtpio
Copy link
Member

jtpio commented Jan 22, 2025

Looks like the default model is claude-2.1, which seems to be slower and less accurate than claude-3-5-sonnet-20241022. Setting it manually improve the performances of the code completion.

Indeed, I just tried it locally and it seems to be faster with Claude 3.5 Sonnet.

There seems to be some issues with the quality of completion being returned sometimes, but it's something that can likely be fixed by improving the prompt or the use of the langchain primitives:

image

@brichet brichet force-pushed the anthropic_provider branch from fbb3833 to 54f96ca Compare January 23, 2025 14:35
@brichet
Copy link
Collaborator Author

brichet commented Jan 23, 2025

Unfortunately #14 does not work out of the box for Anthropic provider.

@brichet brichet marked this pull request as ready for review January 24, 2025 11:15
@jtpio
Copy link
Member

jtpio commented Jan 24, 2025

Unfortunately #14 does not work out of the box for Anthropic provider.

Looks like the settings generator script needs to be updated for each new provided being added?

@brichet
Copy link
Collaborator Author

brichet commented Jan 24, 2025

Looks like the settings generator script needs to be updated for each new provided being added?

Yes, but it would be great if we could have a generic way to retrieve the available settings.

If @langchainjs is using a naming convention for the modules and the "input" interfaces we can make it more generic, but it seems to not be the case (eg. ChatMistralAIInput and AnthropicInput).

The other problem is about the use of ts-json-schema-generator, which raises an error when trying to build the settings for Anthropic without excluding clientOptions (this king of error seems to not be handled correctly):

/home/brichet/projects/jupyterlite-ai/node_modules/ts-json-schema-generator/dist/src/NodeParser/TypeReferenceNodeParser.js:59
        return this.childNodeParser.createType(typeSymbol.declarations.filter((n) => !invalidTypes[n.kind])[0], this.createSubContext(node, context));
                                                                       ^

TypeError: Cannot read properties of undefined (reading 'filter')
    at TypeReferenceNodeParser.createType (/home/brichet/projects/jupyterlite-ai/node_modules/ts-json-schema-generator/dist/src/NodeParser/TypeReferenceNodeParser.js:59:72)
    at ChainNodeParser.createType (/home/brichet/projects/jupyterlite-ai/node_modules/ts-json-schema-generator/dist/src/ChainNodeParser.js:30:45)
    at TypeReferenceNodeParser.createSubContext (/home/brichet/projects/jupyterlite-ai/node_modules/ts-json-schema-generator/dist/src/NodeParser/TypeReferenceNodeParser.js:65:62)
    at TypeReferenceNodeParser.createType (/home/brichet/projects/jupyterlite-ai/node_modules/ts-json-schema-generator/dist/src/NodeParser/TypeReferenceNodeParser.js:59:118)
    at ChainNodeParser.createType (/home/brichet/projects/jupyterlite-ai/node_modules/ts-json-schema-generator/dist/src/ChainNodeParser.js:30:45)
    at TypeAliasNodeParser.createType (/home/brichet/projects/jupyterlite-ai/node_modules/ts-json-schema-generator/dist/src/NodeParser/TypeAliasNodeParser.js:36:43)
    at AnnotatedNodeParser.createType (/home/brichet/projects/jupyterlite-ai/node_modules/ts-json-schema-generator/dist/src/NodeParser/AnnotatedNodeParser.js:29:47)
    at ExposeNodeParser.createType (/home/brichet/projects/jupyterlite-ai/node_modules/ts-json-schema-generator/dist/src/ExposeNodeParser.js:24:45)
    at CircularReferenceNodeParser.createType (/home/brichet/projects/jupyterlite-ai/node_modules/ts-json-schema-generator/dist/src/CircularReferenceNodeParser.js:22:43)
    at ChainNodeParser.createType (/home/brichet/projects/jupyterlite-ai/node_modules/ts-json-schema-generator/dist/src/ChainNodeParser.js:30:45)

In conclusion, for now I think we'll have to update the generator, but hopefully it should be painless next time 😄

@brichet brichet force-pushed the anthropic_provider branch from 3096bc5 to b74d481 Compare January 24, 2025 16:10
let schema;

// Workaround to exclude some properties from a type or interface.
if (desc.excludedProps) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's probably fine to have exclusion clauses like this for now as we figure things out with several providers 👍

Also found that some props had to be excluded over in #27.

// Anthropic does not allow whitespace at the end of the AIMessage
const trimmedPrompt = prompt.trim();

const messages = [
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wondering if this list of messages should be recreated each time, or it should be persisted per provider?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some other open questions: should the list of messages be shared across providers? Or should it be reset when a user switches to another provider?

Maybe a reset would be fine to keep things tidy. Also users probably wouldn't change providers often in practice?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opened #29

Copy link
Member

@jtpio jtpio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

Testing locally and it seems to be working fine 👍

By default the inline completion does seem to be quite slow:

jupyterlite-ai-anthropic-default.mov

But switching to claude-3-5-sonnet-latest as the model does seem to be making things faster:

jupyterlite-ai-anthropic-35.mov

Left a couple of inline comments, that we can also track separately.

@jtpio jtpio merged commit ba99ad6 into jupyterlite:main Jan 24, 2025
7 checks passed
@brichet brichet deleted the anthropic_provider branch January 24, 2025 21:13
@brichet
Copy link
Collaborator Author

brichet commented Jan 24, 2025

Thanks for the review @jtpio

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants