Bug [@langchain/anthropic]: Unable to send images

[guypy]

Checked other resources

• I added a very descriptive title to this issue.
• I searched the LangChain.js documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain.js rather than my code.
• The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

const attachmentsMessage = new HumanMessage({
        content: [
          { type: 'text', text: 'Here are the attachments from the email:' }, 
          {
              type: 'image',
              source: {
                type: 'base64',
                media_type: attachment.content_type,
                data: attachment.base64Data,
              },
           },
        ],
      });

Sending the following message results in an Unsupported message content format error.

Error Message and Stack Trace (if applicable)

No response

Description

According to the Anthropic's documentation, sending images to the model is supported either by Base64, or a URL in the following ways:

base64:

const message = await anthropic.messages.create({
    model: "claude-3-7-sonnet-20250219",
    max_tokens: 1024,
    messages: [
      {
        role: "user",
        content: [
          {
            type: "image",
            source: {
              type: "base64",
              media_type: "image/jpeg",
              data: imageData, // Base64-encoded image data as string
            }
          },
          {
            type: "text",
            text: "Describe this image."
          }
        ]
      }
    ]
  });

URL:

 const message = await anthropic.messages.create({
    model: "claude-3-7-sonnet-20250219",
    max_tokens: 1024,
    messages: [
      {
        role: "user",
        content: [
          {
            type: "image",
            source: {
              type: "url",
              url: "https://upload.wikimedia.org/wikipedia/commons/a/a7/Camponotus_flavomarginatus_ant.jpg"
            }
          },
          {
            type: "text",
            text: "Describe this image."
          }
        ]
      }
    ]
  });

However, the code in the @langchain/anthropic package does not follow those examples, it expects an image_url type (@see https://github.com/langchain-ai/langchainjs/blob/main/libs/langchain-anthropic/src/utils/message_inputs.ts#L125-L136) and therefore throws an Unsupported message content format error.

Passing an image_url type, passes the content to _formatImage which returns a format that does not match Anthropic's docs.

System Info

{
  name: '@langchain/anthropic',
  'dist-tags': {
    next: '0.3.6-rc.0',
    latest: '0.3.15'
  },
  versions: [
    '0.0.0',
    '0.0.1',
    '0.0.2',
    '0.0.3',
    '0.0.4',
    '0.0.5',
    '0.0.6',
    '0.0.7',
    '0.0.8',
    '0.0.9',
    '0.0.10',
    '0.1.0',
    '0.1.1',
    '0.1.2',
    '0.1.3',
    '0.1.4',
    '0.1.5',
    '0.1.6',
    '0.1.7',
    '0.1.8',
    '0.1.9',
    '0.1.10',
    '0.1.11',
    '0.1.12-rc.0',
    '0.1.12',
    '0.1.13',
    '0.1.14',
    '0.1.15',
    '0.1.16',
    '0.1.17-rc.0',
    '0.1.17',
    '0.1.18',
    '0.1.19',
    '0.1.20',
    '0.1.21',
    '0.2.0',
    '0.2.1',
    '0.2.2',
    '0.2.3',
    '0.2.4',
    '0.2.5',
    '0.2.6',
    '0.2.7',
    '0.2.8',
    '0.2.9',
    '0.2.10',
    '0.2.11',
    '0.2.12',
    '0.2.13',
    '0.2.14',
    '0.2.15',
    '0.2.16',
    '0.2.17',
    '0.2.18',
    '0.3.0-rc.0',
    '0.3.0',
    '0.3.1',
    '0.3.2-rc.0',
    '0.3.2',
    '0.3.3',
    '0.3.4',
    '0.3.5',
    '0.3.6-rc.0',
    '0.3.6',
    '0.3.7',
    '0.3.8',
    '0.3.9',
    '0.3.10',
    '0.3.11',
    '0.3.12',
    '0.3.13',
    '0.3.14',
    '0.3.15'
  ],
  time: {
    created: '2023-11-22T19:15:49.166Z',
    modified: '2025-02-28T23:20:43.207Z',
    '0.0.0': '2023-11-22T19:15:49.421Z',
    '0.0.1': '2023-11-22T19:32:30.698Z',
    '0.0.2': '2023-11-22T20:00:08.055Z',
    '0.0.3': '2023-11-23T05:00:00.868Z',
    '0.0.4': '2023-12-12T17:56:28.382Z',
    '0.0.5': '2023-12-17T05:52:44.155Z',
    '0.0.6': '2023-12-20T02:30:20.627Z',
    '0.0.7': '2023-12-20T17:15:12.956Z',
    '0.0.8': '2023-12-21T07:47:51.836Z',
    '0.0.9': '2023-12-29T15:15:44.733Z',
    '0.0.10': '2024-02-02T19:07:12.400Z',
    '0.1.0': '2024-02-26T18:04:05.495Z',
    '0.1.1': '2024-03-04T15:43:55.331Z',
    '0.1.2': '2024-03-05T07:46:01.906Z',
    '0.1.3': '2024-03-05T23:31:46.088Z',
    '0.1.4': '2024-03-07T00:43:42.764Z',
    '0.1.5': '2024-03-10T00:28:09.624Z',
    '0.1.6': '2024-03-12T21:04:18.082Z',
    '0.1.7': '2024-03-14T05:39:14.339Z',
    '0.1.8': '2024-03-16T00:35:41.530Z',
    '0.1.9': '2024-03-26T06:16:31.307Z',
    '0.1.10': '2024-04-04T21:12:54.047Z',
    '0.1.11': '2024-04-09T08:20:28.630Z',
    '0.1.12-rc.0': '2024-04-09T09:17:45.410Z',
    '0.1.12': '2024-04-09T17:50:24.362Z',
    '0.1.13': '2024-04-10T16:52:09.247Z',
    '0.1.14': '2024-04-11T08:48:32.150Z',
    '0.1.15': '2024-04-17T00:17:48.182Z',
    '0.1.16': '2024-04-17T02:35:06.755Z',
    '0.1.17-rc.0': '2024-05-02T21:48:20.743Z',
    '0.1.17': '2024-05-02T21:57:11.684Z',
    '0.1.18': '2024-05-15T17:33:17.208Z',
    '0.1.19': '2024-05-16T18:25:57.384Z',
    '0.1.20': '2024-05-17T22:06:23.181Z',
    '0.1.21': '2024-05-19T02:36:57.880Z',
    '0.2.0': '2024-06-01T17:26:52.996Z',
    '0.2.1': '2024-06-06T17:16:17.089Z',
    '0.2.2': '2024-06-22T00:06:22.068Z',
    '0.2.3': '2024-06-28T22:14:26.804Z',
    '0.2.4': '2024-07-16T04:19:50.285Z',
    '0.2.5': '2024-07-18T05:30:01.582Z',
    '0.2.6': '2024-07-18T05:34:01.289Z',
    '0.2.7': '2024-07-23T19:22:17.837Z',
    '0.2.8': '2024-07-23T20:45:59.541Z',
    '0.2.9': '2024-07-24T23:35:29.324Z',
    '0.2.10': '2024-07-26T23:57:48.346Z',
    '0.2.11': '2024-07-29T21:18:32.348Z',
    '0.2.12': '2024-08-02T03:06:08.792Z',
    '0.2.13': '2024-08-06T23:39:22.325Z',
    '0.2.14': '2024-08-10T15:23:35.615Z',
    '0.2.15': '2024-08-14T13:58:00.341Z',
    '0.2.16': '2024-09-04T20:16:34.551Z',
    '0.2.17': '2024-09-06T23:59:26.824Z',
    '0.3.0-rc.0': '2024-09-07T00:08:22.032Z',
    '0.2.18': '2024-09-10T22:41:27.935Z',
    '0.3.0': '2024-09-13T21:24:25.230Z',
    '0.3.1': '2024-09-17T00:03:52.685Z',
    '0.3.2-rc.0': '2024-09-24T18:19:03.839Z',
    '0.3.2': '2024-09-24T20:37:20.218Z',
    '0.3.3': '2024-09-25T00:30:30.448Z',
    '0.3.4': '2024-10-17T00:03:08.839Z',
    '0.3.5': '2024-10-17T04:01:58.877Z',
    '0.3.6-rc.0': '2024-10-27T20:59:28.129Z',
    '0.3.6': '2024-10-27T21:18:06.651Z',
    '0.3.7': '2024-10-29T21:53:10.000Z',
    '0.3.8': '2024-11-13T02:14:12.147Z',
    '0.3.9': '2024-12-09T23:34:26.422Z',
    '0.3.10': '2024-12-15T17:08:24.730Z',
    '0.3.11': '2024-12-16T23:16:37.131Z',
    '0.3.12': '2025-01-18T08:49:53.708Z',
    '0.3.13': '2025-02-05T19:25:44.353Z',
    '0.3.14': '2025-02-24T23:37:25.274Z',
    '0.3.15': '2025-02-28T23:20:42.927Z'
  },
  bugs: {
    url: 'https://github.com/langchain-ai/langchainjs/issues'
  },
  author: {
    name: 'LangChain'
  },
  license: 'MIT',
  homepage: 'https://github.com/langchain-ai/langchainjs/tree/main/libs/langchain-anthropic/',
  keywords: [
    'llm',
    'ai',
    'gpt3',
    'chain',
    'prompt',
    'prompt engineering',
    'chatgpt',
    'machine learning',
    'ml',
    'anthropic',
    'embeddings',
    'vectorstores'
  ],
  repository: {
    type: 'git',
    url: 'git+ssh://git@github.com/langchain-ai/langchainjs.git'
  },
  description: 'Anthropic integrations for LangChain.js',
  maintainers: [
    {
      name: 'hwchase17',
      email: 'hw.chase.17@gmail.com'
    },
    {
      name: 'jacoblee93',
      email: 'jacoblee93@gmail.com'
    },
    {
      name: 'basproul',
      email: 'braceasproul@gmail.com'
    },
    {
      name: 'eric_langchain',
      email: 'eric@langchain.dev'
    },
    {
      name: 'andrewnguonly',
      email: 'andrewnguonly@gmail.com'
    },
    {
      name: 'nfcampos',
      email: 'nuno@boringbits.io'
    },
    {
      name: 'vbarda',
      email: 'vadym@langchain.dev'
    },
    {
      name: 'davidduong',
      email: 'david@duong.cz'
    },
    {
      name: 'maddyadams',
      email: 'madelyn.adams1@gmail.com'
    },
    {
      name: 'sam_noyes',
      email: 'sam@langchain.dev'
    },
    {
      name: 'benjamincburns',
      email: 'benjamin.c.burns@gmail.com'
    }
  ],
  readmeFilename: 'README.md',
  version: '0.3.15',
  type: 'module',
  engines: {
    node: '>=18'
  },
  main: './index.js',
  types: './index.d.ts',
  scripts: {
    build: 'yarn turbo:command build:internal --filter=@langchain/anthropic',
    'build:internal': 'yarn lc_build --create-entrypoints --pre --tree-shaking --gen-maps',
    'lint:eslint': 'NODE_OPTIONS=--max-old-space-size=4096 eslint --cache --ext .ts,.js src/',
    'lint:dpdm': 'dpdm --exit-code circular:1 --no-warning --no-tree src/*.ts src/**/*.ts',
    lint: 'yarn lint:eslint && yarn lint:dpdm',
    'lint:fix': 'yarn lint:eslint --fix && yarn lint:dpdm',
    clean: 'rm -rf .turbo dist/',
    prepack: 'yarn build',
    test: 'NODE_OPTIONS=--experimental-vm-modules jest --testPathIgnorePatterns=\\.int\\.test.ts --testTimeout 30000 --maxWorkers=50%',
    'test:watch': 'NODE_OPTIONS=--experimental-vm-modules jest --watch --testPathIgnorePatterns=\\.int\\.test.ts',
    'test:single': 'NODE_OPTIONS=--experimental-vm-modules yarn run jest --config jest.config.cjs --testTimeout 100000',
    'test:int': 'NODE_OPTIONS=--experimental-vm-modules jest --testPathPattern=\\.int\\.test.ts --testTimeout 100000 --maxWorkers=50%',
    'test:standard:unit': 'NODE_OPTIONS=--experimental-vm-modules jest --testPathPattern=\\.standard\\.test.ts --testTimeout 100000 --maxWorkers=50%',
    'test:standard:int': 'NODE_OPTIONS=--experimental-vm-modules jest --testPathPattern=\\.standard\\.int\\.test.ts --testTimeout 100000 --maxWorkers=50%',
    'test:standard': 'yarn test:standard:unit && yarn test:standard:int',
    format: 'prettier --config .prettierrc --write "src"',
    'format:check': 'prettier --config .prettierrc --check "src"'
  },
  dependencies: {
    '@anthropic-ai/sdk': '^0.37.0',
    'fast-xml-parser': '^4.4.1',
    zod: '^3.22.4',
    'zod-to-json-schema': '^3.22.4'
  },
  peerDependencies: {
    '@langchain/core': '>=0.2.21 <0.4.0'
  },
  devDependencies: {
    '@anthropic-ai/vertex-sdk': '^0.4.1',
    '@jest/globals': '^29.5.0',
    '@langchain/core': 'workspace:*',
    '@langchain/scripts': '>=0.1.0 <0.2.0',
    '@langchain/standard-tests': '0.0.0',
    '@swc/core': '^1.3.90',
    '@swc/jest': '^0.2.29',
    dpdm: '^3.12.0',
    eslint: '^8.33.0',
    'eslint-config-airbnb-base': '^15.0.0',
    'eslint-config-prettier': '^8.6.0',
    'eslint-plugin-import': '^2.27.5',
    'eslint-plugin-jest': '^27.6.0',
    'eslint-plugin-no-instanceof': '^1.0.1',
    'eslint-plugin-prettier': '^4.2.1',
    jest: '^29.5.0',
    'jest-environment-node': '^29.6.4',
    prettier: '^2.8.3',
    'release-it': '^17.6.0',
    rimraf: '^5.0.1',
    'ts-jest': '^29.1.0',
    typescript: '~5.1.6'
  },
  publishConfig: {
    access: 'public'
  },
  exports: {
    '.': {
      types: {
        import: './index.d.ts',
        require: './index.d.cts',
        default: './index.d.ts'
      },
      import: './index.js',
      require: './index.cjs'
    },
    './experimental': {
      types: {
        import: './experimental.d.ts',
        require: './experimental.d.cts',
        default: './experimental.d.ts'
      },
      import: './experimental.js',
      require: './experimental.cjs'
    },
    './package.json': './package.json'
  },
  gitHead: '6bcf2e7d32dcfe8741ee8a41d8b20d25189d1801',
  dist: {
    integrity: 'sha512-Ar2viYcZ64idgV7EtCBCb36tIkNtPAhQRxSaMTWPHGspFgMfvwRoleVri9e90sCpjpS9xhlHsIQ0LlUS/Atsrw==',
    shasum: '0244cdb345cb492eb40aedd681881ebadfbb73f2',
    tarball: 'https://registry.npmjs.org/@langchain/anthropic/-/anthropic-0.3.15.tgz',
    fileCount: 65,
    unpackedSize: 199787,
    signatures: [
      {
        keyid: 'SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U',
        sig: 'MEQCIGZ7f0I5TEnEL10ay+Fo8/DKCn81/aKBP9sxHipdpHusAiBdfbJ1H62aEwhtO/MVxiQ0nnsajOVPcvKEZYFgU2P0Ng=='
      }
    ]
  },
  directories: {}
}

[dosubot]

Hey, @guypy! I'm here to help you out with any bugs or questions you have while waiting for a human maintainer. Let's see what's going on with the issue you've raised.

The Unsupported message content format error in the @langchain/anthropic package occurs because it currently only supports base64-encoded images. The _formatImage function in the package checks for base64-encoded images using a regular expression, and if the image does not match this format, it throws an error ^[1][2].

To resolve this issue, ensure that your image data is correctly formatted as a base64-encoded string. If you want to use URL formats as supported by Anthropic's documentation, the package would need to be updated to handle both base64 and URL formats. This would involve modifying the _formatImage function and related logic to accommodate URLs as well.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}