Huggingface api-inference API endpoint support

[nicholaswmin]

Summary

Using this settings.json, exactly as documented here: https://zed.dev/docs/assistant/configuration#openai-custom-models:

  "language_models": {
    "openai": {
      "version": "1",
      "api_url": "https://api-inference.huggingface.co/v1/",
      "available_models": [
          {
            "name": "codellama/CodeLlama-34b-Instruct-hf",
            "display_name": "CodeLlama-34b-Instruct",
            "max_tokens": 128000
          }
        ]
    }
  },

  "assistant": {
    "enabled": true,
    "default_model": {
      "provider": "openai",
      "model": "codellama/CodeLlama-34b-Instruct-hf"
    },
    "version": "2",
    "button": true,
    "default_width": 480,
    "dock": "right"
  },

Actual Behavior:

• The CodeLlama-34b-Instruct\model appears in my models
• responds with 404 NOT Found.

Expected Behavior:

• The CodeLlama-34b-Instruct model appears in my models
• responds with answer

Zed Version and System Specs

Zed: v0.173.10 (Zed)
OS: macOS 14.4.1
Memory: 24 GiB
Architecture: aarch64

• restated zed
• refreshed token

The behavior around assistant configuration is very confusing. It mixes up OpenAI itself with OpenAI-like API, insists the settings.json lists an actual openai model in some cases; i've found it confusing At the very least.

there's more, like this example from the decs;
it has double points-of-truth for versioning. but why?

  "language_models": {
    "openai": {
      "api_url": "https://api.x.ai/v1", // version defined here
      "available_models": [
        {
          "name": "grok-beta", 
          "display_name": "X.ai Grok (Beta)",
          "max_tokens": 131072
        }
      ],
      "version": "1" // and here?
    },
  }
```er
- and the assistant object iself has a version?
- it doesn't seem to log problematic 404 in the log file either

although if valid - these would deserve their own issue maybe

[notpeter]

I haven't ever used Zed via api-inference.huggingface.co perhaps someone else can chime in with their experience.

Can you try removing the trailing slash from your api_url?

The version keys are only for zed settings and have no relation to the v1 in the url.

[nicholaswmin]

Can you try removing the trailing slash from your api_url?

I have, same issue.

including a working openAI-compatible sample:

// - run: npm i openai
// - replace "hf_api_key" with HF API key
import { OpenAI } from "openai";

const client = new OpenAI({
  baseURL: "https://router.huggingface.co/hf-inference/v1",
  apiKey: "hf_api_key"
})

const chatCompletion = await client.chat.completions.create({
  model: "meta-llama/Llama-3.2-3B-Instruct",
  messages: [{ role: "user", content: "say hello world" }],
  max_tokens: 500
})

console.log(chatCompletion.choices[0].message)

The version keys are only for zed settings and have no relation to the v1 in the url.

ok that's good to know.

Any way I can log outbound requests?

[notpeter]

Not easily. I've been meaning to write a tutorial for how to use a mitm proxy to inspect requests to/from Zed, but haven't yet. Shouldn't be too hard now that Zed properly support loading CA bundles from your OS.

[nicholaswmin]

was it gonna be with wireshark? maybe i can do it if so

[nicholaswmin]

Ok here's how I've managed to get it working via Proxyman=

Proxyman's free version provides sufficient all you need for basic request monitoring,
without a time limit. Plus it auto-sorts out CA configuration or pf lists which always feels like defusing a time bomb.

Monitor zed in/out traffic

FOR

• Overview
• Installation
• Configuration Steps
  • Install CA Certificate
  • Configure Zed
• Usage

Installation

Install Proxyman using Homebrew:

brew install --cask proxyman

Launch the application:

open /Applications/Proxyman.app/

Configuration Steps

Install CA Certificate

1. Open Proxyman
2. Navigate to:
   • Proxyman → Settings
   • Scroll to bottom of initial settings page
   • Click "Generate new CA and install"

Configure Zed

1. Open Zed's settings.json
2. Add the proxy configuration:

{
    "proxy": "http://localhost:9090"
}

Note: 9090 is Proxyman's default port

Usage

1. Restart both applications:
   • Quit Zed
   • Quit Proxyman
   • Relaunch both applications
2. In Proxyman:
   • Filter requests by "zed"
   • Monitor network traffic

All requests should now be logged while maintaining normal functionality.