`(buffer-file-name)` evaluates to `nil` during the execution of `lsp-diagnostics--request-pull-diagnostics`

[prosoitos]

Thank you for the bug report

• I am using the latest version of lsp-mode related packages.
• I checked FAQ and Troubleshooting sections
• You may also try reproduce the issue using clean environment using the following command: M-x lsp-start-plain

Bug description

Bug initially reported in the emacs-reformatter repo (purcell/emacs-reformatter#63), with cause identified by @dzhu.

Quoting @dzhu here:

[Problem] appears to be caused by (buffer-file-name) evaluating to nil during the execution of lsp-diagnostics--request-pull-diagnostics. As a result of the error, the locations of diagnostics don't update on save if formatting moves around the relevant code.
[...]
The same problem is caused by directly running the Emacs built-in insert-file-contents.

(end quotation).

On my side, this causes me to get:

Error processing message (wrong-type-argument stringp nil)

when I run the reformatter function with the following debugger backtrace:

Debugger entered--Lisp error: (wrong-type-argument stringp nil)
  lsp-diagnostics--update-path(nil [0 0 3 0 0])
  lsp-diagnostics--convert-and-update-path-stats(#s(lsp--workspace :ewoc nil :server-capabilities #<hash-table equal 10/10 0x1f778412574e ...> :registered-server-capabilities (#s(lsp--registered-capability :id "ruff-server-watch" :method "workspace/didChangeWatchedFiles" :options #<hash-table equal 1/1 0x1f7785a1c3ed ...>)) :root "/home/marie/parvus/prog/mint/ai/jxai" :client #s(lsp--client :language-id nil :add-on? t :new-connection (:connect #f(compiled-function (filter sentinel name environment-fn workspace) #<bytecode -0x1090a9e16753dac8>) :test? #f(compiled-function () #<bytecode 0x5fb916beec8a634>)) :ignore-regexps nil :ignore-messages nil :notification-handlers #<hash-table equal 0/0 0x1f7785667a80 ...> :request-handlers #<hash-table equal 0/0 0x1f7785667995 ...> :response-handlers #<hash-table eql 0/24 0x1f77856679ab ...> :prefix-function nil :uri-handlers #<hash-table equal 0/0 0x1f77856679b8 ...> :action-handlers #<hash-table equal 0/0 0x1f77856679d6 ...> :action-filter nil :major-modes nil :activation-fn #f(compiled-function (file-name mode) #<bytecode -0x164da5e46142777e>) :priority -2 :server-id ruff :multi-root nil :initialization-options #f(compiled-function () #<bytecode -0x105191dd44d3b3b1>) :semantic-tokens-faces-overrides nil :custom-capabilities nil :library-folders-fn nil :before-file-open-fn nil :initialized-fn nil :remote? nil :completion-in-comments? nil :path->uri-fn nil :uri->path-fn nil :environment-fn nil :after-open-fn nil :async-request-handlers #<hash-table equal 0/0 0x1f77856679cf ...> :download-server-fn nil :download-in-progress? nil :buffers nil :synchronize-sections nil) :host-root nil :proc #<process ruff> :cmd-proc #<process ruff> :buffers (#<buffer test_jxai.py>) :semantic-tokens-faces nil :semantic-tokens-modifier-faces nil :extra-client-capabilities nil :status initialized :metadata #<hash-table equal 0/0 0x1f7785779313 ...> :watches #<hash-table equal 0/0 0x1f7785779305 ...> :workspace-folders nil :last-id 0 :status-string nil :shutdown-action nil :diagnostics #<hash-table equal 1/6 0x1f778577950e ...> :work-done-tokens #<hash-table equal 0/0 0x1f7785779518 ...>) nil [#<hash-table equal 8/8 0x1f77857bc64d ...> #<hash-table equal 8/8 0x1f77857a4abf ...> #<hash-table equal 8/8 0x1f77857a4e61 ...>])
  lsp-diagnostics--apply-pull-diagnostics(#s(lsp--workspace :ewoc nil :server-capabilities #<hash-table equal 10/10 0x1f778412574e ...> :registered-server-capabilities (#s(lsp--registered-capability :id "ruff-server-watch" :method "workspace/didChangeWatchedFiles" :options #<hash-table equal 1/1 0x1f7785a1c3ed ...>)) :root "/home/marie/parvus/prog/mint/ai/jxai" :client #s(lsp--client :language-id nil :add-on? t :new-connection (:connect #f(compiled-function (filter sentinel name environment-fn workspace) #<bytecode -0x1090a9e16753dac8>) :test? #f(compiled-function () #<bytecode 0x5fb916beec8a634>)) :ignore-regexps nil :ignore-messages nil :notification-handlers #<hash-table equal 0/0 0x1f7785667a80 ...> :request-handlers #<hash-table equal 0/0 0x1f7785667995 ...> :response-handlers #<hash-table eql 0/24 0x1f77856679ab ...> :prefix-function nil :uri-handlers #<hash-table equal 0/0 0x1f77856679b8 ...> :action-handlers #<hash-table equal 0/0 0x1f77856679d6 ...> :action-filter nil :major-modes nil :activation-fn #f(compiled-function (file-name mode) #<bytecode -0x164da5e46142777e>) :priority -2 :server-id ruff :multi-root nil :initialization-options #f(compiled-function () #<bytecode -0x105191dd44d3b3b1>) :semantic-tokens-faces-overrides nil :custom-capabilities nil :library-folders-fn nil :before-file-open-fn nil :initialized-fn nil :remote? nil :completion-in-comments? nil :path->uri-fn nil :uri->path-fn nil :environment-fn nil :after-open-fn nil :async-request-handlers #<hash-table equal 0/0 0x1f77856679cf ...> :download-server-fn nil :download-in-progress? nil :buffers nil :synchronize-sections nil) :host-root nil :proc #<process ruff> :cmd-proc #<process ruff> :buffers (#<buffer test_jxai.py>) :semantic-tokens-faces nil :semantic-tokens-modifier-faces nil :extra-client-capabilities nil :status initialized :metadata #<hash-table equal 0/0 0x1f7785779313 ...> :watches #<hash-table equal 0/0 0x1f7785779305 ...> :workspace-folders nil :last-id 0 :status-string nil :shutdown-action nil :diagnostics #<hash-table equal 1/6 0x1f778577950e ...> :work-done-tokens #<hash-table equal 0/0 0x1f7785779518 ...>) nil "full" [#<hash-table equal 8/8 0x1f77857bc64d ...> #<hash-table equal 8/8 0x1f77857a4abf ...> #<hash-table equal 8/8 0x1f77857a4e61 ...>])
  #f(compiled-function (input0) #<bytecode -0x196b94d9566559d>)(#<hash-table equal 2/2 0x1f77857a54f1 ...>)
  apply(#f(compiled-function (input0) #<bytecode -0x196b94d9566559d>) #<hash-table equal 2/2 0x1f77857a54f1 ...>)
  #f(compiled-function (&rest args) #<bytecode 0xbbba3f9d7c62c5d>)(#<hash-table equal 2/2 0x1f77857a54f1 ...>)
  #f(compiled-function (result) #<bytecode -0x78cfde9da6d3a2a>)(#<hash-table equal 2/2 0x1f77857a54f1 ...>)
  #f(compiled-function (result) #<bytecode -0x1083fc105979d7d6>)(#<hash-table equal 2/2 0x1f77857a54f1 ...>)
  lsp--parser-on-message(#<hash-table equal 3/3 0x1f77857a54e7 ...> #s(lsp--workspace :ewoc nil :server-capabilities #<hash-table equal 10/10 0x1f778412574e ...> :registered-server-capabilities (#s(lsp--registered-capability :id "ruff-server-watch" :method "workspace/didChangeWatchedFiles" :options #<hash-table equal 1/1 0x1f7785a1c3ed ...>)) :root "/home/marie/parvus/prog/mint/ai/jxai" :client #s(lsp--client :language-id nil :add-on? t :new-connection (:connect #f(compiled-function (filter sentinel name environment-fn workspace) #<bytecode -0x1090a9e16753dac8>) :test? #f(compiled-function () #<bytecode 0x5fb916beec8a634>)) :ignore-regexps nil :ignore-messages nil :notification-handlers #<hash-table equal 0/0 0x1f7785667a80 ...> :request-handlers #<hash-table equal 0/0 0x1f7785667995 ...> :response-handlers #<hash-table eql 0/24 0x1f77856679ab ...> :prefix-function nil :uri-handlers #<hash-table equal 0/0 0x1f77856679b8 ...> :action-handlers #<hash-table equal 0/0 0x1f77856679d6 ...> :action-filter nil :major-modes nil :activation-fn #f(compiled-function (file-name mode) #<bytecode -0x164da5e46142777e>) :priority -2 :server-id ruff :multi-root nil :initialization-options #f(compiled-function () #<bytecode -0x105191dd44d3b3b1>) :semantic-tokens-faces-overrides nil :custom-capabilities nil :library-folders-fn nil :before-file-open-fn nil :initialized-fn nil :remote? nil :completion-in-comments? nil :path->uri-fn nil :uri->path-fn nil :environment-fn nil :after-open-fn nil :async-request-handlers #<hash-table equal 0/0 0x1f77856679cf ...> :download-server-fn nil :download-in-progress? nil :buffers nil :synchronize-sections nil) :host-root nil :proc #<process ruff> :cmd-proc #<process ruff> :buffers (#<buffer test_jxai.py>) :semantic-tokens-faces nil :semantic-tokens-modifier-faces nil :extra-client-capabilities nil :status initialized :metadata #<hash-table equal 0/0 0x1f7785779313 ...> :watches #<hash-table equal 0/0 0x1f7785779305 ...> :workspace-folders nil :last-id 0 :status-string nil :shutdown-action nil :diagnostics #<hash-table equal 1/6 0x1f778577950e ...> :work-done-tokens #<hash-table equal 0/0 0x1f7785779518 ...>))
  #f(compiled-function (msg) #<bytecode -0x1fb3287444bdcc24>)(#<hash-table equal 3/3 0x1f77857a54e7 ...>)
  mapc(#f(compiled-function (msg) #<bytecode -0x1fb3287444bdcc24>) (#<hash-table equal 3/3 0x1f77857a54e7 ...>))
  #f(compiled-function (proc input) #<bytecode -0xf77b50a0a6d6a55>)(#<process ruff> "Content-Length: 1871\15\n\15\n{\"jsonrpc\":\"2.0\",\"id\":114,\"result\":{\"items\":[{\"code\":\"F401\",\"codeDescription\":{\"href\":\"https://docs.astral.sh/ruff/rules/unused-import\"},\"data\":{\"code\":\"F401\",\"edits\":[{\"newText\":\"\",\"range\":{\"end\":{\"character\":0,\"line\":1},\"start\":{\"character\":0,\"line\":0}}}],\"kind\":{\"body\":\"`jax` imported but unused\",\"name\":\"UnusedImport\",\"suggestion\":\"Remove unused import: `jax`\"},\"noqa_edit\":{\"newText\":\"  # noqa: F401\\n\",\"range\":{\"end\":{\"character\":0,\"line\":1},\"start\":{\"character\":10,\"line\":0}}}},\"message\":\"`jax` imported but unused\",\"range\":{\"end\":{\"character\":10,\"line\":0},\"start\":{\"character\":7,\"line\":0}},\"severity\":2,\"source\":\"Ruff\",\"tags\":[1]},{\"code\":\"F401\",\"codeDescription\":{\"href\":\"https://docs.astral.sh/ruff/rules/unused-import\"},\"data\":{\"code\":\"F401\",\"edits\":[{\"newText\":\"\",\"range\":{\"end\":{\"character\":0,\"line\":2},\"start\":{\"character\":0,\"line\":1}}}],\"kind\":{\"body\":\"`jax.numpy` imported but unused\",\"name\":\"UnusedImport\",\"suggestion\":\"Remove unused import: `jax.numpy`\"},\"noqa_edit\":{\"newText\":\"  # noqa: F401\\n\",\"range\":{\"end\":{\"character\":0,\"line\":2},\"start\":{\"character\":28,\"line\":1}}}},\"message\":\"`jax.numpy` imported but unused\",\"range\":{\"end\":{\"character\":28,\"line\":1},\"start\":{\"character\":25,\"line\":1}},\"severity\":2,\"source\":\"Ruff\",\"tags\":[1]},{\"code\":\"F401\",\"codeDescription\":{\"href\":\"https://docs.astral.sh/ruff/rules/unused-import\"},\"data\":{\"code\":\"F401\",\"edits\":[{\"newText\":\"\",\"range\":{\"end\":{\"character\":0,\"line\":4},\"start\":{\"character\":0,\"line\":3}}}],\"kind\":{\"body\":\"`optax` imported but unused\",\"name\":\"UnusedImport\",\"suggestion\":\"Remove unused import: `optax`\"},\"noqa_edit\":{\"newText\":\"  # noqa: F401\\n\",\"range\":{\"end\":{\"character\":0,\"line\":4},\"start\":{\"character\":12,\"line\":3}}}},\"message\":\"`optax` imported but unused\",\"range\":{\"end\":{\"character\":12,\"line\":3},\"start\":{\"character\":7,\"line\":3}},\"severity\":2,\"source\":\"Ruff\",\"tags\":[1]}],\"kind\":\"full\"}}")

Steps to reproduce

See above.

Expected behavior

No error message.

Which Language Server did you use?

lsp-pyright

OS

Linux

Error callstack

Anything else?

No response

[wyuenho]

I'm having this problem as well. I wondering which commit introduced this regression.

[wyuenho]

Looks like 8a266b8 introduced this problem. The add-on key on the ty client makes both pyright and ty running and reporting diagnostics on the same file and same lines.

[kiennq]

The ty server only support diagnostics so it definitely an add-on and not a full-blown python server. I think it's more like eslint language server.
Also, it's weird that the buffer-file-name return nil, does the ty activated on non-file python buffer? LSP also work on file based, thus the assumption is that the file should be always available for LSP server to run. We can probably extend it to buffer without file as well, but that would be an extra functionality (copilot.el is doing that with file:///buffer/... uri).

With that, the intermediate change should be removing the lsp-python-ty from lsp-client-packages, investigate why it's activated on non-file buffer. I suspect there might be bug somewhere as lsp is not activatable in non-file buffer.

[wyuenho]

My buffer is backed by a file already written to disk. AFAIK, there's no other python language server that does not report diagnostics, and there are multiple servers that do type checking. Ty is not remotely the same as eslint, eslint lints, as in checks stylistic and semantic problems. Eslint does not type check, as it can be running on top of say ts-ls. There's no reason to be running ty on top of pyright as you'll be getting multiple reports for the same type error everywhere.

[dzhu]

I encounter this issue without running ty and even in languages/modes besides Python, so it does not have anything to do with ty specifically in any case.

Running (insert-file-contents FN nil nil nil t) in a buffer, with FN replaced by the path to any regular file whose contents are not identical to the buffer's current contents, triggers the issue directly, so I think this is a more local issue and there is not actually any activation in a non-file buffer.

I found something in the Emacs source code that kind of looks like insert-file-contents explicitly setting something like buffer-file-name to nil as a workaround for some other function, which may be the root cause: https://cgit.git.savannah.gnu.org/cgit/emacs.git/tree/src/fileio.c?h=emacs-30.1#n4553. Though I'm not familiar with reading that code, and that statement may possibly only affect the buffer-file-name variable and not the function with the same name. I'm not sure if those are inherently related somehow.

Defining this function as a workaround has been working for me:

;; additionally check buffer-file-truename for path to work around (buffer-file-name) being nil during insert-file-contents, which is used by reformatter
(defun lsp-diagnostics--request-pull-diagnostics (workspace)
  "Request new diagnostics for the current file within WORKSPACE.
This is only executed if the server supports pull diagnostics."
  (when (lsp-feature? "textDocument/diagnostic")
    (let ((path (lsp--fix-path-casing (or (buffer-file-name) (expand-file-name buffer-file-truename)))) (*tmp-workspace* workspace))
      (lsp-request-async "textDocument/diagnostic"
                         (list :textDocument (lsp--text-document-identifier))
                         (-lambda ((&DocumentDiagnosticReport :kind :items?))
                           (lsp-diagnostics--apply-pull-diagnostics *tmp-workspace* path kind items?))
                         :mode 'tick))))

[wyuenho]

@dzhu do you still run into this problem using 1d5f769?

[dzhu]

Yes.

[wyuenho]

Confirmed. It's been around since #4280 landed. Currently only affecting servers that support pull diagnostics, so ty and rust-analyzer etc are affected.