[Questions] Generic server terminating - "badmatch,{error,eacces}" error from rabbit_classic_queue_store_v2 #14094
-
Community Support Policy
RabbitMQ version used4.1.0 Erlang version used27.3.x Operating system (distribution) usedWindows Server 2016 How is RabbitMQ deployed?Windows installer rabbitmq-diagnostics status outputSee https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics Logs from node 1 (with sensitive values edited out)See https://www.rabbitmq.com/docs/logging to learn how to collect logs Logs from node 2 (if applicable, with sensitive values edited out)No response Logs from node 3 (if applicable, with sensitive values edited out)No response rabbitmq.confI do not have one. Steps to deploy RabbitMQ cluster
Steps to reproduce the behavior in questionI am unsure what causes this behavior. It happens intermittantly. Other servers with the exact same RabbitMQ/Erlang combination do not have this problem. advanced.configSee https://www.rabbitmq.com/docs/configure#config-location to learn how to find advanced.config file location Application codeNo response Kubernetes deployment fileNo response What problem are you trying to solve?I work at a software company on a on-premise solution that uses RabbitMQ as a message broker between different services on a application server. After updating to RabbitMQ 4.1.0, Erlang 27.3.3, some queues on one specific customer server crash, drop messages and restart. This can happen to any queue experiencing relatively high traffic. When it does, we get the error message in the RabbitMQ logs that I have shared above. Occassionally this leads to a complete failure of RabbitMQ, where no messages get published up at all anymore. When this happens, the RabbitMQ service is still running and we can still reach Rabbit's localhost management console. After restarting the RabbitMQ service, everything runs normally for a while before the whole issue starts over again. I have spent some time trying to solve this, but so far I am unable to find a clear cause or solution. I've tried the following things:
I want to note that this only happens on one specific customers' server. I guess this is probably some kind of permission issue regarding the buffer files (?), but am unsure how to reproduce the issue or how I could attempt to solve it. If anyone could give some advice on what to do, that'd be much appreciated. The error message points to the following line of code: |
Beta Was this translation helpful? Give feedback.
Answered by
michaelklishin
Jul 16, 2025
Replies: 6 comments 32 replies
-
|
@Donderstal this is a known Windows-specific error: sometimes Windows returns We will see in the next patch release or two. |
Beta Was this translation helpful? Give feedback.
6 replies
-
|
This error is for classic queues not quorum queues -- perhaps @lhoguin has some ideas. I've never seen |
Beta Was this translation helpful? Give feedback.
-
|
Using |
Beta Was this translation helpful? Give feedback.
-
|
Just now I reinstalled RabbitMq to the problematic server. If the problem still persists, I wil share the file/folder structure and and a ProcessMonitor trace as you requested. If so I will update you later today. |
Beta Was this translation helpful? Give feedback.
-
|
@Donderstal I doubt that reinstalling RabbitMQ will resolve anything. This is a Windows file handle API behavior that's different from other OSes. It could also be security software side effects, so if you run any on the same hosts, please whitelist all RabbitMQ processes, data directories, and so on. Or even disable the anti-virus tools, if your nodes are isolated from the public Internet (as they should be). |
Beta Was this translation helpful? Give feedback.
-
|
@michaelklishin I've contacted the sysadmin of this customers' network and they have guaranteed me that all RabbitMQ processes are whitelisted. The nodes are also isolated from public internet. Both were already the case before I made a post here. I just figured I could try a reinstall, since the problem is quite troublesome for this customer and I didn't have any better ideas right now. The reinstall seems to have sharply decreased the frequency of the errors to only once today, but this could just be a coincidence. I've used ProcessMonitor to capture the file system reads/writes this afternoon. I've limited it to just the erl.exe while monitoring. During my monitoring today, the error didn't occur. I can share the Logfile.PML and the RabbitMq file/folder structure, but I figured it's probably useless to you all because it doesn't capture the moment of the Monday I will try running the ProcessMonitor all day until I successfully capture the error. When I do I will share it here. Thank you all very much for your help. |
Beta Was this translation helpful? Give feedback.
-
|
This error occured just now while I was running Process Monitor. This is the error in the RabbitMq logs: I've added the ProcessMonitor logfile as .CSV as attachment. I also have a full .PML file available but I can't add .PML files into a github post. I could email it if you'd like. |
Beta Was this translation helpful? Give feedback.
15 replies
-
|
@Donderstal You can try an alpha build (of future 4.2.0) available at https://github.com/rabbitmq/server-packages/releases/tag/alphas.1750838043824 - upgrading should be smooth but make sure to backup your data. |
Beta Was this translation helpful? Give feedback.
-
|
We can also provide a |
Beta Was this translation helpful? Give feedback.
-
|
That is great news guys, thank you very much! If it's not too much trouble I'd prefer a 4.1.2 preview. I'm a bit anxious to deploy an alpha build to a customers' production environment. |
Beta Was this translation helpful? Give feedback.
-
|
#14131 is merged, I will share a link to a |
Beta Was this translation helpful? Give feedback.
-
|
@Donderstal this |
Beta Was this translation helpful? Give feedback.
-
|
@michaelklishin @lhoguin |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
When we try with the Alpha we get it to fail on index instead |
Beta Was this translation helpful? Give feedback.
7 replies
-
|
You can try an alpha build of the next 4.1.x release at https://github.com/rabbitmq/server-packages/releases/tag/alphas.1751455129125 |
Beta Was this translation helpful? Give feedback.
-
|
RabbitMQ 4.1.2-alpha.c529a212 No crash yet, we will continue our testing |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
Anyway, we were unable to reproduce the problem with the two servers we tested on after the fix, without we could trigger it within an hour |
Beta Was this translation helpful? Give feedback.
-
|
@Erior thank you for confirming. I figured I should select the answer that links to a patch release with the relevant fixes. |
Beta Was this translation helpful? Give feedback.
-
|
I installed the alpha version on Tuesday morning 07:30, CET. Since then, we've had the error once on this customers machine. As far as I know this has not resulted in a loss of messages. I will keep an eye on the rabbitMQ logs and will give you guys an update next week to see if the error frequency increases. |
Beta Was this translation helpful? Give feedback.
4 replies
-
|
There can be more places, invoked significantly less frequently, that need to have the same retry added. Good to know that there's been an improvement in any case. |
Beta Was this translation helpful? Give feedback.
-
|
If the stacktrace is the same then the code probably doesn't wait long enough. Hard to figure out how long it should wait though. |
Beta Was this translation helpful? Give feedback.
-
|
@lhoguin The stacktrace is exactly the same. But as I said things seem to be running fine so far except for that single error |
Beta Was this translation helpful? Give feedback.
-
|
As a final update. Since my last comment here, we've had this error occur just twice. The frequency has not increased. As far as I am concerned the problem is solved. Thanks once again for your help. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks to @tomerrt, we have learned that this filesystem API behavior is not only Windows-specific, it is specific to a certain set of Windows versions, and the newest versions While our current workaround helps, it must cover all file deletions, which is a fair number of places even in the CQ storage layer alone. Adopting a newer Windows version can be a viable option that would help those running RabbitMQ 3.13.x or any 4.x version before |
Beta Was this translation helpful? Give feedback.
0 replies
Answer selected by
michaelklishin
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Thanks to @tomerrt, we have learned that this filesystem API behavior is not only Windows-specific, it is specific to a certain set of Windows versions, and the newest versions
should not be affected.
While our current workaround helps, it must cover all file deletions, which is a fair number of places even in the CQ storage layer alone. Adopting a newer Windows version can be a viable option that would help those running RabbitMQ 3.13.x or any 4.x version before
4.1.2where the first slew of workarounds shipped.