nixos-unstable channel is stuck

[vcunat]

The nixos-unstable channel isn't updating although the usual conditions have been met for many hours. It's not just for the current evaluation I have in mind but also the previous one.

[vcunat]

Ah, I recalled grafana... so the corresponding task is shown as failing, but AFAIK for these cases there are no logs or further information publicly shown and very few people have access to it. /cc @NixOS/infra I guess?

[lukegb]

Yeah, sorry. We've had a discussion about this on IRC. The pertinent bits are:

• Nix unstable had a bug where the Brotli compression would sometimes generate corrupt output for large files.
• This was fixed 10 days ago and the Hydra machines updated.
• ...but the Hydra queue runner was not restarted.
• As a result, the .ls NAR manifest files generated as part of uploading to the cache were sometimes being corrupted
• We finally managed to hit a case where we generated a file that was sufficiently broken that it broke generate-programs-index and caused it to segfault (inside nlohmann-json, no less)

The Hydra queue runner's been updated and I submitted a series of PRs (#121534, #121519) to change the problematic package's hash. Once https://hydra.nixos.org/build/142486441 (and its eval) completes we should be golden.

[domenkozar]

I've opened NixOS/nixos-channel-scripts#48 in hopes of catching this earlier and having a better insight into the channel process.

[vcunat]

Oh, thanks for the references.

I had seen some bits around compression but considered them resolved, and I quickly skimmed the last two days of #nixos-dev IRC and some recently opened GitHub issues on a few projects... anyway, it's nice to have a link from status.nixos.org which got achieved by opening this issue.

[lukegb]

nixos-channel-scripts was sat down and given a stern talking to. nixos-unstable has now advanced.