mirror-nixos-branch: add header for "Lockable HTTP Tarball Protocol" (Flakes)

[emilylange]

This is part of another PR over in NixOS/infra (NixOS/infra#562) that implements the configuration change needed for Fastly and allows one to use

{
  inputs = {
    nixpkgs.url = "https://channels.nixos.org/nixos-unstable-small/nixexprs.tar.xz";
  };
}

and have it locked in flake.lock accordingly.

This has multiple advantages over using something like github:NixOS/nixpkgs/nixos-unstable-small.

For example, channels.nixos.org and releases.nixos.org are accessible on not just IPv4 but also IPv6, unlike GitHub.

Furthermore, the tarballs have a higher compression-ratio than those served by GitHub and thus require less bandwidth to download.

Additionally, nixexprs.tar.xz contains the precomputed programs.sqlite which enables the use of the very lightweight programs.command-not-found with Flakes.

Ref: https://github.com/NixOS/nix/blob/61f49de7ae0b3899abdcc102832523153dd40d35/doc/manual/source/protocols/tarball-fetcher.md

[Mic92]

This now has been deployed. Now we need to wait for a nixpkgs channel bump.

[emilylange]

Turns out, the channel script works, it's just that the s3 website function eats all the metadata in a redirect instead of passing it along.

# curl https://nix-channels.s3.amazonaws.com/nixos-unstable/nixexprs.tar.xz -i
HTTP/1.1 200 OK
x-amz-id-2: jOUsd0Df6uQBHaetoBvVGiJSwkb+baHQajBSO2aX3lDnk/5NZvk5evhKZpesPVfqhYqUwjy2wgEoPQJeVatIxnuZBMtYk9xO
x-amz-request-id: VGPKQJX04DFTMDYN
Date: Wed, 12 Mar 2025 18:00:58 GMT
Last-Modified: Mon, 10 Mar 2025 17:34:14 GMT
ETag: "d41d8cd98f00b204e9800998ecf8427e"
x-amz-server-side-encryption: AES256
x-amz-website-redirect-location: https://releases.nixos.org/nixos/unstable/nixos-25.05beta764837.e3e32b642a31/nixexprs.tar.xz
Cache-Control: maxage=600,stale-while-revalidate=1800,public
x-amz-meta-link: <https://releases.nixos.org/nixos/unstable/nixos-25.05beta764837.e3e32b642a31/nixexprs.tar.xz?rev=e3e32b642a31e6714ec1b712de8c91a3352ce7e1>; rel="immutable"
Accept-Ranges: bytes
Content-Type: binary/octet-stream
Content-Length: 0
Server: AmazonS3

vs

# curl http://nix-channels.s3-website-us-east-1.amazonaws.com/nixos-unstable/nixexprs.tar.xz -i
HTTP/1.1 301 Moved Permanently
x-amz-id-2: S+pHEkEgXLjaM3dNhJDUD/i1+5WsLYtSaAHKbU/FbFXGJzEm03tAT1slrPBm4tptNARp7OWx/ZfJkC/in3cMO+1eyMFeuT/V
x-amz-request-id: QCVEJDSJ699AZ5K1
Date: Wed, 12 Mar 2025 18:02:08 GMT
Location: https://releases.nixos.org/nixos/unstable/nixos-25.05beta764837.e3e32b642a31/nixexprs.tar.xz
Content-Length: 0
Server: AmazonS3

which then causes the Fastly fallback to kick in:

# curl https://channels.nixos.org/nixos-unstable/nixexprs.tar.xz -i
HTTP/2 302 
x-amz-id-2: VNbdPCnqSHshL9JP502CasrQeFWVkRNF0T2IstQcGUlduQ3bqhLZSYEu+J9XMWKzf1P4YiS1EZE=
x-amz-request-id: ZZF8253TSB4X9117
location: https://releases.nixos.org/nixos/unstable/nixos-25.05beta764837.e3e32b642a31/nixexprs.tar.xz
server: AmazonS3
link: <https://releases.nixos.org/nixos/unstable/nixos-25.05beta764837.e3e32b642a31/nixexprs.tar.xz>; rel="immutable"
accept-ranges: bytes
via: 1.1 varnish, 1.1 varnish
date: Wed, 12 Mar 2025 18:04:08 GMT
x-served-by: cache-iad-kcgs7200073-IAD, cache-fra-eddf8230075-FRA
x-cache: MISS, MISS
x-cache-hits: 0, 0
content-length: 0

Will have to think about this for a while.

One way to further reduce complexity and unblock it immediately would be to just add ?rev= to the redirect itself (x-amz-website-redirect-location) and hope it does not break downstream consumers.

Also, it feels very risky having to use an unencrypted http link between Fastly and the s3 website because of yet another s3 limitation. But I suppose this is beyond the scope of this PR here.

[emilylange]

Yeah hmmm, I believe just adding the flake metadata to x-amz-website-redirect-location and then having Fastly prune the query string entirely is the best way to reapproach this. It greatly simplifies every aspect of this feature.

Would you be up to another round of PRs? @Mic92

diff --git a/mirror-nixos-branch.pl b/mirror-nixos-branch.pl
index b4653d5..0b85b0d 100755
--- a/mirror-nixos-branch.pl
+++ b/mirror-nixos-branch.pl
@@ -360,7 +360,7 @@ sub redirect {
 
 # Update channels on channels.nixos.org.
 redirect($channelName, $releasePrefix);
-redirect("$channelName/nixexprs.tar.xz", "$releasePrefix/nixexprs.tar.xz", 1);
+redirect("$channelName/nixexprs.tar.xz", "$releasePrefix/nixexprs.tar.xz?rev=$rev");
 redirect("$channelName/git-revision", "$releasePrefix/git-revision");
 redirect("$channelName/packages.json.br", "$releasePrefix/packages.json.br");
 redirect("$channelName/store-paths.xz", "$releasePrefix/store-paths.xz");

this also allows us to remove the logic added to redirect again (essentially undo this PR):

diff --git a/mirror-nixos-branch.pl b/mirror-nixos-branch.pl
index 0b85b0d..75f28be 100755
--- a/mirror-nixos-branch.pl
+++ b/mirror-nixos-branch.pl
@@ -342,19 +342,10 @@ run("git push origin $rev:refs/heads/$channelName >&2");
 my $cache_control = "maxage=600,stale-while-revalidate=1800,public";
 
 sub redirect {
-    my ($from, $to, $immutable_flake_tarball) = @_;
+    my ($from, $to) = @_;
     $to = "https://releases.nixos.org/" . $to;
     print STDERR "redirect $from -> $to\n";
-    my $object_metadata = { "x-amz-website-redirect-location" => $to, "cache-control" => $cache_control };
-    if ($immutable_flake_tarball // 0) {
-        # Optionally sets a header for the "Lockable HTTP Tarball Protocol".
-        # See <https://github.com/NixOS/nix/blob/61f49de7ae0b3899abdcc102832523153dd40d35/doc/manual/source/protocols/tarball-fetcher.md>.
-        # AWS S3 does not not allow setting the "Link" header directly.
-        # Instead, we prefix the header with "x-amz-meta" and let Fastly
-        # rename the header for us.
-        $object_metadata->{'x-amz-meta-link'} = "<$to?rev=$rev>; rel=\"immutable\"";
-    }
-    $bucketChannels->add_key($from, "", $object_metadata)
+    $bucketChannels->add_key($from, "", { "x-amz-website-redirect-location" => $to, "cache-control" => $cache_control })
         or die $bucketChannels->err . ": " . $bucketChannels->errstr;
 }

and then over in NixOS/infra, we also simplify the logic and remove some vendor specific behavior:

diff --git a/terraform/channels.tf b/terraform/channels.tf
index 3c8c7cb..2920fbf 100644
--- a/terraform/channels.tf
+++ b/terraform/channels.tf
@@ -238,14 +238,11 @@ resource "fastly_service_vcl" "channels" {
         set beresp.grace = 0s;
         set beresp.cacheable = false;
         if (req.backend.is_origin && std.suffixof(bereq.url, "/nixexprs.tar.xz")) {
-          # rename prepared link header if available
-          if (beresp.http.x-amz-meta-link) {
-            set beresp.http.link = beresp.http.x-amz-meta-link;
-            unset beresp.http.x-amz-meta-link;
-          # otherwise, use fallback that contains no flake attributes (e.g. rev)
-          } else {
-            set beresp.http.link = "<" + beresp.http.location + {">; rel="immutable""};
-          }
+          # pass redirect location into special flake "immutable tarball" header
+          set beresp.http.link = "<" + beresp.http.location + {">; rel="immutable""};
+          # clear query string from redirect destination as precaution in case
+          # legacy consumers can't handle flake attributes like "?rev=" in it
+          set beresp.http.location = querystring.remove(beresp.http.location);
         }
         return (pass);
       }

[Mic92]

Looks good?

#78
NixOS/infra#584