Package management

[jneem]

This is a hacky prototype of package management. I'm pushing it here not because it's useful yet, but to see if it's a direction we want to pursue.

First of all, I think we had discussed trying to piggy-back off npm (or other registries), but their terms (specifically, point 4) seem to prohibit it. So this prototype ditches registries altogether, allowing only git deps and path deps. In particular, we don't have any version resolution yet.

Anyway, what this PR allows is imports like import "file.ncl"@my-org/my-package. For such imports to work, you need to specify a manifest file that lists the "my-org/my-package" package. The manifest files are not auto-detected; you need to explicitly provide them with the --manifest-file option. So for example, given these files:

# foo/package.ncl
{
  name = "example/foo",
  version = "0.1.0",

  dependencies = {
    "example/leaf" = 'Path "../leaf"
    "jneem/ncl-package" = 'Git { url = "https://github.com/jneem/ncl-package.git" },

  },
} | std.package.Manifest

# foo/main.ncl
(import "hi.ncl"@example/foo) ++ ((import "package.ncl")@jneem/ncl-package).version

# leaf/package.ncl
{
  name = "example/leaf",
  version = "0.1.0",
} | std.package.Manifest

# leaf/hi.ncl
"hi"

if you run nickel eval --manifest-file foo/package.ncl foo/main.ncl it will print "hi0.1.0".

The current prototype allows for different copies of the same package, and it properly keeps them separate: if you depend on "example/foo" from one location and one of your dependencies depends on "example/foo" from some other location, import "bar.ncl"@example/foo will be interpreted according to whether it appears in your code or your dependency's code.

[ShalokShalom]

I have already shared this on the Discord with @yannham, hope its fine to bring it in this discussion here also.

This one is Rust based, and self aware about many aspects, that need to be explicitly typed in other package management solutions.

I found it very easy to package, as its files need very little input.
Another rather rare benefit is, that is it available on Windows.

https://github.com/habitat-sh/habitat

[jneem]

Thanks for sharing this! It will be cool if we can piggy-back on their registry instead of building our own. But even if we can, there are definitely some package-management parts that we'll have to build ourselves (like having the interpreter understand imports from packages vs imports from the local filesystem). I'm going to focus on this part first, and look into the registry aspect later.

I did look into their docs a bit and it seems to be focused on running services. Do you know if they have a mode where you can just say "give me a bunch of files (including transitive deps) and put them in a well-known place"?

[ShalokShalom]

Thanks for sharing this! It will be cool if we can piggy-back on their registry instead of building our own. But even if we can, there are definitely some package-management parts that we'll have to build ourselves (like having the interpreter understand imports from packages vs imports from the local filesystem). I'm going to focus on this part first, and look into the registry aspect later.

I did look into their docs a bit and it seems to be focused on running services. Do you know if they have a mode where you can just say "give me a bunch of files (including transitive deps) and put them in a well-known place"?

I really recommend that you ask these questions the Habitat folks directly. I did use it a couple of years ago, and dont remember most.

[jneem]

The macOS build is failing because of this, which no one seems to know how to solve...

[github-actions]

Bencher Report

Branch	npm
Testbed	ubuntu-latest

Click to view all benchmark results

Benchmark	Latency	nanoseconds (ns)
fibonacci 10	📈 view plot 🚷 view threshold	488,550.00
foldl arrays 50	📈 view plot 🚷 view threshold	1,665,200.00
foldl arrays 500	📈 view plot 🚷 view threshold	6,691,500.00
foldr strings 50	📈 view plot 🚷 view threshold	7,108,800.00
foldr strings 500	📈 view plot 🚷 view threshold	61,236,000.00
generate normal 250	📈 view plot 🚷 view threshold	46,077,000.00
generate normal 50	📈 view plot 🚷 view threshold	1,955,800.00
generate normal unchecked 1000	📈 view plot 🚷 view threshold	3,329,200.00
generate normal unchecked 200	📈 view plot 🚷 view threshold	760,330.00
pidigits 100	📈 view plot 🚷 view threshold	3,218,600.00
pipe normal 20	📈 view plot 🚷 view threshold	1,481,800.00
pipe normal 200	📈 view plot 🚷 view threshold	9,882,900.00
product 30	📈 view plot 🚷 view threshold	832,670.00
scalar 10	📈 view plot 🚷 view threshold	1,518,700.00
sum 30	📈 view plot 🚷 view threshold	822,320.00

🐰 View full continuous benchmarking report in Bencher