Allow to install extra packages from extra indexes from jupyter-lite.json

[Carreau]

This is in progress, but the idea is to be able to point to dev wheels on rtd or similar without having to make a custom pyidide distribution.

For example:

// jupyter-lite.json
{
  "jupyter-config-data": {
    "litePluginSettings": {
       "extraPackagesAndIndexes": [
          {
            "indexes": [
              "https://cors.carreau.workers.dev/scientific-python-nightly-wheels/simple",
              "https://pypi.org/simple"
            ],
            "package": "xarray"
          }
        ]
      }
    }
  }

Package could also be a simple url to a whl (I need to check how it works if it's a local URL), and indexes can be ommited.

WIP as I'm stuggling to have the typescript build process work.

[github-actions]

👈 Try it on ReadTheDocs

[Carreau]

cc @agriyakhetarpal

[Carreau]

make that Array<{ package: string[]; ... }>;

[Carreau]

same here.

[Carreau]

I think lower in initKernel the

    for (const pkgName of toLoad) {
      if (!preloaded.includes(pkgName)) {
        scriptLines.push(`await piplite.install('${pkgName}', keep_going=True)`);
      }
    }

Could be modified to be a single call to piplite.install(list), instead of repeated install calls.

[Carreau]

https://cors.carreau.workers.dev/scientific-python-nightly-wheels/simple should not be used in production of course.

[Carreau]

In both cases, if pkg ends with whl, prepend baseurl,
and from testing, check the index urls do not end in /, otherwise it redirects and there are CORS issues.

[Carreau]

and cc @gabalafou

[bollwyvl]

Hmmmm, while I take no issue with the indexes (indeed, maybe a pipliteInstallDefaultOptions would allow for overriding all of the non-requirements micropip.install kwargs), we've been trying not to add new sources of run-time unpredictability before a user can make their first input and get useful feedback in the UI, which really should be all the kernel is on the hook for... but yeah, this is the browser, and ain't nothing easy.

As of 0.4.7, upstream's pyodide-lock.json is missing only comm, as ipython (and its 100k of vendored tests) is good to go. Pulling in a single wheel by URL might bring in many other, unpredictable wheels, which could (and it sounds like in this case, with nightly) expect to be different between two different site visits, and very possible breaking when installed against other things, and stirred together with dependencies from PyPI.

So my feeling would be: we could add hooks to allow another extension to load extra packages, run dynamic startup code, configure environment variables, write files on disk, make telemetry posts, etc. and all the other things that folk have asked for, but well after the core plumbing has loaded so that when errors happen, those messages appear, with attribution, somewhere.

[Carreau]

Sure that would be fine with me, what Im really hopping is to allow most of the Python Ecosystem to be able to get interactive documentation without having to rebuild a pyodide distribution.

I'm fine if extra dependencies and breakage are up to the maintainers' using this configuration option.

I'm also happy to move this to another place in the launching process.

[Carreau]

crosslinking to #60

[Carreau]

Would you agree if this were change to only look at locally available wheels ?

Basically when using jupyter lite build --piplite-wheels; I don't think the users should need an extra install step;

For me including wheels in a jupyterlite build should make them immediately available to the end user, or it feels weird. I'm ok if wheels needs to be referred to in two locations.

[agriyakhetarpal]

Basically when using jupyter lite build --piplite-wheels; I don't think the users should need an extra install step;

For me including wheels in a jupyterlite build should make them immediately available to the end user, or it feels weird. I'm ok if wheels needs to be referred to in two locations.

I would agree with this, as this is one of the points that I've made in sympy/live#26 and it would be nicer to see that micropip/piplite consider pre-installing only if they find all wheels locally (say, if I were to ship SymPy and all its dependencies – mpmath, together) and look at the local index at a priority over Pyodide.

[gabalafou]

As promised, I gave this a look and it looks good to me.

It seems that your comment about getting TypeScript to work was based on a previous commit and that you've resolved the issues with the TypeScript compiler, is that correct?

[Carreau]

So my feeling would be: we could add hooks to allow another extension to load extra packages, run dynamic startup code, configure environment variables, write files on disk, make telemetry posts, etc. and all the other things that folk have asked for, but well after the core plumbing has loaded so that when errors happen, those messages appear, with attribution, somewhere.

It seem to be that this asking way more than just having packages ready to use. Right now usually all the file and setup are ready, the only step missing is having pre-installed package. I understand the need to not wanting to download and extra URL, but there is already the --piplite-wheels so we only need to make them available at import.

Now if you really want a different extension fine, we can do tham but assuming I provide the necessary patches for install(...) to only look into the lock file and all.json, would that be accepted here ?

[bollwyvl]

There is a big difference between blocking the kernel being available, with proper error reporting, on:

• using js.loadPackage to install a known-good wheel, which comes from the same host as pyodide.js, and is guaranteed to have working dependencies
• use this random package, maybe without a pin, that depends on other packages, maybe without pins, downloaded from somewhere on the internet, that may have changed since the site was built

There was a previously-rejected PR (don't see it here, maybe was before the SplitLite) that did the scut work to make that happen (discovering top-level names, etc.), by patching the as-loaded pyode-lock.json, but still wasn't a solver, so could definitely get things wrong, and added extra dependencies and complexity.

For those reasons, I've been pursuing upstream means to make this more predictable, such as:

• clobber pyodide packages with pip packages in freeze pyodide/micropip#172
• Add support for constraints, install wheels before pyodide packages pyodide/micropip#181

...and downstream means to make this work in pyodide-kernel:

• https://github.com/deathbeds/jupyterlite-pyodide-lock

Maybe once all of that works, every time, without npm/nodejs or a full browser at build time, we can consider making something that isn't too bad in this repo.