Automatically test built image in CI

[yuvipanda]

#2950 could've been caught fairly easily in #2933 if we had some simple unit tests that check that otter imports/command line works properly. We should add that as part of our infrastructure to make it more resilient.

Tasks to be Completed:

• All hub images are split into their own repository. Following Stat159 hub's precedence!
• Test cases are developed for each image - Check whether the Jupyter/R Studio notebooks exist and install a package in the notebook
• @yuvipanda to talk more about image building during a session!

[balajialg]

@yuvipanda Scope this for spring 2022?

[yuvipanda]

sure.

[balajialg]

Getting to a semester-level roadmap something on the lines of Github's roadmap will be an interesting milestone!

[ryanlovett]

@yuvipanda Are you thinking we can just start up a singleuser container within CI, start up jupyter inside it, then run those imports?

Would we do hub tests as well? Is this what would be needed to automate either as part of CI?

Client Side

• create a service account user for each hub
• generate a token for it on each hub and save it to a secret
• use that secret token in CI to login to the hub, start a server, then run various tests

Hub Side

• start up a test hub
• start up a test user user server
• run various client tests

[balajialg]

*Listing the exhaustive manual test cases that can be checked when a major PR gets merged

1. Users can successfully authenticate using Gmail/bcourses account to login to the hub
2. Users can successfully check whether Jupyter notebook, lab, retro lab, and R studio start-up when clicked
3. Users can successfully install R and Python Package in the respective interfaces
4. Users can successfully access files stored previously in the home directory or the shared read-write directories
5. Users can successfully upload/download a file to/from the tree
6. Users can successfully kill and restart a server
7. [For admin users] Admin users can successfully access nonadmin users server
8. Users can successfully install an extension in JupyterLab and RetroLab

We can narrow down from this list the critical test cases that can be automated and can also highlight the status of hub!

[balajialg]

Proof of concept for per image tests is already implemented by Yuvi here - jupyterhub/repo2docker-action#83

[balajialg]

@yuvipanda to develop a proof of concept test suite for Stat 159 hub!