cm-service

This is the Rubin Observatory data processing campaign management ReST service. cm-service is developed with FastAPI and Safir. Learn more at https://cm-service.lsst.io.

Developer Quick Start

You can build and run cm-service on any system which has Python 3.11 or greater, uv, make, and Docker w/ the Docker Compose V2 CLI plugin (this includes, in particular, recent MacOS with Docker Desktop). Proceed as follows:

• Ensure uv is installed (via Homebrew (macOS), pipx, or a preferred alternative method).

  • You may run make uv to bootstrap uv in your environment.
  • Suggested: set UV_PYTHON_PREFERENCE=only-managed to prevent non-uv Pythons from being used.
  • USDF: set UV_NATIVE_TLS=true for compatibility with the Squid proxy.

• Install Python virtual environment, dependencies and activate pre-commit hooks with make init.

  • uv will automatically install any needed Python version.
  • Note: the ctrl-bps-htcondor plugin is not available on macOS and will not be installed on this platform.

• Ensure the virtual environment is activated before interacting with other make targets:

  • source .venv/bin/activate

• Spin up a debug instance of the service running in the foreground of the current shell with make run. This will launch a subsidiary Postgres instance locally in a Docker container via Docker Compose. The foreground debug service instance will log to stdout and will dynamically reload as you edit and save source files.

  • You may also choose to run both the service/worker and the database in Docker by running docker compose --profile full up; in particular this will exercise the Docker build process.

  • You may choose to (re)build the service container with docker compose build [--no-cache] cmservice to build, but not start, the service container (with --no-cache invalidating the build cache if needed).

• Access the monitoring web application at http://localhost:8080/web_app/campaigns/

• Exit your debug instance with ^C. The subsidiary Postgres container launched under Docker Compose will remain active, and will be re-used on any subsequent make run.

• Shut down the subsidiary Postgres container if/when desired with docker compose down. Database state will be maintained in local Docker volumes and re-used on the next run. If you wish to clear the database state volumes as well to start completely fresh, do docker compose down -v.

Additional developer conveniences:

• Browse the integrated online ReST API documentation while the service is running at http://localhost:8080/docs.

• Run the pytest test suite at any point with make test. This will launch subsidiary containers if necessary via Docker Compose, or will make use of any that may already be running. Tests are performed in their own Postgres schema, so as not interfere with debugging state in a running debug service instance. (You may run docker compose down -v to clean up completely between runs.)

• To run the playwright tests, add --run-playwright to the pytest arguments

• Run the pre-commit hooks to lint and reformat your code via make lint.

• Run the mypy static type hint checker with make typing.

• Connect the psql command line client directly to the subsidiary Postgres for direct database inspection or manipulation via make psql.

Project Management

• uv add <dependency> will add a dependency to the project; uv add --group dev ... will mark it as a development dependency.

• make update will cause uv to upgrade all available dependencies within the constraints set for the dependency in the pyproject.toml file.

• uv run <cmd> will execute <cmd> within the project environment, which is useful for cases where the virtual environment is not activated.

• make clean will remove the virtual environment and allow make init to create it from scratch. The lock file is not removed by this target.

• The lock file can be regenerated by removing (or renaming) the uv.lock file and running make uv.lock or the direct command uv lock.

• uv uses a global package cache for speed and efficiency; you manage this cache with uv cache prune or uv cache clean for the nuclear option.