Document registries and packaging process (#962)

Resolves #734 and #901.

---------

Signed-off-by: maciektr <[email protected]>
Co-authored-by: maciektr <[email protected]>
Co-authored-by: Marek Kaput <[email protected]>
Co-authored-by: Maksim Zdobnikau <[email protected]>
Co-authored-by: Maksim Zdobnikau <[email protected]>
Co-authored-by: Wojciech Szymczyk <[email protected]>

website/.vitepress/config.mjs

@@ -75,7 +75,12 @@ const sidebar = {
     },
     {
       text: "Registries",
-      items: [p("Package tarball", "/docs/registries/package-tarball")],
+      items: [
+        p("Overview", "/docs/registries/overview"),
+        p("Publishing", "/docs/registries/publishing"),
+        p("Package tarball", "/docs/registries/package-tarball"),
+        p("Custom registry", "/docs/registries/custom-registry"),
+      ],
     },
     {
       text: "Appendices",

website/docs/cheatsheet.md

@@ -61,6 +61,13 @@ starknet = "{{ rel.stable.starknetPackageVersionReq }}"
 
 ### In manifest
 
+Add dependency hosted on the official [scarbs.xyz](https://scarbs.xyz) registry:
+
+```toml
+[dependencies]
+alexandria_math = "0.1.0"
+```
+
 Add dependency hosted on a Git repository:
 
 ```toml
@@ -89,6 +96,12 @@ alexandria_math = { path = "../path-to-alexandria-checkout/alexandria" }
 
 ### Via `scarb add`
 
+Add dependency from the official [scarbs.xyz](https://scarbs.xyz) registry:
+
+```shell
+scarb add [email protected]
+```
+
 Add dependency hosted on a Git repository:
 
 ```shell

website/docs/guides/dependencies.md

@@ -3,9 +3,11 @@
 Scarb manages dependencies by cloning packages from their Git repositories.
 To add a dependency, simply declare it in your `Scarb.toml`.
 
-> [!NOTE]
-> Using Git repositories as a foundation for package management is not an ideal
-> approach. Therefore, we plan to create a proper package registry in long term.
+> [!WARNING]
+> Using Git repositories as a foundation for package management is not the recommended approach anymore.
+> Instead, [registries](../registries/overview.md) are now the primary way to manage dependencies.
+> This guide will be updated to reflect the new approach after official registry is out of beta and considered stable.
+> For details on how to specify dependencies from the official registry, see [here](../reference/specifying-dependencies#specifying-dependencies-from-official-registry).
 
 ## Adding a dependency
 

website/docs/reference/specifying-dependencies.md

@@ -1,6 +1,35 @@
 # Specifying Dependencies
 
-Your packages can depend on other libraries from Git repositories or subdirectories on your local file system.
+Your packages can depend on other libraries from registries, Git repositories, or subdirectories on your local file system.
+
+## Specifying dependencies from registries
+
+Scarb supports the official cairo package registry [scarbs.xyz](https://scarbs.xyz) by default.
+You can see the list of available packages by visiting the website.
+To depend on a package located in the registry, you need to specify the package name and the version requirement:
+
+```toml
+[dependencies]
+alexandria_math = "0.1.0"
+```
+
+Alternatively, you can specify the version requirement with the `version` key:
+
+```toml
+[dependencies]
+alexandria_math = { version = "0.1.0" }
+```
+
+The two formats above are equivalent.
+
+Unlike other dependency types, it is required to specify the version requirement for packages from the registry.
+
+To use a custom, non-official registry you need to specify its url address as well:
+
+```toml
+[dependencies]
+alexandria_math = { registry = "https://example.com/", version = "0.1.0" }
+```
 
 ## Specifying dependencies from Git repositories
 
@@ -53,7 +82,7 @@ In order to add development dependency, specify it under `[dev-dependencies]` se
 
 ```toml
 [dev-dependencies]
-alexandria_math = { git = "https://github.com/keep-starknet-strange/alexandria.git" }
+alexandria_math = "0.1.0"
 ```
 
 Development dependencies are not used when compiling a package for building, but are used for compiling tests.

website/docs/registries/custom-registry.md

@@ -0,0 +1,84 @@
+# Hosting a registry
+
+Although Scarb uses the official [scarbs.xyz](https://scarbs.xyz) registry by default, we provide you with the option to host your own.
+You just need to implement an interface imposed by Scarb.
+
+## Basic package registry
+
+The most basic registry needs to serve three types of files over HTTP: index file, package index info and package archives.
+That means a simple file server can be used as a package registry.
+
+### Registry index file
+
+A file that contains all the info about various endpoints of the registry that Scarb needs to be able to operate over.
+
+The index file has a defined structure:
+
+```json
+{
+  "version": 1,
+  "api": "https://your.domain.com/api/v1",
+  "dl": "https://your.domain.com/api/v1/dl/{package}/{version}",
+  "index": "https://your.domain.com/api/v1/index/{prefix}/{package}.json"
+}
+```
+
+- `version` - value reserved for versioning the registry interface.
+- `api` - a URL of your registry API.
+  Currently, isn't used at all, will be used for things like uploading or yanking a package.
+- `dl` - a download template URL.
+  You can use `{package}` and `{version}` to construct a template that Scarb will populate with the name and version of the package that it needs to download.
+  In case of a simple server it could look like `https://your.registry.com/{package}-{version}.tar.zst`.
+  The request to that URL must return a package `.tar.zst` archive created by the `scarb package` command.
+- `index` - a URL template that functions like the one in `dl` but points to JSON files with index info about specific packages.
+  It takes a `{package}` parameter, which is the package name, and a `{prefix}` value.
+
+  This prefix is useful when you want to organize a file structure into a deeper but narrower one.
+  This is the structure `scarb publish` creates when you use it with a local registry.
+  Essentially, for a package with a name of 4 characters or longer (e.g. `foobar`), the prefix will be `/fo/ob/`.
+  For 3 characters, it will be `/3/f`, and for 2 or 1 character(s), just `/2` and `/1`, respectively.
+
+### Package index file
+
+When Scarb needs to figure out which version of a package it needs to fetch in order to resolve dependency requirements, it needs to know what versions are available.
+That's what package index files are used for.
+They contain information about package versions present in the registry, along with data about their dependencies and checksums needed to verify that the package hasn't been tampered with.
+
+A structure of an example `foo` package index file looks like this:
+
+```json
+[
+  {
+    "v": "0.1.0",
+    "deps": [],
+    "cksum": "sha256:6607a3b860f35f55738360ff55917642282d772423e8120a013b479ddb9e3f89"
+  },
+  {
+    "v": "0.1.1",
+    "deps": [
+      {
+        "name": "bar",
+        "req": "^0.1.3"
+      }
+    ],
+    "cksum": "sha256:5917642282d772423e8120a013b4796607a3b860f35f55738360ff5ddb9e3f89"
+  }
+]
+```
+
+As you can see, it is a JSON array with each entry representing a version of the `foo` package available in the registry.
+An entry consists of the `v` key that describes the version, a `deps` array describing each version's dependency requirements, and a `cksum`, which is a `sha256` hash used to verify integrity.
+
+### Package archive
+
+The last type of files that needs to be served are the package archives.
+These are the outputs of the `scarb package` command, as described in the [Packaging](./publishing) section.
+
+## Using custom registry
+
+To use a custom registry to download a specific dependency, you need to add a `registry` key to the entry.
+It needs to point to a URL that returns the registry index file.
+
+```toml
+foo = { version = "0.1.3", registry = "https://custom.registry/index" }
+```

website/docs/registries/overview.md

@@ -0,0 +1,27 @@
+# Overview
+
+Scarb is not only a Cairo build toolchain but also a package manager.
+It uses registries to store and manage packages.
+While Scarb supports alternative ways to host packages, registries make it easy to publish, discover, and integrate packages into your projects.
+
+## Official registry
+
+Currently, the default way to host packages is via the official [scarbs.xyz](https://scarbs.xyz) registry.
+Please note that the official registry is still in development.
+
+You can already use it to discover and [add](#adding-dependencies) packages to your projects.
+
+[Publishing](#packaging-and-publishing) packages is currently limited, but if there are any other packages you would like to be available - please reach out to us on [Telegram](https://t.me/scarbs_xyz) or [Discord](https://discord.gg/7YXj4Z2).
+
+## Adding dependencies
+
+If you want to add a package from the official registry as a dependency, you can read about it [here](./../reference/specifying-dependencies#specifying-dependencies-from-official-registry).
+
+## Packaging and publishing
+
+If you are interested in learning about the packaging and publishing process, you can read about it [here](./publishing).
+
+## Custom registry
+
+Although Scarb uses the official registry by default,
+you can [host your own](./custom-registry) registry or search for and [use](./custom-registry#using-custom-registry) a community-hosted one instead.

website/docs/registries/package-tarball.md

@@ -1,13 +1,13 @@
 # Package tarball
 
-Package tarballs are a distributable, compressed `.tar.zst` files with the source code of the package and additional
+Package tarballs are distributable, compressed `.tar.zst` files with the source code of the package and additional
 metadata for use by registries and other services.
 Tarballs are regular [GNU tar archives](<https://en.wikipedia.org/wiki/Tar_(computing)>) compressed
 with [Zstandard](https://facebook.github.io/zstd/) algorithm.
 The `scarb package` command can be used to create a package tarball from a package directory.
 
-In general a package tarball consists minimum amount of files copied from package source directory and several
-additional metadata files.
+In general, a package tarball consists of the minimum amount of files copied from the package source directory,
+along with several additional metadata files.
 Scarb does not permit source files named like metadata files (case-insensitive) to be included in the tarball.
 
 ## Metadata
@@ -21,13 +21,13 @@ The current tarball version is `1`.
 
 ### `Scarb.toml`
 
-The package's `Scarb.toml` rewritten and normalized, so that contains only the most important information about a
+The package's `Scarb.toml` rewritten and normalized to contain only the most important information about a
 package in order to be built, processed in version resolution algorithm and presented in the registry.
 
 The normalization process consists of the following:
 
 1. All workspace references are expanded.
-2. All dependency specifications are stripped from non-registry source properties. For example:
+2. All dependency specifications are stripped of non-registry source properties. For example:
 
    ```toml
    [dependencies]
@@ -42,14 +42,35 @@ The normalization process consists of the following:
    ```
 
 3. All sections other than `[package]`, `[dependencies]` and `[tool]` are removed from the manifest.
-4. All auto-detected properties, like `package.readme` are explicitly stated.
+4. All auto-detected properties, like `package.readme`, are explicitly stated.
 
 ### `Scarb.orig.toml`
 
 The original `Scarb.toml` file from the package source directory, without any processing.
 
+### `README` and `LICENSE`
+
+If these files are present in the project root, or if paths to them are provided in the manifest, they are copied to the archive as `README.md` and `LICENSE` respectively.
+
+### `VCS.json`
+
+File containing info about the version control system used in your package.
+Currently, only Git is supported.
+
+```json
+{
+  "git": {
+    "sha1": "a928d5ba03fc09d3316b39f04f30ee135df0c606"
+  },
+  "path_in_vcs": ""
+}
+```
+
+It contains information about the hash of the commit that the package was created on, together with `path_in_vcs`, which describes the package's relative position to the Git working directory root.
+It will be an empty string if it is the same as the package root.
+
 ## Package source
 
 By default, only the `src` directory from package source is included in the tarball.
 Additionally, the readme and license files may be included, if relevant fields are present in the source `Scarb.toml`
-file (or their values were auto-detected).
+file (or if their values were auto-detected).

website/docs/registries/publishing.md

@@ -0,0 +1,39 @@
+# Publishing your package
+
+To share your package, it needs to be packaged into an archive and uploaded to the registry.
+Once uploaded, it will be available for other users to download.
+
+## Packaging your package
+
+Use the `scarb package` command to create an archive of your package.
+You can read about the package compression algorithm and contents in the [Package tarball](./package-tarball) section.
+Basically when you run the command, Scarb gathers the source code of your package along with metadata files, such as the manifest file, and places them in an archive in `target/package` directory.
+
+If you are in a Git repository, Scarb will first check if the repo state is clean and error out in case of any changes present in the Git working directory.
+To bypass this check, you can use the `--allow-dirty` flag.
+
+The next step is package verification.
+After creating the initial archive, Scarb will attempt to unpack it and compile to check for any corruptions in the packaging process.
+If you want to speed up the packaging process, you can disable this step using the `--no-verify` flag.
+
+> [!WARNING]
+> This is a dangerous operation as it can lead to uploading a corrupted package to the registry.
+> Please use with caution.
+
+After successfully completing the whole process, the `{name}-{version}.tar.zst` archive waits in the `target/package` directory for being uploaded, where both `name` and `version` correspond to the values in `Scarb.toml`.
+
+## Publishing the package
+
+> [!WARNING]
+> Currently, packages can only be published to a local [custom registry](./custom-registry.md).
+> Publishing packages over HTTP is not yet supported.
+>
+> If you're interested in making your package available in the official [scarbs.xyz](https://scarbs.xyz) registry,
+> please reach out to us on [Telegram](https://t.me/scarbs_xyz) or [Discord](https://discord.gg/7YXj4Z2).
+
+To upload your package, you can use the `scarb publish` command.
+The command takes the `--index` argument that you can use to pass the local directory path where you want to store the packages.
+
+```shell
+scarb publish --index file:///Users/foo/bar
+```