From 27989122c2d7d1ede906aacc40e69d211cea3124 Mon Sep 17 00:00:00 2001 From: Travis Vachon Date: Thu, 7 Mar 2024 15:53:09 -0800 Subject: [PATCH 01/15] feat: first pass at docs for the HTTP bridge Write up basic documentation for using the HTTP bridge with popular HTTP command line tool `curl`. I am very close to having sharded uploads working on the command line thanks to the helpful `split` utility, but I don't think I'm choosing the `root` CID properly - looking for pointers from reviewers! --- src/pages/docs/how-to/http-bridge.mdx | 274 ++++++++++++++++++++++++++ 1 file changed, 274 insertions(+) create mode 100644 src/pages/docs/how-to/http-bridge.mdx diff --git a/src/pages/docs/how-to/http-bridge.mdx b/src/pages/docs/how-to/http-bridge.mdx new file mode 100644 index 0000000..e895b89 --- /dev/null +++ b/src/pages/docs/how-to/http-bridge.mdx @@ -0,0 +1,274 @@ +import { Callout } from 'nextra/components' + +# Use the HTTP API Bridge + +Our API is built around [UCAN Invocation](https://github.com/ucan-wg/invocation). This +allows clients to verify that our service executed a submitted UCAN invocation and produced a +given result. This gives us a flexible, interoperable, verifiable computational abstraction +to build upon, but is not widely supported by programming languages or operating system utilities, which +is a significant disadvantage compared to a more traditional auth-token-based HTTP API. + +To compensate for this, we've implemented a "bridge" that lets you use the tools you +know and love to interact with our API in a manner that should feel familiar if you've +used an HTTP-based API before. + +## Limitations + +The HTTP Bridge is currently designed to be used to invoke capabilities on +spaces. `store/add`, `upload/add` and `upload/list` are examples of such capabilities. Other +capabilities like `access/request` are not supported at this time - please let us know +if you would like to invoke other capabilities using the bridge so that we can +prioritize adding support for them. + +## Authentication and Authorization + +Our UCAN Invocation-based API uses public/private keypairs for to identify users and UCAN Delegation +to authorized keypairs to invoke specific capabilities. The HTTP Bridge uses the same fundamental primitives +but confines them to two separate HTTP headers, `X-Auth-Secret` and `Authorization`, that can be generated +by clients without interacting with our service. + +Details of how to generate the header values can be found in the +[W3UP UCAN Bridge Protocol](https://github.com/web3-storage/specs/blob/main/w3-ucan-bridge.md) specification, but +they can also be generated for you by the [`w3` CLI](https://github.com/web3-storage/w3cli). + +### Generating Auth{n|z} headers with `w3cli` + +If you have not already logged in and created a space with the `w3` CLI, do that first: + +```sh +w3 login you@example.com +# follow onscreen instructions +w3 space create +# follow onscreen instructions +``` + +Next, find the DID of the space you'd like to use with the bridge: + +```sh +w3 space ls +* did:key:z6Mkabc123 +``` + +And use the `bridge generate-tokens` command to generate header values: + +```sh +w3 bridge generate-tokens did:key:z6Mkabc123 --can 'store/add' --can 'upload/add' --can 'upload/list' --expiration `date -v +24H +%s` +``` + +The example command above uses the `--expiration` option to specify that these headers will be +valid for 24 hours. It uses the `--can` options to specify that the headers can be used to invoke +`store/add`, `upload/add` or `upload/list` capabilities. + +The result of `w3 bridge generate-tokens` will contain the values of the `X-Auth-Secret` and `Authorization` headers: + +``` +X-Auth-Secret header: uOGQyYzRhYWQwNmY3NtEwOTg1ZWU0NDU0NjByNTg2ZGE + +Authorization header: uOqJlcm9vdHOB2CpYKQABcRIgYVymlT6sxiDd45CA0f... +``` + +These can be used with `curl` like: + +```sh +curl -X POST \ +-H "X-Auth-Secret: uOGQyYzRhYWQwNmY3NtEwOTg1ZWU0NDU0NjByNTg2ZGE" \ +-H "Authorization: uOqJlcm9vdHOB2CpYKQABcRIgYVymlT6sxiDd45CA0f..." \ +... +https://up.web3.storage/bridge +``` + +## Listing Uploads + +Listing uploads in a space is one of the easiest ways to ensure the bridge is working. We'll +give an example using the common unix utility `curl`: + +First, create a file named `list.json` and put the following JSON in it, replacing `did:key:z6Mkabc123` +with the space you used to generate auth headers above: + +```json +{ + "tasks": [ + [ + "upload/list", + "did:key:z6Mkabc123", + {} + ] + ] +} +``` + +Next, run `curl` as follows, replacing the values of the `X-Auth-Secret` and `Authorization` +headers with the values you generated above: + +```sh +curl -X POST \ +-H "X-Auth-Secret: uOGQyYzRhYWQwNmY3NtEwOTg1ZWU0NDU0NjByNTg2ZGE" \ +-H "Authorization: uOqJlcm9vdHOB2CpYKQABcRIgYVymlT6sxiDd45CA0f..." \ +-d @list.json \ +https://up.web3.storage/bridge +``` + +## Storing Files + +TODO: add information about max upload size (5gb?) + +To upload a file to your space you'll first need convert it to a CAR. Assuming your file is +named `hello.txt` you can first use `ipfs-car` to create a CAR: + +```sh +ipfs-car pack hello.txt > hello.car +bafybeidhkumeonuwkebh2i4fc7o7lguehauradvlk57gzake6ggjsy372a +``` +Hang on to the `bafy...` Content ID output here! You'll need it for the "Registering Uploads" section below. Next, +calculate the Content ID of the CAR you just created: + +```sh +ipfs-car hash hello.car +bagbaiera4tntawdwlxf33uld7hd5yfaadct5galsr45vawbomjbiytdx4dzq +``` + +You'll also need to calculate the size of `hello.car` in bytes. The common utility `wc` works well: + +```sh +wc -c hello.car +161 hello.car +``` + +Next, create a file (in this example we'll name it `store.json`) and put the following JSON +in it, replacing `did:key:z6Mkabc123` with the space you used to generate auth headers above: + +```json +{ + "tasks": [ + [ + "store/add", + "did:key:z6Mkabc123", + { + "link": "bagbaiera4tntawdwlxf33uld7hd5yfaadct5galsr45vawbomjbiytdx4dzq", + "size": 161 + } + ] + ] +} +``` + +Finally, run `curl` replacing the values of the `X-Auth-Secret` and `Authorization` +headers with the values you generated above: + +```sh +curl -X POST \ +-H "X-Auth-Secret: uOGQyYzRhYWQwNmY3NtEwOTg1ZWU0NDU0NjByNTg2ZGE" \ +-H "Authorization: uOqJlcm9vdHOB2CpYKQABcRIgYVymlT6sxiDd45CA0f..." \ +-d @store.json \ +https://up.web3.storage/bridge +``` + +You'll get a response that includes a number of different fields: + +```json +[ + { + "p": { + "fx": { + "fork": [] + }, + "iss": "did:web:web3.storage", + "meta": {}, + "out": { + "ok": { + "allocated": 161, + "headers": { + "content-length": "161", + "x-amz-checksum-sha256": "5NswWHZdy73RY/nH3BQAGKfTAXKPO1BYLmJCjEx34PM=" + }, + "link": { + "/": "bagbaiera4tntawdwlxf33uld7hd5yfaadct5galsr45vawbomjbiytdx4dzq" + }, + "status": "upload", + "url": "https://carpark-prod-0.s3.us-west-2.amazonaws.com/bagbaiera4tntawdwlxf33uld7hd5yfaadct5galsr45vawbomjbiytdx4dzq/bagbaiera4tntawdwlxf33uld7hd5yfaadct5galsr45vawbomjbiytdx4dzq.car?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=ASIAXLN6VFMM76NYBTG7%2F20240307%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20240307T170854Z&X-Amz-Expires=86400&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEMn%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLXdlc3QtMiJHMEUCIQC2snXc59uBD2ugjc7C9DiKP0447T7fZ0A010Ph6iDswgIgfq0fYWE4Y9FByHz%2B6mYRussCgVr9KlBV1eGd1kgA22Yq3AMIwv%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARAAGgw1MDU1OTUzNzQzNjEiDCjNsn%2FQBypcXnybPCqwAyGtjDvgxmoQg8IYvO4vQ5fxZw%2F69mzSB%2BJmJMl9gnMaQ7SqUuTQnamPEZHM%2BaztPTbSBFm26S9o4GcZWn5S9UCOLb4YV6sAeCC3XCMkc58RPvyehMiVf0xdN1lBCLkFDd%2BHDb0cPEOvvJbTor22rhCo9L%2FsAVZPAYBu1efwvifCMnUebFeA3Qc0lvYG9PqLtZwo2ZMeVCwXYZiwec%2FpwdobopEWve88GJtxrVyiWMuV4pO7A68cty5ukFr0aRwNHM9g2e2y%2BSymH8pS6mfWpdope%2F%2BB6JoYvKKPN%2FQMpvROl0BxNdTD%2FgNUNqKjlNfUDtV7NqaDJlTqtlTXp44%2F7AUk7yJgK9UlVmp7B30oNH3fPtNXEFBbowoVczqcDBfp%2BxDBMZRFDa37VNcoIdjEBpBTCZk5hONDpjlp%2FyGd1EPSswbQyr6nA5xwLunfF6HhWA%2B8ZMp5raiepAPm%2FS7QfpbiZGtEZ%2FoVWENmPI4HLjDKYjXRJ7i680K9cMDeFSHjHvaTYsgOh09jmL5OnKSMRbANl4bNihk%2FX7pWgxVmbV0Qj%2BQ5eDpvsYUe9i%2FTUVAyODDU5KevBjqeATdV%2Fp9WiSN1mc1%2FUxn60TYZRFAjcTm6lfdbdtEEWneexsLLj6BpoW6gUA1c73JjEXsOEGbaSfHXi%2BI6nFtlNJpKlP%2BbrFlylA2qmIvlnKJzv2ALrsARUmCH1Kl26CVFfOtnJv7VZmDSVnS9nr4mYLHLkPjIhBaF3IxWVcFR7He0CCfRv%2BYkHzTlVdzoK%2BmJkUVyofY%2B3Bt3dDkpk7No&X-Amz-Signature=1d4cbc65a8440c89eb2770b2edfe4e3fe91831856ee03d0d939911d9963337f0&X-Amz-SignedHeaders=content-length%3Bhost%3Bx-amz-checksum-sha256&x-id=PutObject", + "with": "did:key:z6Mkm5qHN9g9NQSGbBfL7iGp9sexdssioT4CzyVap9ATqGqX" + } + }, + "prf": [], + "ran": { + "/": "bafyreib732osv25yuv4zs7fhr6ijywa4itb62g6o7chej2jh6ierf6o6jy" + } + }, + "s": { + "/": { + "bytes": "7aEDQH53KYLjnCneSTc9nslSC1AjFCU1dtGOgwwaCopfxS1Stur+GzYpubUnvjykRbqcyUvaHutl+XVxoZuOwV2mRgY" + } + } + } +] +``` + +The three you care about are `p.out.ok.status`, `p.out.ok.url` and `p.out.ok.headers`. If the value of `p.out.ok.status` is `done`, +congratulations! You're done. The file has already been uploaded to our system and you don't need to do anything else. + +If the value of `p.out.ok.status` is `upload` you'll need to send an HTTP `PUT` to the value of `p.out.ok.url`, passing the bytes of the CAR +in the body of the request and including the HTTP headers in `p.out.ok.headers`: + +```sh +curl -v -X PUT \ +-H "content-length: 161" \ +-H "x-amz-checksum-sha256: 5NswWHZdy73RY/nH3BQAGKfTAXKPO1BYLmJCjEx34PM=" \ +--data-binary @hello.car \ +https://carpark-prod-0.s3.us-west-2.amazonaws.com/bagbaiera4tntawdwlxf33uld7hd5yfaadct5galsr45vawbomjbiytdx4dzq/bagbaiera4tntawdwlxf33uld7hd5yfaadct5galsr45vawbomjbiytdx4dzq.car\?X-Amz-Algorithm\=AWS4-HMAC-SHA256\&X-Amz-Content-Sha256\=UNSIGNED-PAYLOAD\&X-Amz-Credential\=ASIAXLN6VFMM76NYBTG7%2F20240307%2Fus-west-2%2Fs3%2Faws4_request\&X-Amz-Date\=20240307T170854Z\&X-Amz-Expires\=86400\&X-Amz-Security-Token\=IQoJb3JpZ2luX2VjEMn%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLXdlc3QtMiJHMEUCIQC2snXc59uBD2ugjc7C9DiKP0447T7fZ0A010Ph6iDswgIgfq0fYWE4Y9FByHz%2B6mYRussCgVr9KlBV1eGd1kgA22Yq3AMIwv%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARAAGgw1MDU1OTUzNzQzNjEiDCjNsn%2FQBypcXnybPCqwAyGtjDvgxmoQg8IYvO4vQ5fxZw%2F69mzSB%2BJmJMl9gnMaQ7SqUuTQnamPEZHM%2BaztPTbSBFm26S9o4GcZWn5S9UCOLb4YV6sAeCC3XCMkc58RPvyehMiVf0xdN1lBCLkFDd%2BHDb0cPEOvvJbTor22rhCo9L%2FsAVZPAYBu1efwvifCMnUebFeA3Qc0lvYG9PqLtZwo2ZMeVCwXYZiwec%2FpwdobopEWve88GJtxrVyiWMuV4pO7A68cty5ukFr0aRwNHM9g2e2y%2BSymH8pS6mfWpdope%2F%2BB6JoYvKKPN%2FQMpvROl0BxNdTD%2FgNUNqKjlNfUDtV7NqaDJlTqtlTXp44%2F7AUk7yJgK9UlVmp7B30oNH3fPtNXEFBbowoVczqcDBfp%2BxDBMZRFDa37VNcoIdjEBpBTCZk5hONDpjlp%2FyGd1EPSswbQyr6nA5xwLunfF6HhWA%2B8ZMp5raiepAPm%2FS7QfpbiZGtEZ%2FoVWENmPI4HLjDKYjXRJ7i680K9cMDeFSHjHvaTYsgOh09jmL5OnKSMRbANl4bNihk%2FX7pWgxVmbV0Qj%2BQ5eDpvsYUe9i%2FTUVAyODDU5KevBjqeATdV%2Fp9WiSN1mc1%2FUxn60TYZRFAjcTm6lfdbdtEEWneexsLLj6BpoW6gUA1c73JjEXsOEGbaSfHXi%2BI6nFtlNJpKlP%2BbrFlylA2qmIvlnKJzv2ALrsARUmCH1Kl26CVFfOtnJv7VZmDSVnS9nr4mYLHLkPjIhBaF3IxWVcFR7He0CCfRv%2BYkHzTlVdzoK%2BmJkUVyofY%2B3Bt3dDkpk7No\&X-Amz-Signature\=1d4cbc65a8440c89eb2770b2edfe4e3fe91831856ee03d0d939911d9963337f0\&X-Amz-SignedHeaders\=content-length%3Bhost%3Bx-amz-checksum-sha256\&x-id\=PutObject +``` + +Your bits are now on IPFS, en route to the Filecoin Network, *but they will not show up as uploads in your web3.storage account!* To +finish "uploading" your file to web3.storage you'll need to register your upload as described in the next section. + +## Registering Uploads + +To support arbitrarily large file uploads, our upload service requires large files to be sharded into multiple CARs, each +of which is uploaded seperately according to the instructions in the "Storing Files" section above. + +Once the files have been uploaded, create a file (in this example we'll name it `upload.json`) with the following contents: + +```json +{ + "tasks": [ + [ + "upload/add", + "did:key:z6Mkm5qHN9g9NQSGbBfL7iGp9sexdssioT4CzyVap9ATqGqX", + { + "root": { + "/": "bafybeidhkumeonuwkebh2i4fc7o7lguehauradvlk57gzake6ggjsy372a" + }, + "shards": [ + { + "/": "bagbaiera4tntawdwlxf33uld7hd5yfaadct5galsr45vawbomjbiytdx4dzq" + } + ] + } + ] + ] +} +``` + +Change the `root` CID to the CID output when you ran `ipfs-car pack` above. Change the `shards` CID to the CID output +when you ran `ipfs-car hash` above. + +Finally, run: + +```sh +curl -X POST \ +-H "X-Auth-Secret: uOGQyYzRhYWQwNmY3NtEwOTg1ZWU0NDU0NjByNTg2ZGE" \ +-H "Authorization: uOqJlcm9vdHOB2CpYKQABcRIgYVymlT6sxiDd45CA0f..." \ +-d @upload.json \ +https://up.web3.storage/bridge +``` + +being sure to again replace `X-Auth-Secret` and `Authorization` with the values generated by `w3 bridge generate-tokens`. + +If everything works as expected you will now see your upload when you invoke `upload/list` per the instructions in +"Listing Uploads" above! + +## Uploading Large Files + +Large files should be split into multiple shards. + +TODO: figure out what to use as the `root` CID for sharded uploads - I tried the CID of the first shard and the CID +I get when I `ipfs-car pack` the whole file, but neither seemed quite right. \ No newline at end of file From cdb1c11e9b2d63a5e276a55ce2d57e50a204ab14 Mon Sep 17 00:00:00 2001 From: Travis Vachon Date: Thu, 7 Mar 2024 15:56:51 -0800 Subject: [PATCH 02/15] fix: compilation error --- src/pages/docs/how-to/http-bridge.mdx | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/src/pages/docs/how-to/http-bridge.mdx b/src/pages/docs/how-to/http-bridge.mdx index e895b89..144a7ae 100644 --- a/src/pages/docs/how-to/http-bridge.mdx +++ b/src/pages/docs/how-to/http-bridge.mdx @@ -1,5 +1,3 @@ -import { Callout } from 'nextra/components' - # Use the HTTP API Bridge Our API is built around [UCAN Invocation](https://github.com/ucan-wg/invocation). This @@ -31,7 +29,7 @@ Details of how to generate the header values can be found in the [W3UP UCAN Bridge Protocol](https://github.com/web3-storage/specs/blob/main/w3-ucan-bridge.md) specification, but they can also be generated for you by the [`w3` CLI](https://github.com/web3-storage/w3cli). -### Generating Auth{n|z} headers with `w3cli` +### Generating Auth Headers With `w3cli` If you have not already logged in and created a space with the `w3` CLI, do that first: From 2b7eaa4c828692cffc4b39a24862d5be5ca5e123 Mon Sep 17 00:00:00 2001 From: Travis Vachon Date: Thu, 7 Mar 2024 15:59:13 -0800 Subject: [PATCH 03/15] fix: lint --- src/pages/docs/how-to/http-bridge.mdx | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/src/pages/docs/how-to/http-bridge.mdx b/src/pages/docs/how-to/http-bridge.mdx index 144a7ae..48e6722 100644 --- a/src/pages/docs/how-to/http-bridge.mdx +++ b/src/pages/docs/how-to/http-bridge.mdx @@ -12,7 +12,7 @@ used an HTTP-based API before. ## Limitations -The HTTP Bridge is currently designed to be used to invoke capabilities on +The HTTP Bridge is currently designed to be used to invoke capabilities on spaces. `store/add`, `upload/add` and `upload/list` are examples of such capabilities. Other capabilities like `access/request` are not supported at this time - please let us know if you would like to invoke other capabilities using the bridge so that we can @@ -59,7 +59,7 @@ valid for 24 hours. It uses the `--can` options to specify that the headers can The result of `w3 bridge generate-tokens` will contain the values of the `X-Auth-Secret` and `Authorization` headers: -``` +```txt X-Auth-Secret header: uOGQyYzRhYWQwNmY3NtEwOTg1ZWU0NDU0NjByNTg2ZGE Authorization header: uOqJlcm9vdHOB2CpYKQABcRIgYVymlT6sxiDd45CA0f... @@ -117,7 +117,8 @@ named `hello.txt` you can first use `ipfs-car` to create a CAR: ipfs-car pack hello.txt > hello.car bafybeidhkumeonuwkebh2i4fc7o7lguehauradvlk57gzake6ggjsy372a ``` -Hang on to the `bafy...` Content ID output here! You'll need it for the "Registering Uploads" section below. Next, + +Hang on to the `bafy...` Content ID output here! You'll need it for the "Registering Uploads" section below. Next, calculate the Content ID of the CAR you just created: ```sh @@ -201,10 +202,10 @@ You'll get a response that includes a number of different fields: ] ``` -The three you care about are `p.out.ok.status`, `p.out.ok.url` and `p.out.ok.headers`. If the value of `p.out.ok.status` is `done`, -congratulations! You're done. The file has already been uploaded to our system and you don't need to do anything else. +The three you care about are `p.out.ok.status`, `p.out.ok.url` and `p.out.ok.headers`. If the value of `p.out.ok.status` is `done`, +congratulations! You're done. The file has already been uploaded to our system and you don't need to do anything else. -If the value of `p.out.ok.status` is `upload` you'll need to send an HTTP `PUT` to the value of `p.out.ok.url`, passing the bytes of the CAR +If the value of `p.out.ok.status` is `upload` you'll need to send an HTTP `PUT` to the value of `p.out.ok.url`, passing the bytes of the CAR in the body of the request and including the HTTP headers in `p.out.ok.headers`: ```sh @@ -215,7 +216,7 @@ curl -v -X PUT \ https://carpark-prod-0.s3.us-west-2.amazonaws.com/bagbaiera4tntawdwlxf33uld7hd5yfaadct5galsr45vawbomjbiytdx4dzq/bagbaiera4tntawdwlxf33uld7hd5yfaadct5galsr45vawbomjbiytdx4dzq.car\?X-Amz-Algorithm\=AWS4-HMAC-SHA256\&X-Amz-Content-Sha256\=UNSIGNED-PAYLOAD\&X-Amz-Credential\=ASIAXLN6VFMM76NYBTG7%2F20240307%2Fus-west-2%2Fs3%2Faws4_request\&X-Amz-Date\=20240307T170854Z\&X-Amz-Expires\=86400\&X-Amz-Security-Token\=IQoJb3JpZ2luX2VjEMn%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLXdlc3QtMiJHMEUCIQC2snXc59uBD2ugjc7C9DiKP0447T7fZ0A010Ph6iDswgIgfq0fYWE4Y9FByHz%2B6mYRussCgVr9KlBV1eGd1kgA22Yq3AMIwv%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARAAGgw1MDU1OTUzNzQzNjEiDCjNsn%2FQBypcXnybPCqwAyGtjDvgxmoQg8IYvO4vQ5fxZw%2F69mzSB%2BJmJMl9gnMaQ7SqUuTQnamPEZHM%2BaztPTbSBFm26S9o4GcZWn5S9UCOLb4YV6sAeCC3XCMkc58RPvyehMiVf0xdN1lBCLkFDd%2BHDb0cPEOvvJbTor22rhCo9L%2FsAVZPAYBu1efwvifCMnUebFeA3Qc0lvYG9PqLtZwo2ZMeVCwXYZiwec%2FpwdobopEWve88GJtxrVyiWMuV4pO7A68cty5ukFr0aRwNHM9g2e2y%2BSymH8pS6mfWpdope%2F%2BB6JoYvKKPN%2FQMpvROl0BxNdTD%2FgNUNqKjlNfUDtV7NqaDJlTqtlTXp44%2F7AUk7yJgK9UlVmp7B30oNH3fPtNXEFBbowoVczqcDBfp%2BxDBMZRFDa37VNcoIdjEBpBTCZk5hONDpjlp%2FyGd1EPSswbQyr6nA5xwLunfF6HhWA%2B8ZMp5raiepAPm%2FS7QfpbiZGtEZ%2FoVWENmPI4HLjDKYjXRJ7i680K9cMDeFSHjHvaTYsgOh09jmL5OnKSMRbANl4bNihk%2FX7pWgxVmbV0Qj%2BQ5eDpvsYUe9i%2FTUVAyODDU5KevBjqeATdV%2Fp9WiSN1mc1%2FUxn60TYZRFAjcTm6lfdbdtEEWneexsLLj6BpoW6gUA1c73JjEXsOEGbaSfHXi%2BI6nFtlNJpKlP%2BbrFlylA2qmIvlnKJzv2ALrsARUmCH1Kl26CVFfOtnJv7VZmDSVnS9nr4mYLHLkPjIhBaF3IxWVcFR7He0CCfRv%2BYkHzTlVdzoK%2BmJkUVyofY%2B3Bt3dDkpk7No\&X-Amz-Signature\=1d4cbc65a8440c89eb2770b2edfe4e3fe91831856ee03d0d939911d9963337f0\&X-Amz-SignedHeaders\=content-length%3Bhost%3Bx-amz-checksum-sha256\&x-id\=PutObject ``` -Your bits are now on IPFS, en route to the Filecoin Network, *but they will not show up as uploads in your web3.storage account!* To +Your bits are now on IPFS, en route to the Filecoin Network, _but they will not show up as uploads in your web3.storage account!_ To finish "uploading" your file to web3.storage you'll need to register your upload as described in the next section. ## Registering Uploads @@ -266,7 +267,7 @@ If everything works as expected you will now see your upload when you invoke `up ## Uploading Large Files -Large files should be split into multiple shards. +Large files should be split into multiple shards. -TODO: figure out what to use as the `root` CID for sharded uploads - I tried the CID of the first shard and the CID -I get when I `ipfs-car pack` the whole file, but neither seemed quite right. \ No newline at end of file +TODO: figure out what to use as the `root` CID for sharded uploads - I tried the CID of the first shard and the CID +I get when I `ipfs-car pack` the whole file, but neither seemed quite right. From 9563841888229c9b4ce6b8eacfef7664ab1b12c1 Mon Sep 17 00:00:00 2001 From: Travis Vachon Date: Fri, 8 Mar 2024 12:19:59 -0800 Subject: [PATCH 04/15] fix: minor edits --- src/pages/docs/how-to/http-bridge.mdx | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/pages/docs/how-to/http-bridge.mdx b/src/pages/docs/how-to/http-bridge.mdx index 48e6722..cffa352 100644 --- a/src/pages/docs/how-to/http-bridge.mdx +++ b/src/pages/docs/how-to/http-bridge.mdx @@ -20,16 +20,16 @@ prioritize adding support for them. ## Authentication and Authorization -Our UCAN Invocation-based API uses public/private keypairs for to identify users and UCAN Delegation -to authorized keypairs to invoke specific capabilities. The HTTP Bridge uses the same fundamental primitives +Our UCAN Invocation-based API uses public/private keypairs to identify users and UCAN Delegation +to allow those keypairs to invoke specific capabilities. The HTTP Bridge uses the same fundamental primitives but confines them to two separate HTTP headers, `X-Auth-Secret` and `Authorization`, that can be generated by clients without interacting with our service. Details of how to generate the header values can be found in the [W3UP UCAN Bridge Protocol](https://github.com/web3-storage/specs/blob/main/w3-ucan-bridge.md) specification, but -they can also be generated for you by the [`w3` CLI](https://github.com/web3-storage/w3cli). +they can also be generated for you by the [`w3` CLI](https://github.com/web3-storage/w3cli): -### Generating Auth Headers With `w3cli` +### Generating Auth Headers with `w3cli` If you have not already logged in and created a space with the `w3` CLI, do that first: From c2f07ce3e12b8a8317b0228eb80d1e898dd1a5a0 Mon Sep 17 00:00:00 2001 From: Travis Vachon Date: Fri, 8 Mar 2024 12:58:35 -0800 Subject: [PATCH 05/15] chore: remove uploading large files section need to figure out how sharding would even work on the command line --- src/pages/docs/how-to/http-bridge.mdx | 5 ----- 1 file changed, 5 deletions(-) diff --git a/src/pages/docs/how-to/http-bridge.mdx b/src/pages/docs/how-to/http-bridge.mdx index cffa352..5a3adcd 100644 --- a/src/pages/docs/how-to/http-bridge.mdx +++ b/src/pages/docs/how-to/http-bridge.mdx @@ -265,9 +265,4 @@ being sure to again replace `X-Auth-Secret` and `Authorization` with the values If everything works as expected you will now see your upload when you invoke `upload/list` per the instructions in "Listing Uploads" above! -## Uploading Large Files -Large files should be split into multiple shards. - -TODO: figure out what to use as the `root` CID for sharded uploads - I tried the CID of the first shard and the CID -I get when I `ipfs-car pack` the whole file, but neither seemed quite right. From a726713d1cb1cc38b7c5113b4c856f54b1fa2282 Mon Sep 17 00:00:00 2001 From: Travis Vachon Date: Fri, 8 Mar 2024 13:20:10 -0800 Subject: [PATCH 06/15] feat: add a section on batching operations --- src/pages/docs/how-to/http-bridge.mdx | 128 +++++++++++++++++++++++++- 1 file changed, 126 insertions(+), 2 deletions(-) diff --git a/src/pages/docs/how-to/http-bridge.mdx b/src/pages/docs/how-to/http-bridge.mdx index 5a3adcd..37ff61e 100644 --- a/src/pages/docs/how-to/http-bridge.mdx +++ b/src/pages/docs/how-to/http-bridge.mdx @@ -224,14 +224,15 @@ finish "uploading" your file to web3.storage you'll need to register your upload To support arbitrarily large file uploads, our upload service requires large files to be sharded into multiple CARs, each of which is uploaded seperately according to the instructions in the "Storing Files" section above. -Once the files have been uploaded, create a file (in this example we'll name it `upload.json`) with the following contents: +Once the files have been uploaded, create a file (in this example we'll name it `upload.json`) with the following contents, replacing +`did:key:z6Mkabc123` with the DID of the space you used to generate auth headers above: ```json { "tasks": [ [ "upload/add", - "did:key:z6Mkm5qHN9g9NQSGbBfL7iGp9sexdssioT4CzyVap9ATqGqX", + "did:key:z6Mkabc123", { "root": { "/": "bafybeidhkumeonuwkebh2i4fc7o7lguehauradvlk57gzake6ggjsy372a" @@ -265,4 +266,127 @@ being sure to again replace `X-Auth-Secret` and `Authorization` with the values If everything works as expected you will now see your upload when you invoke `upload/list` per the instructions in "Listing Uploads" above! +## Batching Operations + +The operations described in "Storing Files" and "Registering Uploads" above can be combined into a single HTTP +request to the bridge. To see this in action, create a file (in this example we'll name it `store-and-upload.json`) +with the following contents, replacing `did:key:z6Mkabc123` with the DID of the space you used to generate +auth headers above and the values of the `link`, `root` and `shards` keys with the CIDs of your upload: + +```json +{ + "tasks": [ + [ + "store/add", + "did:key:z6Mkabc123", + { + "link": "bagbaiera4tntawdwlxf33uld7hd5yfaadct5galsr45vawbomjbiytdx4dzq", + "size": 161 + } + ], + [ + "upload/add", + "did:key:z6Mkabc123", + { + "root": { + "/": "bafybeidhkumeonuwkebh2i4fc7o7lguehauradvlk57gzake6ggjsy372a" + }, + "shards": [ + { + "/": "bagbaiera4tntawdwlxf33uld7hd5yfaadct5galsr45vawbomjbiytdx4dzq" + } + ] + } + ] + ] +} +``` + +And run: + +```sh +curl -X POST \ +-H "X-Auth-Secret: uOGQyYzRhYWQwNmY3NtEwOTg1ZWU0NDU0NjByNTg2ZGE" \ +-H "Authorization: uOqJlcm9vdHOB2CpYKQABcRIgYVymlT6sxiDd45CA0f..." \ +-d @store-and-upload.json \ +https://up.web3.storage/bridge +``` + +being sure to again replace `X-Auth-Secret` and `Authorization` with the values generated by `w3 bridge generate-tokens`. + +You'll get a result that looks like this: + +```json +[ + { + "p": { + "fx": { + "fork": [] + }, + "iss": "did:web:web3.storage", + "meta": {}, + "out": { + "ok": { + "allocated": 161, + "headers": { + "content-length": "161", + "x-amz-checksum-sha256": "5NswWHZdy73RY/nH3BQAGKfTAXKPO1BYLmJCjEx34PM=" + }, + "link": { + "/": "bagbaiera4tntawdwlxf33uld7hd5yfaadct5galsr45vawbomjbiytdx4dzq" + }, + "status": "upload", + "url": "https://carpark-prod-0.s3.us-west-2.amazonaws.com/bagbaiera4tntawdwlxf33uld7hd5yfaadct5galsr45vawbomjbiytdx4dzq/bagbaiera4tntawdwlxf33uld7hd5yfaadct5galsr45vawbomjbiytdx4dzq.car?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=ASIAXLN6VFMM76NYBTG7%2F20240307%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20240307T170854Z&X-Amz-Expires=86400&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEMn%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLXdlc3QtMiJHMEUCIQC2snXc59uBD2ugjc7C9DiKP0447T7fZ0A010Ph6iDswgIgfq0fYWE4Y9FByHz%2B6mYRussCgVr9KlBV1eGd1kgA22Yq3AMIwv%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARAAGgw1MDU1OTUzNzQzNjEiDCjNsn%2FQBypcXnybPCqwAyGtjDvgxmoQg8IYvO4vQ5fxZw%2F69mzSB%2BJmJMl9gnMaQ7SqUuTQnamPEZHM%2BaztPTbSBFm26S9o4GcZWn5S9UCOLb4YV6sAeCC3XCMkc58RPvyehMiVf0xdN1lBCLkFDd%2BHDb0cPEOvvJbTor22rhCo9L%2FsAVZPAYBu1efwvifCMnUebFeA3Qc0lvYG9PqLtZwo2ZMeVCwXYZiwec%2FpwdobopEWve88GJtxrVyiWMuV4pO7A68cty5ukFr0aRwNHM9g2e2y%2BSymH8pS6mfWpdope%2F%2BB6JoYvKKPN%2FQMpvROl0BxNdTD%2FgNUNqKjlNfUDtV7NqaDJlTqtlTXp44%2F7AUk7yJgK9UlVmp7B30oNH3fPtNXEFBbowoVczqcDBfp%2BxDBMZRFDa37VNcoIdjEBpBTCZk5hONDpjlp%2FyGd1EPSswbQyr6nA5xwLunfF6HhWA%2B8ZMp5raiepAPm%2FS7QfpbiZGtEZ%2FoVWENmPI4HLjDKYjXRJ7i680K9cMDeFSHjHvaTYsgOh09jmL5OnKSMRbANl4bNihk%2FX7pWgxVmbV0Qj%2BQ5eDpvsYUe9i%2FTUVAyODDU5KevBjqeATdV%2Fp9WiSN1mc1%2FUxn60TYZRFAjcTm6lfdbdtEEWneexsLLj6BpoW6gUA1c73JjEXsOEGbaSfHXi%2BI6nFtlNJpKlP%2BbrFlylA2qmIvlnKJzv2ALrsARUmCH1Kl26CVFfOtnJv7VZmDSVnS9nr4mYLHLkPjIhBaF3IxWVcFR7He0CCfRv%2BYkHzTlVdzoK%2BmJkUVyofY%2B3Bt3dDkpk7No&X-Amz-Signature=1d4cbc65a8440c89eb2770b2edfe4e3fe91831856ee03d0d939911d9963337f0&X-Amz-SignedHeaders=content-length%3Bhost%3Bx-amz-checksum-sha256&x-id=PutObject", + "with": "did:key:z6Mkabc123" + } + }, + "prf": [], + "ran": { + "/": "bafyreiclycv7vdfutbtakytkbvcpx2ontlp3fusfb7niugvy5a6kawnhk4" + } + }, + "s": { + "/": { + "bytes": "7aEDQDosaCizSSLXXm55dAeuDACNw2H9/JoUa6oLkLY7mv6TXeW+Z0xJouYj1MGf6LpHn756IW2NprqXq4c20EjlOwM" + } + } + }, + { + "p": { + "fx": { + "fork": [] + }, + "iss": "did:web:web3.storage", + "meta": {}, + "out": { + "ok": { + "root": { + "/": "bafybeidhkumeonuwkebh2i4fc7o7lguehauradvlk57gzake6ggjsy372a" + }, + "shards": [ + { + "/": "bagbaiera4tntawdwlxf33uld7hd5yfaadct5galsr45vawbomjbiytdx4dzq" + }, + { + "/": "bagbaieraw4kz73gtuttgxlckdby7fezy4o5hdfspcrl7nqxsh3ultzse2q2a" + } + ] + } + }, + "prf": [], + "ran": { + "/": "bafyreihobt6uxochgcmfs5bhetynrccrqsuipwp5cy5xxclvqgzw7qsmui" + } + }, + "s": { + "/": { + "bytes": "7aEDQIa1JFOzEWHHfUZQJqRhZzElISfhq4b9Xqca0TgjX+3N7SLrYTB6SN/2GdLSgkRuIKZivB+tSdG4s/+Fvh1CxAQ" + } + } + } +] +``` + +Remember that if the `status` field of the `store/add` result is `upload` you will then need to `PUT` your file to the +to the given URL with the given headers. From 6feafdfe1185922141b85fc9310ab59dd49580c0 Mon Sep 17 00:00:00 2001 From: Travis Vachon Date: Mon, 11 Mar 2024 10:46:53 -0700 Subject: [PATCH 07/15] fix: link JSON syntax Co-authored-by: Alan Shaw --- src/pages/docs/how-to/http-bridge.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/pages/docs/how-to/http-bridge.mdx b/src/pages/docs/how-to/http-bridge.mdx index 37ff61e..1422ee6 100644 --- a/src/pages/docs/how-to/http-bridge.mdx +++ b/src/pages/docs/how-to/http-bridge.mdx @@ -143,7 +143,7 @@ in it, replacing `did:key:z6Mkabc123` with the space you used to generate auth h "store/add", "did:key:z6Mkabc123", { - "link": "bagbaiera4tntawdwlxf33uld7hd5yfaadct5galsr45vawbomjbiytdx4dzq", + "link": { "/": "bagbaiera4tntawdwlxf33uld7hd5yfaadct5galsr45vawbomjbiytdx4dzq" }, "size": 161 } ] @@ -280,7 +280,7 @@ auth headers above and the values of the `link`, `root` and `shards` keys with t "store/add", "did:key:z6Mkabc123", { - "link": "bagbaiera4tntawdwlxf33uld7hd5yfaadct5galsr45vawbomjbiytdx4dzq", + "link": { "/": "bagbaiera4tntawdwlxf33uld7hd5yfaadct5galsr45vawbomjbiytdx4dzq" }, "size": 161 } ], From 765bc09f15bedc1ec8006dfd15774406788962d6 Mon Sep 17 00:00:00 2001 From: Travis Vachon Date: Mon, 11 Mar 2024 12:38:30 -0700 Subject: [PATCH 08/15] feat: add section on sharded uploads --- src/pages/docs/how-to/http-bridge.mdx | 268 +++++++++++++++++++++++++- 1 file changed, 267 insertions(+), 1 deletion(-) diff --git a/src/pages/docs/how-to/http-bridge.mdx b/src/pages/docs/how-to/http-bridge.mdx index 1422ee6..4e59ce6 100644 --- a/src/pages/docs/how-to/http-bridge.mdx +++ b/src/pages/docs/how-to/http-bridge.mdx @@ -108,7 +108,8 @@ https://up.web3.storage/bridge ## Storing Files -TODO: add information about max upload size (5gb?) +You can store individual files as big as 5GB in spaces on web3.storage. To store larger files, see the [Sharded Uploads] +section below. To upload a file to your space you'll first need convert it to a CAR. Assuming your file is named `hello.txt` you can first use `ipfs-car` to create a CAR: @@ -390,3 +391,268 @@ You'll get a result that looks like this: Remember that if the `status` field of the `store/add` result is `upload` you will then need to `PUT` your file to the to the given URL with the given headers. +## Sharded Uploads + +If you'd like to store a file larger than 5GB you'll need to shard it into smaller CARs before storing. We don't +proscribe any particular sharding algorithm - you'll need to: + +1) read the input CAR +2) write the first CAR file header +3) write blocks until the first CAR hits your limit (a number lower than 5GB) +4) write a new CAR file header for the second CAR +5) write blocks until the second CAR hits your limit +6) repeat until end of blocks + +If you're implementing this yourself it may be helpful to look +into [our JavaScript implementation](https://github.com/web3-storage/w3up/blob/d6978d7ab299be76987c6533d18e6857f6998fe6/packages/upload-client/src/index.js#L38). +We provide an example here using the [`carbites-cli`](https://www.npmjs.com/package/carbites-cli) utility: + +First we'll create an absolutely gigantic file: + +```sh +echo "this is an absolutely gigantic file created by travis" > big.txt +ipfs-car pack big.txt > big.car +bafybeicq2urqdqum3ykblswqzzxf6wpy3mcbr6jg52ai2zlhds4l7ho5wa +``` + +We'll save the CID `ipfs-car` outputs so we can use it in our `store/add` invocation later. + +Next we'll use `carbites` to split this file. We are using the `50B` splitting into 50 byte +chunks - you'll probably want to use something much larger like `100MB`: + +```sh +carbites split big.car --size 50B +``` + +You should now see three files, `big-0.car`, `big-1.car` and `big-2.car`. Calculate the CID and size of each one: + +```sh +wc big-* + 1 12 109 big-0.car + 2 7 118 big-1.car + 0 1 26 big-2.car + 3 20 253 total +ipfs-car hash big-0.car +bagbaiera5pcs7vgks34rflh47c6cprzoift7vfch6gmbm3odq7auw2iou3cq +ipfs-car hash big-1.car +bagbaierau2ocmnrl5ilikq5xtxckz3owbe5xallbx24fkvislzu7k7ydqzia +ipfs-car hash big-2.car +bagbaierawa335d45pwohko5s4fbut7nlfjavq2kan7z3gbzvm2k3zutifv5q +``` + +Next, create a file (in this example we'll name it `store-and-upload.json`) and put the following JSON +in it. The `link` and `size` in each `store/add` should match the CID and size calculated above, and +each `link` should also appear in the `shards` array of the `upload/add` call. The `root` CID in the `upload/add` +should be the CID output by `ipfs-car pack` above. Be sure to update the space DID `did:key:z6Mkabc123` to match +the space you used when you generated auth tokens! + +```json +{ + "tasks": [ + [ + "store/add", + "did:key:z6Mkabc123", + { + "link": { + "/": "bagbaiera5pcs7vgks34rflh47c6cprzoift7vfch6gmbm3odq7auw2iou3cq" + }, + "size": 109 + } + ], + [ + "store/add", + "did:key:z6Mkabc123", + { + "link": { + "/": "bagbaierau2ocmnrl5ilikq5xtxckz3owbe5xallbx24fkvislzu7k7ydqzia" + }, + "size": 118 + } + ], + [ + "store/add", + "did:key:z6Mkabc123", + { + "link": { + "/": "bagbaierawa335d45pwohko5s4fbut7nlfjavq2kan7z3gbzvm2k3zutifv5q" + }, + "size": 26 + } + ], + [ + "upload/add", + "did:key:z6Mkabc123", + { + "root": { + "/": "bafybeicq2urqdqum3ykblswqzzxf6wpy3mcbr6jg52ai2zlhds4l7ho5wa" + }, + "shards": [ + { + "/": "bagbaiera5pcs7vgks34rflh47c6cprzoift7vfch6gmbm3odq7auw2iou3cq" + }, + { + "/": "bagbaierau2ocmnrl5ilikq5xtxckz3owbe5xallbx24fkvislzu7k7ydqzia" + }, + { + "/": "bagbaierawa335d45pwohko5s4fbut7nlfjavq2kan7z3gbzvm2k3zutifv5q" + } + ] + } + ] + ] +} +``` + +Once you receive a response, you'll need to `PUT` each of the cars to their respective upload URLs. Given this result: + +```json +[ + { + "p": { + "fx": { + "fork": [] + }, + "iss": "did:web:web3.storage", + "meta": {}, + "out": { + "ok": { + "allocated": 0, + "link": { + "/": "bagbaiera5pcs7vgks34rflh47c6cprzoift7vfch6gmbm3odq7auw2iou3cq" + }, + "status": "done", + "with": "did:key:z6Mkabc123" + } + }, + "prf": [], + "ran": { + "/": "bafyreia526axkta24us4fxjghfqbwu2d7lk2neucrseoapakkcmgoy6x7u" + } + }, + "s": { + "/": { + "bytes": "7aEDQN9dWcLsWIwCMOLmzXP971VkJTU1hXliFpEpZZAYQhmOzReViM4TE4tp1AS4vwLcXzawiwaWJ+bkzvh2DKdx4go" + } + } + }, + { + "p": { + "fx": { + "fork": [] + }, + "iss": "did:web:web3.storage", + "meta": {}, + "out": { + "ok": { + "allocated": 0, + "link": { + "/": "bagbaierau2ocmnrl5ilikq5xtxckz3owbe5xallbx24fkvislzu7k7ydqzia" + }, + "status": "done", + "with": "did:key:z6Mkabc123" + } + }, + "prf": [], + "ran": { + "/": "bafyreidi655w32on25adjxnoxro3dnqcsncqly62uc6jyz6l25r45bd5lu" + } + }, + "s": { + "/": { + "bytes": "7aEDQALj9TzBJdqsXs8RY5NoIwy63wvALLw/U1gRcOkBp8VBoIqwEuL3xjjvr0RCnXzLoPQGszTP/xp/A51BJ0mlYQs" + } + } + }, + { + "p": { + "fx": { + "fork": [] + }, + "iss": "did:web:web3.storage", + "meta": {}, + "out": { + "ok": { + "allocated": 0, + "link": { + "/": "bagbaierawa335d45pwohko5s4fbut7nlfjavq2kan7z3gbzvm2k3zutifv5q" + }, + "status": "done", + "with": "did:key:z6Mkabc123" + } + }, + "prf": [], + "ran": { + "/": "bafyreic5wcfzvafpkt2haz77ya7zq5ka5ort73i4ahq2mvenj7zssyy6eq" + } + }, + "s": { + "/": { + "bytes": "7aEDQCdO8OKGjxTpEgsQaJV0ZEoFBfXw6mZGWx+gMUPxHMFmiO8AVrzyNXXGP1dy0q16yI+6bnnLpuMC684/n5dZQAU" + } + } + }, + { + "p": { + "fx": { + "fork": [] + }, + "iss": "did:web:web3.storage", + "meta": {}, + "out": { + "ok": { + "root": { + "/": "bafybeicq2urqdqum3ykblswqzzxf6wpy3mcbr6jg52ai2zlhds4l7ho5wa" + }, + "shards": [ + { + "/": "bagbaiera5pcs7vgks34rflh47c6cprzoift7vfch6gmbm3odq7auw2iou3cq" + }, + { + "/": "bagbaierau2ocmnrl5ilikq5xtxckz3owbe5xallbx24fkvislzu7k7ydqzia" + }, + { + "/": "bagbaierawa335d45pwohko5s4fbut7nlfjavq2kan7z3gbzvm2k3zutifv5q" + } + ] + } + }, + "prf": [], + "ran": { + "/": "bafyreicirpydnj66okf2bqmuv4qbdgxggxqmj5ybr3pyngcp3iqdwckc6e" + } + }, + "s": { + "/": { + "bytes": "7aEDQHJEvkeOSIkKQ0SaMbEgNXjLOuwNPWg047O08pCgRSSQMKrM3B++k6Qs49m/xSxpswY1FxuA/dQuh1NWrhqTdAY" + } + } + } +] +``` + +You'll need to run the following `curl` commands: + +```sh +curl -X PUT \ +-H "content-length: 109" \ +-H "x-amz-checksum-sha256: 68Uv1MqW+RKs/Pi8J8cuQWf6lEfxmBZtw4fBS2kOpsU=" \ +-H "content-type: application/vnd.ipld.car" \ +--data-binary @big-0.car \ +https://carpark-prod-0.s3.us-west-2.amazonaws.com/bagbaiera5pcs7vgks34rflh47c6cprzoift7vfch6gmbm3odq7auw2iou3cq/bagbaiera5pcs7vgks34rflh47c6cprzoift7vfch6gmbm3odq7auw2iou3cq.car?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=ASIAXLN6VFMMXACSF74D%2F20240311%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20240311T185406Z&X-Amz-Expires=86400&X-Amz-Security-Token=IQoJb3JpZ2luX2VjECsaCXVzLXdlc3QtMiJIMEYCIQCKTNRl%2FrZ47BKRxhGtQfkvjcvPMxBJMWUENt7EL4VYTgIhAO14%2Fs%2Bc3sh2TevvAxplSRT75wFFD9D88Fi3UXDKKCx3KtMDCDQQABoMNTA1NTk1Mzc0MzYxIgz9y4RR%2FZSZB4cqKpcqsAOKYIdtPJflZPodH36ke9NbJNTtfbD7E2LujMS7XEyU2sZnwQyTEzsVL%2FJvB3s8Z3eeKhN4xKnkbUAf%2BryO1ig1plAwzjnIpQzauv89AalyJTS7UM%2FymvWnLo2Qx%2Br92owtQCzOZmrREMVqAU0rCZrMv6NWcOH0bkaM%2FmuA6MijF2dfH3mGeW5E0lFqXaJL5OJqclREtJwRomotSljBVoYxoz2PHRvgIKFuezC1reS3XJ3EnWRlbohQzhfGov%2BtOtcUhJIVk8wqXQvyEFOjErapALiIq4oeUBoIpeKXR5iHJDgndnxGvPOG9ihjUMzKwcbEwqQZTSkTK2whRoHcSZgedzWpsQgae3vQjr797H%2Flh7XLSGzHAuy0Bb9zB61ZZRUyKbhODFCw%2BOpk9ihPKedaewtRHe4SYvAYwBcM0%2FZC%2BwcAOOK987eUxnULeHei%2Fs%2BigKBnzGWwojcuqSzdvaRX3yaK9yKG8mT6aMsAYKS%2BjG%2FX%2FlAu3UL4pe7KK2fxHEH1KrQgdaTCsdzaB%2FApfsixRH4XeGOTNk9ZYM2rI70aqMd%2F5fveFO2%2F33EKZSrQg9kw65i9rwY6nQFkK7gFDZd2oCG7e3u3lqDe5%2BQAqBshYAKbOzpMM6VQq0AqLFeO3TWv%2FhlSP8z9Xxadf3IFRciPlQbMV85VXxEDCzyTYAkLQJsqVuyI%2FQF9hW1rk1UcvOTVpcDRXuwBm%2B66Cq48OnTGzdMu32f%2FsKe7aacdi4msVqU2Z4JwvyjewD4biHJ7lNUkAUf16Zz7SKkLvSvxO%2FwqV7HcxrEs&X-Amz-Signature=5ce3660c1fc5364457e8f3732b759b9c17b3336011e0a04c119cb44bc56913ed&X-Amz-SignedHeaders=content-length%3Bhost%3Bx-amz-checksum-sha256&x-id=PutObject + +curl -X PUT \ +-H "content-length: 118" \ +-H "x-amz-checksum-sha256: ppwmNivqFoVDt53ErO3WCTtwLWG+uFVVEl5p9X8DhlA=" \ +-H "content-type: application/vnd.ipld.car" \ +--data-binary @big-1.car \ +https://carpark-prod-0.s3.us-west-2.amazonaws.com/bagbaierau2ocmnrl5ilikq5xtxckz3owbe5xallbx24fkvislzu7k7ydqzia/bagbaierau2ocmnrl5ilikq5xtxckz3owbe5xallbx24fkvislzu7k7ydqzia.car?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=ASIAXLN6VFMMXACSF74D%2F20240311%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20240311T185406Z&X-Amz-Expires=86400&X-Amz-Security-Token=IQoJb3JpZ2luX2VjECsaCXVzLXdlc3QtMiJIMEYCIQCKTNRl%2FrZ47BKRxhGtQfkvjcvPMxBJMWUENt7EL4VYTgIhAO14%2Fs%2Bc3sh2TevvAxplSRT75wFFD9D88Fi3UXDKKCx3KtMDCDQQABoMNTA1NTk1Mzc0MzYxIgz9y4RR%2FZSZB4cqKpcqsAOKYIdtPJflZPodH36ke9NbJNTtfbD7E2LujMS7XEyU2sZnwQyTEzsVL%2FJvB3s8Z3eeKhN4xKnkbUAf%2BryO1ig1plAwzjnIpQzauv89AalyJTS7UM%2FymvWnLo2Qx%2Br92owtQCzOZmrREMVqAU0rCZrMv6NWcOH0bkaM%2FmuA6MijF2dfH3mGeW5E0lFqXaJL5OJqclREtJwRomotSljBVoYxoz2PHRvgIKFuezC1reS3XJ3EnWRlbohQzhfGov%2BtOtcUhJIVk8wqXQvyEFOjErapALiIq4oeUBoIpeKXR5iHJDgndnxGvPOG9ihjUMzKwcbEwqQZTSkTK2whRoHcSZgedzWpsQgae3vQjr797H%2Flh7XLSGzHAuy0Bb9zB61ZZRUyKbhODFCw%2BOpk9ihPKedaewtRHe4SYvAYwBcM0%2FZC%2BwcAOOK987eUxnULeHei%2Fs%2BigKBnzGWwojcuqSzdvaRX3yaK9yKG8mT6aMsAYKS%2BjG%2FX%2FlAu3UL4pe7KK2fxHEH1KrQgdaTCsdzaB%2FApfsixRH4XeGOTNk9ZYM2rI70aqMd%2F5fveFO2%2F33EKZSrQg9kw65i9rwY6nQFkK7gFDZd2oCG7e3u3lqDe5%2BQAqBshYAKbOzpMM6VQq0AqLFeO3TWv%2FhlSP8z9Xxadf3IFRciPlQbMV85VXxEDCzyTYAkLQJsqVuyI%2FQF9hW1rk1UcvOTVpcDRXuwBm%2B66Cq48OnTGzdMu32f%2FsKe7aacdi4msVqU2Z4JwvyjewD4biHJ7lNUkAUf16Zz7SKkLvSvxO%2FwqV7HcxrEs&X-Amz-Signature=dc200cb342555a9a40fe50c45fc7433a7f9e6eaadb233a5f2ddd5f5bacf3ef2b&X-Amz-SignedHeaders=content-length%3Bhost%3Bx-amz-checksum-sha256&x-id=PutObject + +curl -X PUT \ +-H "content-length: 26" \ +-H "x-amz-checksum-sha256: sDe+j519nHU7suFDSf2rKkFYaUBv87MHNWaVvNJoLXs=" \ +-H "content-type: application/vnd.ipld.car" \ +--data-binary @big-2.car \ +https://carpark-prod-0.s3.us-west-2.amazonaws.com/bagbaierawa335d45pwohko5s4fbut7nlfjavq2kan7z3gbzvm2k3zutifv5q/bagbaierawa335d45pwohko5s4fbut7nlfjavq2kan7z3gbzvm2k3zutifv5q.car?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=ASIAXLN6VFMMXACSF74D%2F20240311%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20240311T185406Z&X-Amz-Expires=86400&X-Amz-Security-Token=IQoJb3JpZ2luX2VjECsaCXVzLXdlc3QtMiJIMEYCIQCKTNRl%2FrZ47BKRxhGtQfkvjcvPMxBJMWUENt7EL4VYTgIhAO14%2Fs%2Bc3sh2TevvAxplSRT75wFFD9D88Fi3UXDKKCx3KtMDCDQQABoMNTA1NTk1Mzc0MzYxIgz9y4RR%2FZSZB4cqKpcqsAOKYIdtPJflZPodH36ke9NbJNTtfbD7E2LujMS7XEyU2sZnwQyTEzsVL%2FJvB3s8Z3eeKhN4xKnkbUAf%2BryO1ig1plAwzjnIpQzauv89AalyJTS7UM%2FymvWnLo2Qx%2Br92owtQCzOZmrREMVqAU0rCZrMv6NWcOH0bkaM%2FmuA6MijF2dfH3mGeW5E0lFqXaJL5OJqclREtJwRomotSljBVoYxoz2PHRvgIKFuezC1reS3XJ3EnWRlbohQzhfGov%2BtOtcUhJIVk8wqXQvyEFOjErapALiIq4oeUBoIpeKXR5iHJDgndnxGvPOG9ihjUMzKwcbEwqQZTSkTK2whRoHcSZgedzWpsQgae3vQjr797H%2Flh7XLSGzHAuy0Bb9zB61ZZRUyKbhODFCw%2BOpk9ihPKedaewtRHe4SYvAYwBcM0%2FZC%2BwcAOOK987eUxnULeHei%2Fs%2BigKBnzGWwojcuqSzdvaRX3yaK9yKG8mT6aMsAYKS%2BjG%2FX%2FlAu3UL4pe7KK2fxHEH1KrQgdaTCsdzaB%2FApfsixRH4XeGOTNk9ZYM2rI70aqMd%2F5fveFO2%2F33EKZSrQg9kw65i9rwY6nQFkK7gFDZd2oCG7e3u3lqDe5%2BQAqBshYAKbOzpMM6VQq0AqLFeO3TWv%2FhlSP8z9Xxadf3IFRciPlQbMV85VXxEDCzyTYAkLQJsqVuyI%2FQF9hW1rk1UcvOTVpcDRXuwBm%2B66Cq48OnTGzdMu32f%2FsKe7aacdi4msVqU2Z4JwvyjewD4biHJ7lNUkAUf16Zz7SKkLvSvxO%2FwqV7HcxrEs&X-Amz-Signature=232f28f1b4392e185aadc27c50e96ba6101529f7a9cbca8eced582602a9a0e81&X-Amz-SignedHeaders=content-length%3Bhost%3Bx-amz-checksum-sha256&x-id=PutObject +``` + +Once this is all done you'll be able to see your upload in `w3 ls` and through the gateway, eg at https://bafybeicq2urqdqum3ykblswqzzxf6wpy3mcbr6jg52ai2zlhds4l7ho5wa.ipfs.w3s.link From 05f5c6e735fcbb3beea88c0c391f3bf8695f7fb2 Mon Sep 17 00:00:00 2001 From: Travis Vachon Date: Mon, 11 Mar 2024 13:03:07 -0700 Subject: [PATCH 09/15] chore: lint --- src/pages/docs/how-to/http-bridge.mdx | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/src/pages/docs/how-to/http-bridge.mdx b/src/pages/docs/how-to/http-bridge.mdx index 4e59ce6..ac03cae 100644 --- a/src/pages/docs/how-to/http-bridge.mdx +++ b/src/pages/docs/how-to/http-bridge.mdx @@ -1,3 +1,5 @@ +import { Callout } from 'nextra/components' + # Use the HTTP API Bridge Our API is built around [UCAN Invocation](https://github.com/ucan-wg/invocation). This @@ -108,7 +110,7 @@ https://up.web3.storage/bridge ## Storing Files -You can store individual files as big as 5GB in spaces on web3.storage. To store larger files, see the [Sharded Uploads] +You can store individual files as big as 5GB in spaces on web3.storage. To store larger files, see the [Sharded Uploads](#sharded-uploads) section below. To upload a file to your space you'll first need convert it to a CAR. Assuming your file is @@ -217,6 +219,11 @@ curl -v -X PUT \ https://carpark-prod-0.s3.us-west-2.amazonaws.com/bagbaiera4tntawdwlxf33uld7hd5yfaadct5galsr45vawbomjbiytdx4dzq/bagbaiera4tntawdwlxf33uld7hd5yfaadct5galsr45vawbomjbiytdx4dzq.car\?X-Amz-Algorithm\=AWS4-HMAC-SHA256\&X-Amz-Content-Sha256\=UNSIGNED-PAYLOAD\&X-Amz-Credential\=ASIAXLN6VFMM76NYBTG7%2F20240307%2Fus-west-2%2Fs3%2Faws4_request\&X-Amz-Date\=20240307T170854Z\&X-Amz-Expires\=86400\&X-Amz-Security-Token\=IQoJb3JpZ2luX2VjEMn%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLXdlc3QtMiJHMEUCIQC2snXc59uBD2ugjc7C9DiKP0447T7fZ0A010Ph6iDswgIgfq0fYWE4Y9FByHz%2B6mYRussCgVr9KlBV1eGd1kgA22Yq3AMIwv%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARAAGgw1MDU1OTUzNzQzNjEiDCjNsn%2FQBypcXnybPCqwAyGtjDvgxmoQg8IYvO4vQ5fxZw%2F69mzSB%2BJmJMl9gnMaQ7SqUuTQnamPEZHM%2BaztPTbSBFm26S9o4GcZWn5S9UCOLb4YV6sAeCC3XCMkc58RPvyehMiVf0xdN1lBCLkFDd%2BHDb0cPEOvvJbTor22rhCo9L%2FsAVZPAYBu1efwvifCMnUebFeA3Qc0lvYG9PqLtZwo2ZMeVCwXYZiwec%2FpwdobopEWve88GJtxrVyiWMuV4pO7A68cty5ukFr0aRwNHM9g2e2y%2BSymH8pS6mfWpdope%2F%2BB6JoYvKKPN%2FQMpvROl0BxNdTD%2FgNUNqKjlNfUDtV7NqaDJlTqtlTXp44%2F7AUk7yJgK9UlVmp7B30oNH3fPtNXEFBbowoVczqcDBfp%2BxDBMZRFDa37VNcoIdjEBpBTCZk5hONDpjlp%2FyGd1EPSswbQyr6nA5xwLunfF6HhWA%2B8ZMp5raiepAPm%2FS7QfpbiZGtEZ%2FoVWENmPI4HLjDKYjXRJ7i680K9cMDeFSHjHvaTYsgOh09jmL5OnKSMRbANl4bNihk%2FX7pWgxVmbV0Qj%2BQ5eDpvsYUe9i%2FTUVAyODDU5KevBjqeATdV%2Fp9WiSN1mc1%2FUxn60TYZRFAjcTm6lfdbdtEEWneexsLLj6BpoW6gUA1c73JjEXsOEGbaSfHXi%2BI6nFtlNJpKlP%2BbrFlylA2qmIvlnKJzv2ALrsARUmCH1Kl26CVFfOtnJv7VZmDSVnS9nr4mYLHLkPjIhBaF3IxWVcFR7He0CCfRv%2BYkHzTlVdzoK%2BmJkUVyofY%2B3Bt3dDkpk7No\&X-Amz-Signature\=1d4cbc65a8440c89eb2770b2edfe4e3fe91831856ee03d0d939911d9963337f0\&X-Amz-SignedHeaders\=content-length%3Bhost%3Bx-amz-checksum-sha256\&x-id\=PutObject ``` + +Please note that your storage space has been allocated at this point - if you fail to `PUT` the file you will still be charged for storage! +The good news is that you can store the same file as many times as you like without being charged for any additional space. + + Your bits are now on IPFS, en route to the Filecoin Network, _but they will not show up as uploads in your web3.storage account!_ To finish "uploading" your file to web3.storage you'll need to register your upload as described in the next section. @@ -396,12 +403,12 @@ to the given URL with the given headers. If you'd like to store a file larger than 5GB you'll need to shard it into smaller CARs before storing. We don't proscribe any particular sharding algorithm - you'll need to: -1) read the input CAR -2) write the first CAR file header -3) write blocks until the first CAR hits your limit (a number lower than 5GB) -4) write a new CAR file header for the second CAR -5) write blocks until the second CAR hits your limit -6) repeat until end of blocks +1. read the input CAR +2. write the first CAR file header +3. write blocks until the first CAR hits your limit (a number lower than 5GB) +4. write a new CAR file header for the second CAR +5. write blocks until the second CAR hits your limit +6. repeat until end of blocks If you're implementing this yourself it may be helpful to look into [our JavaScript implementation](https://github.com/web3-storage/w3up/blob/d6978d7ab299be76987c6533d18e6857f6998fe6/packages/upload-client/src/index.js#L38). @@ -417,7 +424,7 @@ bafybeicq2urqdqum3ykblswqzzxf6wpy3mcbr6jg52ai2zlhds4l7ho5wa We'll save the CID `ipfs-car` outputs so we can use it in our `store/add` invocation later. -Next we'll use `carbites` to split this file. We are using the `50B` splitting into 50 byte +Next we'll use `carbites` to split this file. We are using the `50B` option to split into 50 byte chunks - you'll probably want to use something much larger like `100MB`: ```sh From d769276251467452aae4e2a5347ced89809e07b1 Mon Sep 17 00:00:00 2001 From: Travis Vachon Date: Mon, 11 Mar 2024 13:11:05 -0700 Subject: [PATCH 10/15] fix: two more tweaks from PR --- src/pages/docs/how-to/http-bridge.mdx | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/src/pages/docs/how-to/http-bridge.mdx b/src/pages/docs/how-to/http-bridge.mdx index ac03cae..e71fc37 100644 --- a/src/pages/docs/how-to/http-bridge.mdx +++ b/src/pages/docs/how-to/http-bridge.mdx @@ -110,8 +110,8 @@ https://up.web3.storage/bridge ## Storing Files -You can store individual files as big as 5GB in spaces on web3.storage. To store larger files, see the [Sharded Uploads](#sharded-uploads) -section below. +You can store individual files as [big as 5GB](https://github.com/web3-storage/w3infra/blob/26106e315e79bf8072d2bd052786e46563d9a1da/upload-api/service.js#L12-L16) in +spaces on web3.storage. To store larger files, see the [Sharded Uploads](#sharded-uploads) section below. To upload a file to your space you'll first need convert it to a CAR. Assuming your file is named `hello.txt` you can first use `ipfs-car` to create a CAR: @@ -121,6 +121,14 @@ ipfs-car pack hello.txt > hello.car bafybeidhkumeonuwkebh2i4fc7o7lguehauradvlk57gzake6ggjsy372a ``` + +If you already have a file available on the public IPFS network by CID, you can turn it into a CAR that can be uploaded to our service with: + +```sh +ipfs dag export bafybeidhkumeonuwkebh2i4fc7o7lguehauradvlk57gzake6ggjsy372a > hello.car +``` + + Hang on to the `bafy...` Content ID output here! You'll need it for the "Registering Uploads" section below. Next, calculate the Content ID of the CAR you just created: From 658f1179b5ed1bbdbca9f703d7a56ef2b7d7417d Mon Sep 17 00:00:00 2001 From: Travis Vachon Date: Mon, 11 Mar 2024 13:28:08 -0700 Subject: [PATCH 11/15] feat: Add more information about "supported" capabilities --- src/pages/docs/how-to/http-bridge.mdx | 24 +++++++++++++++++++----- 1 file changed, 19 insertions(+), 5 deletions(-) diff --git a/src/pages/docs/how-to/http-bridge.mdx b/src/pages/docs/how-to/http-bridge.mdx index e71fc37..465d167 100644 --- a/src/pages/docs/how-to/http-bridge.mdx +++ b/src/pages/docs/how-to/http-bridge.mdx @@ -14,11 +14,25 @@ used an HTTP-based API before. ## Limitations -The HTTP Bridge is currently designed to be used to invoke capabilities on -spaces. `store/add`, `upload/add` and `upload/list` are examples of such capabilities. Other -capabilities like `access/request` are not supported at this time - please let us know -if you would like to invoke other capabilities using the bridge so that we can -prioritize adding support for them. +The instructions below are oriented toward capabilities invoked on spaces. These include: + +| `access/delegate` | +| `space/info` | +| `space/allocate` | +| `store/add` | +| `store/get` | +| `store/remove` | +| `store/list` | +| `upload/add` | +| `upload/list` | +| `upload/remove` | +| `usage/report` | + + +Other capabilities are invoked on other entities. They are supported by the bridge, but +you will need to generate auth tokens that are targetted at different DIDs and may not +be well supported by our tools - please let us know if you are interested in this functionality +so we can better support your use cases! ## Authentication and Authorization From 1f115033bdb39e634aa94ca66a49f18592069246 Mon Sep 17 00:00:00 2001 From: Travis Vachon Date: Mon, 11 Mar 2024 15:48:00 -0700 Subject: [PATCH 12/15] fix: formatting of capability list --- src/pages/docs/how-to/http-bridge.mdx | 17 +++-------------- 1 file changed, 3 insertions(+), 14 deletions(-) diff --git a/src/pages/docs/how-to/http-bridge.mdx b/src/pages/docs/how-to/http-bridge.mdx index 465d167..a0b7de7 100644 --- a/src/pages/docs/how-to/http-bridge.mdx +++ b/src/pages/docs/how-to/http-bridge.mdx @@ -14,20 +14,9 @@ used an HTTP-based API before. ## Limitations -The instructions below are oriented toward capabilities invoked on spaces. These include: - -| `access/delegate` | -| `space/info` | -| `space/allocate` | -| `store/add` | -| `store/get` | -| `store/remove` | -| `store/list` | -| `upload/add` | -| `upload/list` | -| `upload/remove` | -| `usage/report` | - +The instructions below are oriented toward capabilities invoked on spaces. These include `access/delegate`, +`space/info`, `space/allocate`, `store/add`, `store/get`, `store/remove`, `store/list`, `upload/add`, +`upload/list`, `upload/remove` and `usage/report`. Other capabilities are invoked on other entities. They are supported by the bridge, but you will need to generate auth tokens that are targetted at different DIDs and may not From 5ca5b103dd50d2b65319d0cd1c74147f7620424e Mon Sep 17 00:00:00 2001 From: Travis Vachon Date: Thu, 14 Mar 2024 10:00:02 -0700 Subject: [PATCH 13/15] Apply suggestions from code review Co-authored-by: Alan Shaw --- src/pages/docs/how-to/http-bridge.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/pages/docs/how-to/http-bridge.mdx b/src/pages/docs/how-to/http-bridge.mdx index a0b7de7..2054271 100644 --- a/src/pages/docs/how-to/http-bridge.mdx +++ b/src/pages/docs/how-to/http-bridge.mdx @@ -117,7 +117,7 @@ You can store individual files as [big as 5GB](https://github.com/web3-storage/w spaces on web3.storage. To store larger files, see the [Sharded Uploads](#sharded-uploads) section below. To upload a file to your space you'll first need convert it to a CAR. Assuming your file is -named `hello.txt` you can first use `ipfs-car` to create a CAR: +named `hello.txt` you can first use [`ipfs-car`](https://github.com/web3-storage/ipfs-car) to create a CAR: ```sh ipfs-car pack hello.txt > hello.car @@ -412,7 +412,7 @@ to the given URL with the given headers. ## Sharded Uploads If you'd like to store a file larger than 5GB you'll need to shard it into smaller CARs before storing. We don't -proscribe any particular sharding algorithm - you'll need to: +prescribe any particular sharding algorithm - you'll need to: 1. read the input CAR 2. write the first CAR file header From 17d5872d07e965c237232108794b635c9ad196c6 Mon Sep 17 00:00:00 2001 From: Travis Vachon Date: Thu, 14 Mar 2024 10:22:18 -0700 Subject: [PATCH 14/15] Update src/pages/docs/how-to/http-bridge.mdx Co-authored-by: Alan Shaw --- src/pages/docs/how-to/http-bridge.mdx | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/src/pages/docs/how-to/http-bridge.mdx b/src/pages/docs/how-to/http-bridge.mdx index 2054271..9a03fcf 100644 --- a/src/pages/docs/how-to/http-bridge.mdx +++ b/src/pages/docs/how-to/http-bridge.mdx @@ -445,11 +445,11 @@ carbites split big.car --size 50B You should now see three files, `big-0.car`, `big-1.car` and `big-2.car`. Calculate the CID and size of each one: ```sh -wc big-* - 1 12 109 big-0.car - 2 7 118 big-1.car - 0 1 26 big-2.car - 3 20 253 total +wc -c big-* + 109 big-0.car + 118 big-1.car + 26 big-2.car + 253 total ipfs-car hash big-0.car bagbaiera5pcs7vgks34rflh47c6cprzoift7vfch6gmbm3odq7auw2iou3cq ipfs-car hash big-1.car From 07a4608216412be81d14be7b9b90bc893312f349 Mon Sep 17 00:00:00 2001 From: Travis Vachon Date: Thu, 14 Mar 2024 11:10:37 -0700 Subject: [PATCH 15/15] fix: update sharding example to use a really big file the small file was funny but this is more realistic --- src/pages/docs/how-to/http-bridge.mdx | 103 +++++++++++++++----------- 1 file changed, 59 insertions(+), 44 deletions(-) diff --git a/src/pages/docs/how-to/http-bridge.mdx b/src/pages/docs/how-to/http-bridge.mdx index 9a03fcf..9a8ebb6 100644 --- a/src/pages/docs/how-to/http-bridge.mdx +++ b/src/pages/docs/how-to/http-bridge.mdx @@ -428,9 +428,9 @@ We provide an example here using the [`carbites-cli`](https://www.npmjs.com/pack First we'll create an absolutely gigantic file: ```sh -echo "this is an absolutely gigantic file created by travis" > big.txt +printf -- 'this is an absolutely gigantic file created by travis%.0s' {1..1000000} > big.txt ipfs-car pack big.txt > big.car -bafybeicq2urqdqum3ykblswqzzxf6wpy3mcbr6jg52ai2zlhds4l7ho5wa +bafybeiao5ir3xsp4vhg46b2cjvnl4ucmg5wxrdpy4cypolwexitmpz7hwu ``` We'll save the CID `ipfs-car` outputs so we can use it in our `store/add` invocation later. @@ -439,23 +439,23 @@ Next we'll use `carbites` to split this file. We are using the `50B` option to s chunks - you'll probably want to use something much larger like `100MB`: ```sh -carbites split big.car --size 50B +carbites split big.car --size 25MB ``` You should now see three files, `big-0.car`, `big-1.car` and `big-2.car`. Calculate the CID and size of each one: ```sh wc -c big-* - 109 big-0.car - 118 big-1.car - 26 big-2.car - 253 total + 26218318 big-0.car + 26227074 big-1.car + 573086 big-2.car + 53018478 total ipfs-car hash big-0.car -bagbaiera5pcs7vgks34rflh47c6cprzoift7vfch6gmbm3odq7auw2iou3cq +bagbaierau4qyqlx3bsbgeprw4wzbxc4c5olajasbcdcqk3aetsakieywf6cq ipfs-car hash big-1.car -bagbaierau2ocmnrl5ilikq5xtxckz3owbe5xallbx24fkvislzu7k7ydqzia +bagbaiera6cg7n32box5trcbif2v5btozvbze2rg5mx2cgwff7643zrpreuua ipfs-car hash big-2.car -bagbaierawa335d45pwohko5s4fbut7nlfjavq2kan7z3gbzvm2k3zutifv5q +bagbaiera5ommidzo57zza4ssqx2orgi25gcnfsucsry2nwmt2kd352baujoa ``` Next, create a file (in this example we'll name it `store-and-upload.json`) and put the following JSON @@ -472,7 +472,7 @@ the space you used when you generated auth tokens! "did:key:z6Mkabc123", { "link": { - "/": "bagbaiera5pcs7vgks34rflh47c6cprzoift7vfch6gmbm3odq7auw2iou3cq" + "/": "bagbaierau4qyqlx3bsbgeprw4wzbxc4c5olajasbcdcqk3aetsakieywf6cq" }, "size": 109 } @@ -482,7 +482,7 @@ the space you used when you generated auth tokens! "did:key:z6Mkabc123", { "link": { - "/": "bagbaierau2ocmnrl5ilikq5xtxckz3owbe5xallbx24fkvislzu7k7ydqzia" + "/": "bagbaiera6cg7n32box5trcbif2v5btozvbze2rg5mx2cgwff7643zrpreuua" }, "size": 118 } @@ -492,7 +492,7 @@ the space you used when you generated auth tokens! "did:key:z6Mkabc123", { "link": { - "/": "bagbaierawa335d45pwohko5s4fbut7nlfjavq2kan7z3gbzvm2k3zutifv5q" + "/": "bagbaiera5ommidzo57zza4ssqx2orgi25gcnfsucsry2nwmt2kd352baujoa" }, "size": 26 } @@ -502,17 +502,17 @@ the space you used when you generated auth tokens! "did:key:z6Mkabc123", { "root": { - "/": "bafybeicq2urqdqum3ykblswqzzxf6wpy3mcbr6jg52ai2zlhds4l7ho5wa" + "/": "bafybeiao5ir3xsp4vhg46b2cjvnl4ucmg5wxrdpy4cypolwexitmpz7hwu" }, "shards": [ { - "/": "bagbaiera5pcs7vgks34rflh47c6cprzoift7vfch6gmbm3odq7auw2iou3cq" + "/": "bagbaierau4qyqlx3bsbgeprw4wzbxc4c5olajasbcdcqk3aetsakieywf6cq" }, { - "/": "bagbaierau2ocmnrl5ilikq5xtxckz3owbe5xallbx24fkvislzu7k7ydqzia" + "/": "bagbaiera6cg7n32box5trcbif2v5btozvbze2rg5mx2cgwff7643zrpreuua" }, { - "/": "bagbaierawa335d45pwohko5s4fbut7nlfjavq2kan7z3gbzvm2k3zutifv5q" + "/": "bagbaiera5ommidzo57zza4ssqx2orgi25gcnfsucsry2nwmt2kd352baujoa" } ] } @@ -535,21 +535,26 @@ Once you receive a response, you'll need to `PUT` each of the cars to their resp "out": { "ok": { "allocated": 0, + "headers": { + "content-length": "26218318", + "x-amz-checksum-sha256": "pyGILvsMgmI+NuWyG4uC65YEgkEQxQVsBJyApBMWL4U=" + }, "link": { - "/": "bagbaiera5pcs7vgks34rflh47c6cprzoift7vfch6gmbm3odq7auw2iou3cq" + "/": "bagbaierau4qyqlx3bsbgeprw4wzbxc4c5olajasbcdcqk3aetsakieywf6cq" }, - "status": "done", + "status": "upload", + "url": "https://carpark-prod-0.s3.us-west-2.amazonaws.com/bagbaierau4qyqlx3bsbgeprw4wzbxc4c5olajasbcdcqk3aetsakieywf6cq/bagbaierau4qyqlx3bsbgeprw4wzbxc4c5olajasbcdcqk3aetsakieywf6cq.car?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=ASIAXLN6VFMM665ZYAHR%2F20240314%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20240314T175533Z&X-Amz-Expires=86400&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEHEaCXVzLXdlc3QtMiJHMEUCIQCUXvGoqROkM7v%2FLVCivP8q5dqoiRe4dbXtqTCV3MP9TAIgKGTZvwjmQ365IcVthNaMXHytEQmm74ABJmUKxFSon14q0wMIehABGgw1MDU1OTUzNzQzNjEiDD3M45t0320FZ6VOriqwA422dYlN9z%2F89MSJjGZQitkOPlF8wdp4t4kU3I4wvjyThjkfSGeyBV20XS%2FfsF8Dcm1L3iuZwUwhMZImvX406MZDzEYLKlyejvoDPQLpGOsopDfLOygdFUUQUvsbZPXvgJwyZp5r5YSUUUEOhmafdGHztdQVzqkiKaQ%2BUrXA1KAHb7qEXlM3NvPAyDNO0GJD8cVDUlygg%2Bw6mZodsjeIt%2B250MRvDU0DL%2BRPIG1OhCM81GMgnkmGpyTh4pSuZPPUqINRm7qKCssk0Fg84isHOCMa7OEonwoAccIHBS%2BQB8XtuvQ%2FQWDZeQUYxFFhvXKQu5tze9ZOHxFC%2FOAvmpoAQklXXzWeySKplID5AihmbCiNqisqqjQ9MuTuJmNfHHEIqFfKx00Rdxkl42IWNp8gtzyjTSjytWbEd%2Bm6yStClUvFo3i4pwn3zvCgZ3aV5NjWuOufQpAnU7Tb7lK02Q0JikIOz5hHHtOFLh5%2F8ldnKGaBp0qs9Nj1E2dEFU%2B%2BuyZ3bx6l1MrVknVhPsWUqCoEmT9me%2BJXKGFUV0hPjDKJNrPEtMLDbFoaW3yIh3hwVR9MWzCX3syvBjqeAZDeWPQKO8Xv%2B%2Fti9cMOU6R7EtRbJ1hHmz95u9SJLKD6E%2F19RQnPFj5f4TcDyoqt4UT6p8fRcK9xkAVg2e1qamlLmtEMEhgavD6YgqSLbEPpRe9CitthIZ9jyPvZYtxccY%2BhUoQSeSAQ11JHTRtYzsjzOs%2BcgXTSDRAaj6ZnCcLpEYgRt7hRO%2BqIhDdUNxbUenM1edJhEW%2FyKBn%2BEfvm&X-Amz-Signature=28ad621b458a783fe69bf50bea52b8cfd9421127873e516bf237863e74c124dd&X-Amz-SignedHeaders=content-length%3Bhost%3Bx-amz-checksum-sha256&x-id=PutObject", "with": "did:key:z6Mkabc123" } }, "prf": [], "ran": { - "/": "bafyreia526axkta24us4fxjghfqbwu2d7lk2neucrseoapakkcmgoy6x7u" + "/": "bafyreifl635eqromltu6asqekh3gmdxiiasubcuvpc37j5ticopvjwmeqi" } }, "s": { "/": { - "bytes": "7aEDQN9dWcLsWIwCMOLmzXP971VkJTU1hXliFpEpZZAYQhmOzReViM4TE4tp1AS4vwLcXzawiwaWJ+bkzvh2DKdx4go" + "bytes": "7aEDQJS7nJycSU2r7941+5dbhqCuGbHsm2GXKbRggayUDxpfukNwJSyNR1GydX6pgqhfhLFCutlx3yKPOUH+/eoxtwg" } } }, @@ -563,21 +568,26 @@ Once you receive a response, you'll need to `PUT` each of the cars to their resp "out": { "ok": { "allocated": 0, + "headers": { + "content-length": "26227074", + "x-amz-checksum-sha256": "8I3270F1+ziIKC6r0M3ZqHJNRN1l9CNYpf+5vMXxJSg=" + }, "link": { - "/": "bagbaierau2ocmnrl5ilikq5xtxckz3owbe5xallbx24fkvislzu7k7ydqzia" + "/": "bagbaiera6cg7n32box5trcbif2v5btozvbze2rg5mx2cgwff7643zrpreuua" }, - "status": "done", + "status": "upload", + "url": "https://carpark-prod-0.s3.us-west-2.amazonaws.com/bagbaiera6cg7n32box5trcbif2v5btozvbze2rg5mx2cgwff7643zrpreuua/bagbaiera6cg7n32box5trcbif2v5btozvbze2rg5mx2cgwff7643zrpreuua.car?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=ASIAXLN6VFMM665ZYAHR%2F20240314%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20240314T175533Z&X-Amz-Expires=86400&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEHEaCXVzLXdlc3QtMiJHMEUCIQCUXvGoqROkM7v%2FLVCivP8q5dqoiRe4dbXtqTCV3MP9TAIgKGTZvwjmQ365IcVthNaMXHytEQmm74ABJmUKxFSon14q0wMIehABGgw1MDU1OTUzNzQzNjEiDD3M45t0320FZ6VOriqwA422dYlN9z%2F89MSJjGZQitkOPlF8wdp4t4kU3I4wvjyThjkfSGeyBV20XS%2FfsF8Dcm1L3iuZwUwhMZImvX406MZDzEYLKlyejvoDPQLpGOsopDfLOygdFUUQUvsbZPXvgJwyZp5r5YSUUUEOhmafdGHztdQVzqkiKaQ%2BUrXA1KAHb7qEXlM3NvPAyDNO0GJD8cVDUlygg%2Bw6mZodsjeIt%2B250MRvDU0DL%2BRPIG1OhCM81GMgnkmGpyTh4pSuZPPUqINRm7qKCssk0Fg84isHOCMa7OEonwoAccIHBS%2BQB8XtuvQ%2FQWDZeQUYxFFhvXKQu5tze9ZOHxFC%2FOAvmpoAQklXXzWeySKplID5AihmbCiNqisqqjQ9MuTuJmNfHHEIqFfKx00Rdxkl42IWNp8gtzyjTSjytWbEd%2Bm6yStClUvFo3i4pwn3zvCgZ3aV5NjWuOufQpAnU7Tb7lK02Q0JikIOz5hHHtOFLh5%2F8ldnKGaBp0qs9Nj1E2dEFU%2B%2BuyZ3bx6l1MrVknVhPsWUqCoEmT9me%2BJXKGFUV0hPjDKJNrPEtMLDbFoaW3yIh3hwVR9MWzCX3syvBjqeAZDeWPQKO8Xv%2B%2Fti9cMOU6R7EtRbJ1hHmz95u9SJLKD6E%2F19RQnPFj5f4TcDyoqt4UT6p8fRcK9xkAVg2e1qamlLmtEMEhgavD6YgqSLbEPpRe9CitthIZ9jyPvZYtxccY%2BhUoQSeSAQ11JHTRtYzsjzOs%2BcgXTSDRAaj6ZnCcLpEYgRt7hRO%2BqIhDdUNxbUenM1edJhEW%2FyKBn%2BEfvm&X-Amz-Signature=f47614b5ec4751ec5b46617d172f86daf91bda5a8c15b771e7fb488d15394a4e&X-Amz-SignedHeaders=content-length%3Bhost%3Bx-amz-checksum-sha256&x-id=PutObject", "with": "did:key:z6Mkabc123" } }, "prf": [], "ran": { - "/": "bafyreidi655w32on25adjxnoxro3dnqcsncqly62uc6jyz6l25r45bd5lu" + "/": "bafyreihskuuxwtluxs2pndfjqr56jqlwfgpfoga7teitmbsbgzk466wbwa" } }, "s": { "/": { - "bytes": "7aEDQALj9TzBJdqsXs8RY5NoIwy63wvALLw/U1gRcOkBp8VBoIqwEuL3xjjvr0RCnXzLoPQGszTP/xp/A51BJ0mlYQs" + "bytes": "7aEDQI6K8xBybt+lthsCEfMjxbql7lml8XUeUeSxZsnF2duxuK7YKjEcETTr3gmPOARP0ulgr1yDOvWbUcKmumyIWQo" } } }, @@ -591,21 +601,26 @@ Once you receive a response, you'll need to `PUT` each of the cars to their resp "out": { "ok": { "allocated": 0, + "headers": { + "content-length": "573086", + "x-amz-checksum-sha256": "65jEDy7v85ByUoX06Jka6YTSyoKUcabZk9KHvuggolw=" + }, "link": { - "/": "bagbaierawa335d45pwohko5s4fbut7nlfjavq2kan7z3gbzvm2k3zutifv5q" + "/": "bagbaiera5ommidzo57zza4ssqx2orgi25gcnfsucsry2nwmt2kd352baujoa" }, - "status": "done", + "status": "upload", + "url": "https://carpark-prod-0.s3.us-west-2.amazonaws.com/bagbaiera5ommidzo57zza4ssqx2orgi25gcnfsucsry2nwmt2kd352baujoa/bagbaiera5ommidzo57zza4ssqx2orgi25gcnfsucsry2nwmt2kd352baujoa.car?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=ASIAXLN6VFMM665ZYAHR%2F20240314%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20240314T175533Z&X-Amz-Expires=86400&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEHEaCXVzLXdlc3QtMiJHMEUCIQCUXvGoqROkM7v%2FLVCivP8q5dqoiRe4dbXtqTCV3MP9TAIgKGTZvwjmQ365IcVthNaMXHytEQmm74ABJmUKxFSon14q0wMIehABGgw1MDU1OTUzNzQzNjEiDD3M45t0320FZ6VOriqwA422dYlN9z%2F89MSJjGZQitkOPlF8wdp4t4kU3I4wvjyThjkfSGeyBV20XS%2FfsF8Dcm1L3iuZwUwhMZImvX406MZDzEYLKlyejvoDPQLpGOsopDfLOygdFUUQUvsbZPXvgJwyZp5r5YSUUUEOhmafdGHztdQVzqkiKaQ%2BUrXA1KAHb7qEXlM3NvPAyDNO0GJD8cVDUlygg%2Bw6mZodsjeIt%2B250MRvDU0DL%2BRPIG1OhCM81GMgnkmGpyTh4pSuZPPUqINRm7qKCssk0Fg84isHOCMa7OEonwoAccIHBS%2BQB8XtuvQ%2FQWDZeQUYxFFhvXKQu5tze9ZOHxFC%2FOAvmpoAQklXXzWeySKplID5AihmbCiNqisqqjQ9MuTuJmNfHHEIqFfKx00Rdxkl42IWNp8gtzyjTSjytWbEd%2Bm6yStClUvFo3i4pwn3zvCgZ3aV5NjWuOufQpAnU7Tb7lK02Q0JikIOz5hHHtOFLh5%2F8ldnKGaBp0qs9Nj1E2dEFU%2B%2BuyZ3bx6l1MrVknVhPsWUqCoEmT9me%2BJXKGFUV0hPjDKJNrPEtMLDbFoaW3yIh3hwVR9MWzCX3syvBjqeAZDeWPQKO8Xv%2B%2Fti9cMOU6R7EtRbJ1hHmz95u9SJLKD6E%2F19RQnPFj5f4TcDyoqt4UT6p8fRcK9xkAVg2e1qamlLmtEMEhgavD6YgqSLbEPpRe9CitthIZ9jyPvZYtxccY%2BhUoQSeSAQ11JHTRtYzsjzOs%2BcgXTSDRAaj6ZnCcLpEYgRt7hRO%2BqIhDdUNxbUenM1edJhEW%2FyKBn%2BEfvm&X-Amz-Signature=5792f7d2847ab963c54e34c44c7f821ccded0baf951d2be262be1a2e9c59101f&X-Amz-SignedHeaders=content-length%3Bhost%3Bx-amz-checksum-sha256&x-id=PutObject", "with": "did:key:z6Mkabc123" } }, "prf": [], "ran": { - "/": "bafyreic5wcfzvafpkt2haz77ya7zq5ka5ort73i4ahq2mvenj7zssyy6eq" + "/": "bafyreidf7aj7tmrf54rkh2hrartvbbsg5ojbg2dmhnluvzrliyzq4o2cdi" } }, "s": { "/": { - "bytes": "7aEDQCdO8OKGjxTpEgsQaJV0ZEoFBfXw6mZGWx+gMUPxHMFmiO8AVrzyNXXGP1dy0q16yI+6bnnLpuMC684/n5dZQAU" + "bytes": "7aEDQBiyN3Nxpc+upjyfyHABtfscOjECEE9jXTG9H4R2/jOjlJrR39VPv2IkBFIBtbg0hPHs8I/XGftG1dl+iHur4wo" } } }, @@ -619,29 +634,29 @@ Once you receive a response, you'll need to `PUT` each of the cars to their resp "out": { "ok": { "root": { - "/": "bafybeicq2urqdqum3ykblswqzzxf6wpy3mcbr6jg52ai2zlhds4l7ho5wa" + "/": "bafybeiao5ir3xsp4vhg46b2cjvnl4ucmg5wxrdpy4cypolwexitmpz7hwu" }, "shards": [ { - "/": "bagbaiera5pcs7vgks34rflh47c6cprzoift7vfch6gmbm3odq7auw2iou3cq" + "/": "bagbaiera5ommidzo57zza4ssqx2orgi25gcnfsucsry2nwmt2kd352baujoa" }, { - "/": "bagbaierau2ocmnrl5ilikq5xtxckz3owbe5xallbx24fkvislzu7k7ydqzia" + "/": "bagbaiera6cg7n32box5trcbif2v5btozvbze2rg5mx2cgwff7643zrpreuua" }, { - "/": "bagbaierawa335d45pwohko5s4fbut7nlfjavq2kan7z3gbzvm2k3zutifv5q" + "/": "bagbaierau4qyqlx3bsbgeprw4wzbxc4c5olajasbcdcqk3aetsakieywf6cq" } ] } }, "prf": [], "ran": { - "/": "bafyreicirpydnj66okf2bqmuv4qbdgxggxqmj5ybr3pyngcp3iqdwckc6e" + "/": "bafyreigl6rzvfcba47bj4eapo6csalvrmweq67dhejqmmoqw45rlemitnq" } }, "s": { "/": { - "bytes": "7aEDQHJEvkeOSIkKQ0SaMbEgNXjLOuwNPWg047O08pCgRSSQMKrM3B++k6Qs49m/xSxpswY1FxuA/dQuh1NWrhqTdAY" + "bytes": "7aEDQBypjHmf0ROjk7VzXM3FDtEybD8c2ykGXkrBMjzXyNP4koXC9XWRKaeKYSTb8JK+8jpIDHnW/n0gMjEUihIgsA8" } } } @@ -652,25 +667,25 @@ You'll need to run the following `curl` commands: ```sh curl -X PUT \ --H "content-length: 109" \ --H "x-amz-checksum-sha256: 68Uv1MqW+RKs/Pi8J8cuQWf6lEfxmBZtw4fBS2kOpsU=" \ +-H "content-length: 26218318" \ +-H "x-amz-checksum-sha256: pyGILvsMgmI+NuWyG4uC65YEgkEQxQVsBJyApBMWL4U=" \ -H "content-type: application/vnd.ipld.car" \ --data-binary @big-0.car \ -https://carpark-prod-0.s3.us-west-2.amazonaws.com/bagbaiera5pcs7vgks34rflh47c6cprzoift7vfch6gmbm3odq7auw2iou3cq/bagbaiera5pcs7vgks34rflh47c6cprzoift7vfch6gmbm3odq7auw2iou3cq.car?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=ASIAXLN6VFMMXACSF74D%2F20240311%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20240311T185406Z&X-Amz-Expires=86400&X-Amz-Security-Token=IQoJb3JpZ2luX2VjECsaCXVzLXdlc3QtMiJIMEYCIQCKTNRl%2FrZ47BKRxhGtQfkvjcvPMxBJMWUENt7EL4VYTgIhAO14%2Fs%2Bc3sh2TevvAxplSRT75wFFD9D88Fi3UXDKKCx3KtMDCDQQABoMNTA1NTk1Mzc0MzYxIgz9y4RR%2FZSZB4cqKpcqsAOKYIdtPJflZPodH36ke9NbJNTtfbD7E2LujMS7XEyU2sZnwQyTEzsVL%2FJvB3s8Z3eeKhN4xKnkbUAf%2BryO1ig1plAwzjnIpQzauv89AalyJTS7UM%2FymvWnLo2Qx%2Br92owtQCzOZmrREMVqAU0rCZrMv6NWcOH0bkaM%2FmuA6MijF2dfH3mGeW5E0lFqXaJL5OJqclREtJwRomotSljBVoYxoz2PHRvgIKFuezC1reS3XJ3EnWRlbohQzhfGov%2BtOtcUhJIVk8wqXQvyEFOjErapALiIq4oeUBoIpeKXR5iHJDgndnxGvPOG9ihjUMzKwcbEwqQZTSkTK2whRoHcSZgedzWpsQgae3vQjr797H%2Flh7XLSGzHAuy0Bb9zB61ZZRUyKbhODFCw%2BOpk9ihPKedaewtRHe4SYvAYwBcM0%2FZC%2BwcAOOK987eUxnULeHei%2Fs%2BigKBnzGWwojcuqSzdvaRX3yaK9yKG8mT6aMsAYKS%2BjG%2FX%2FlAu3UL4pe7KK2fxHEH1KrQgdaTCsdzaB%2FApfsixRH4XeGOTNk9ZYM2rI70aqMd%2F5fveFO2%2F33EKZSrQg9kw65i9rwY6nQFkK7gFDZd2oCG7e3u3lqDe5%2BQAqBshYAKbOzpMM6VQq0AqLFeO3TWv%2FhlSP8z9Xxadf3IFRciPlQbMV85VXxEDCzyTYAkLQJsqVuyI%2FQF9hW1rk1UcvOTVpcDRXuwBm%2B66Cq48OnTGzdMu32f%2FsKe7aacdi4msVqU2Z4JwvyjewD4biHJ7lNUkAUf16Zz7SKkLvSvxO%2FwqV7HcxrEs&X-Amz-Signature=5ce3660c1fc5364457e8f3732b759b9c17b3336011e0a04c119cb44bc56913ed&X-Amz-SignedHeaders=content-length%3Bhost%3Bx-amz-checksum-sha256&x-id=PutObject +https://carpark-prod-0.s3.us-west-2.amazonaws.com/bagbaierau4qyqlx3bsbgeprw4wzbxc4c5olajasbcdcqk3aetsakieywf6cq/bagbaierau4qyqlx3bsbgeprw4wzbxc4c5olajasbcdcqk3aetsakieywf6cq.car?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=ASIAXLN6VFMM665ZYAHR%2F20240314%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20240314T175533Z&X-Amz-Expires=86400&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEHEaCXVzLXdlc3QtMiJHMEUCIQCUXvGoqROkM7v%2FLVCivP8q5dqoiRe4dbXtqTCV3MP9TAIgKGTZvwjmQ365IcVthNaMXHytEQmm74ABJmUKxFSon14q0wMIehABGgw1MDU1OTUzNzQzNjEiDD3M45t0320FZ6VOriqwA422dYlN9z%2F89MSJjGZQitkOPlF8wdp4t4kU3I4wvjyThjkfSGeyBV20XS%2FfsF8Dcm1L3iuZwUwhMZImvX406MZDzEYLKlyejvoDPQLpGOsopDfLOygdFUUQUvsbZPXvgJwyZp5r5YSUUUEOhmafdGHztdQVzqkiKaQ%2BUrXA1KAHb7qEXlM3NvPAyDNO0GJD8cVDUlygg%2Bw6mZodsjeIt%2B250MRvDU0DL%2BRPIG1OhCM81GMgnkmGpyTh4pSuZPPUqINRm7qKCssk0Fg84isHOCMa7OEonwoAccIHBS%2BQB8XtuvQ%2FQWDZeQUYxFFhvXKQu5tze9ZOHxFC%2FOAvmpoAQklXXzWeySKplID5AihmbCiNqisqqjQ9MuTuJmNfHHEIqFfKx00Rdxkl42IWNp8gtzyjTSjytWbEd%2Bm6yStClUvFo3i4pwn3zvCgZ3aV5NjWuOufQpAnU7Tb7lK02Q0JikIOz5hHHtOFLh5%2F8ldnKGaBp0qs9Nj1E2dEFU%2B%2BuyZ3bx6l1MrVknVhPsWUqCoEmT9me%2BJXKGFUV0hPjDKJNrPEtMLDbFoaW3yIh3hwVR9MWzCX3syvBjqeAZDeWPQKO8Xv%2B%2Fti9cMOU6R7EtRbJ1hHmz95u9SJLKD6E%2F19RQnPFj5f4TcDyoqt4UT6p8fRcK9xkAVg2e1qamlLmtEMEhgavD6YgqSLbEPpRe9CitthIZ9jyPvZYtxccY%2BhUoQSeSAQ11JHTRtYzsjzOs%2BcgXTSDRAaj6ZnCcLpEYgRt7hRO%2BqIhDdUNxbUenM1edJhEW%2FyKBn%2BEfvm&X-Amz-Signature=28ad621b458a783fe69bf50bea52b8cfd9421127873e516bf237863e74c124dd&X-Amz-SignedHeaders=content-length%3Bhost%3Bx-amz-checksum-sha256&x-id=PutObject curl -X PUT \ --H "content-length: 118" \ --H "x-amz-checksum-sha256: ppwmNivqFoVDt53ErO3WCTtwLWG+uFVVEl5p9X8DhlA=" \ +-H "content-length: 26227074" \ +-H "x-amz-checksum-sha256: 8I3270F1+ziIKC6r0M3ZqHJNRN1l9CNYpf+5vMXxJSg=" \ -H "content-type: application/vnd.ipld.car" \ --data-binary @big-1.car \ -https://carpark-prod-0.s3.us-west-2.amazonaws.com/bagbaierau2ocmnrl5ilikq5xtxckz3owbe5xallbx24fkvislzu7k7ydqzia/bagbaierau2ocmnrl5ilikq5xtxckz3owbe5xallbx24fkvislzu7k7ydqzia.car?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=ASIAXLN6VFMMXACSF74D%2F20240311%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20240311T185406Z&X-Amz-Expires=86400&X-Amz-Security-Token=IQoJb3JpZ2luX2VjECsaCXVzLXdlc3QtMiJIMEYCIQCKTNRl%2FrZ47BKRxhGtQfkvjcvPMxBJMWUENt7EL4VYTgIhAO14%2Fs%2Bc3sh2TevvAxplSRT75wFFD9D88Fi3UXDKKCx3KtMDCDQQABoMNTA1NTk1Mzc0MzYxIgz9y4RR%2FZSZB4cqKpcqsAOKYIdtPJflZPodH36ke9NbJNTtfbD7E2LujMS7XEyU2sZnwQyTEzsVL%2FJvB3s8Z3eeKhN4xKnkbUAf%2BryO1ig1plAwzjnIpQzauv89AalyJTS7UM%2FymvWnLo2Qx%2Br92owtQCzOZmrREMVqAU0rCZrMv6NWcOH0bkaM%2FmuA6MijF2dfH3mGeW5E0lFqXaJL5OJqclREtJwRomotSljBVoYxoz2PHRvgIKFuezC1reS3XJ3EnWRlbohQzhfGov%2BtOtcUhJIVk8wqXQvyEFOjErapALiIq4oeUBoIpeKXR5iHJDgndnxGvPOG9ihjUMzKwcbEwqQZTSkTK2whRoHcSZgedzWpsQgae3vQjr797H%2Flh7XLSGzHAuy0Bb9zB61ZZRUyKbhODFCw%2BOpk9ihPKedaewtRHe4SYvAYwBcM0%2FZC%2BwcAOOK987eUxnULeHei%2Fs%2BigKBnzGWwojcuqSzdvaRX3yaK9yKG8mT6aMsAYKS%2BjG%2FX%2FlAu3UL4pe7KK2fxHEH1KrQgdaTCsdzaB%2FApfsixRH4XeGOTNk9ZYM2rI70aqMd%2F5fveFO2%2F33EKZSrQg9kw65i9rwY6nQFkK7gFDZd2oCG7e3u3lqDe5%2BQAqBshYAKbOzpMM6VQq0AqLFeO3TWv%2FhlSP8z9Xxadf3IFRciPlQbMV85VXxEDCzyTYAkLQJsqVuyI%2FQF9hW1rk1UcvOTVpcDRXuwBm%2B66Cq48OnTGzdMu32f%2FsKe7aacdi4msVqU2Z4JwvyjewD4biHJ7lNUkAUf16Zz7SKkLvSvxO%2FwqV7HcxrEs&X-Amz-Signature=dc200cb342555a9a40fe50c45fc7433a7f9e6eaadb233a5f2ddd5f5bacf3ef2b&X-Amz-SignedHeaders=content-length%3Bhost%3Bx-amz-checksum-sha256&x-id=PutObject +https://carpark-prod-0.s3.us-west-2.amazonaws.com/bagbaiera6cg7n32box5trcbif2v5btozvbze2rg5mx2cgwff7643zrpreuua/bagbaiera6cg7n32box5trcbif2v5btozvbze2rg5mx2cgwff7643zrpreuua.car?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=ASIAXLN6VFMM665ZYAHR%2F20240314%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20240314T175533Z&X-Amz-Expires=86400&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEHEaCXVzLXdlc3QtMiJHMEUCIQCUXvGoqROkM7v%2FLVCivP8q5dqoiRe4dbXtqTCV3MP9TAIgKGTZvwjmQ365IcVthNaMXHytEQmm74ABJmUKxFSon14q0wMIehABGgw1MDU1OTUzNzQzNjEiDD3M45t0320FZ6VOriqwA422dYlN9z%2F89MSJjGZQitkOPlF8wdp4t4kU3I4wvjyThjkfSGeyBV20XS%2FfsF8Dcm1L3iuZwUwhMZImvX406MZDzEYLKlyejvoDPQLpGOsopDfLOygdFUUQUvsbZPXvgJwyZp5r5YSUUUEOhmafdGHztdQVzqkiKaQ%2BUrXA1KAHb7qEXlM3NvPAyDNO0GJD8cVDUlygg%2Bw6mZodsjeIt%2B250MRvDU0DL%2BRPIG1OhCM81GMgnkmGpyTh4pSuZPPUqINRm7qKCssk0Fg84isHOCMa7OEonwoAccIHBS%2BQB8XtuvQ%2FQWDZeQUYxFFhvXKQu5tze9ZOHxFC%2FOAvmpoAQklXXzWeySKplID5AihmbCiNqisqqjQ9MuTuJmNfHHEIqFfKx00Rdxkl42IWNp8gtzyjTSjytWbEd%2Bm6yStClUvFo3i4pwn3zvCgZ3aV5NjWuOufQpAnU7Tb7lK02Q0JikIOz5hHHtOFLh5%2F8ldnKGaBp0qs9Nj1E2dEFU%2B%2BuyZ3bx6l1MrVknVhPsWUqCoEmT9me%2BJXKGFUV0hPjDKJNrPEtMLDbFoaW3yIh3hwVR9MWzCX3syvBjqeAZDeWPQKO8Xv%2B%2Fti9cMOU6R7EtRbJ1hHmz95u9SJLKD6E%2F19RQnPFj5f4TcDyoqt4UT6p8fRcK9xkAVg2e1qamlLmtEMEhgavD6YgqSLbEPpRe9CitthIZ9jyPvZYtxccY%2BhUoQSeSAQ11JHTRtYzsjzOs%2BcgXTSDRAaj6ZnCcLpEYgRt7hRO%2BqIhDdUNxbUenM1edJhEW%2FyKBn%2BEfvm&X-Amz-Signature=f47614b5ec4751ec5b46617d172f86daf91bda5a8c15b771e7fb488d15394a4e&X-Amz-SignedHeaders=content-length%3Bhost%3Bx-amz-checksum-sha256&x-id=PutObject curl -X PUT \ --H "content-length: 26" \ --H "x-amz-checksum-sha256: sDe+j519nHU7suFDSf2rKkFYaUBv87MHNWaVvNJoLXs=" \ +-H "content-length: 573086" \ +-H "x-amz-checksum-sha256: 65jEDy7v85ByUoX06Jka6YTSyoKUcabZk9KHvuggolw=" \ -H "content-type: application/vnd.ipld.car" \ --data-binary @big-2.car \ -https://carpark-prod-0.s3.us-west-2.amazonaws.com/bagbaierawa335d45pwohko5s4fbut7nlfjavq2kan7z3gbzvm2k3zutifv5q/bagbaierawa335d45pwohko5s4fbut7nlfjavq2kan7z3gbzvm2k3zutifv5q.car?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=ASIAXLN6VFMMXACSF74D%2F20240311%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20240311T185406Z&X-Amz-Expires=86400&X-Amz-Security-Token=IQoJb3JpZ2luX2VjECsaCXVzLXdlc3QtMiJIMEYCIQCKTNRl%2FrZ47BKRxhGtQfkvjcvPMxBJMWUENt7EL4VYTgIhAO14%2Fs%2Bc3sh2TevvAxplSRT75wFFD9D88Fi3UXDKKCx3KtMDCDQQABoMNTA1NTk1Mzc0MzYxIgz9y4RR%2FZSZB4cqKpcqsAOKYIdtPJflZPodH36ke9NbJNTtfbD7E2LujMS7XEyU2sZnwQyTEzsVL%2FJvB3s8Z3eeKhN4xKnkbUAf%2BryO1ig1plAwzjnIpQzauv89AalyJTS7UM%2FymvWnLo2Qx%2Br92owtQCzOZmrREMVqAU0rCZrMv6NWcOH0bkaM%2FmuA6MijF2dfH3mGeW5E0lFqXaJL5OJqclREtJwRomotSljBVoYxoz2PHRvgIKFuezC1reS3XJ3EnWRlbohQzhfGov%2BtOtcUhJIVk8wqXQvyEFOjErapALiIq4oeUBoIpeKXR5iHJDgndnxGvPOG9ihjUMzKwcbEwqQZTSkTK2whRoHcSZgedzWpsQgae3vQjr797H%2Flh7XLSGzHAuy0Bb9zB61ZZRUyKbhODFCw%2BOpk9ihPKedaewtRHe4SYvAYwBcM0%2FZC%2BwcAOOK987eUxnULeHei%2Fs%2BigKBnzGWwojcuqSzdvaRX3yaK9yKG8mT6aMsAYKS%2BjG%2FX%2FlAu3UL4pe7KK2fxHEH1KrQgdaTCsdzaB%2FApfsixRH4XeGOTNk9ZYM2rI70aqMd%2F5fveFO2%2F33EKZSrQg9kw65i9rwY6nQFkK7gFDZd2oCG7e3u3lqDe5%2BQAqBshYAKbOzpMM6VQq0AqLFeO3TWv%2FhlSP8z9Xxadf3IFRciPlQbMV85VXxEDCzyTYAkLQJsqVuyI%2FQF9hW1rk1UcvOTVpcDRXuwBm%2B66Cq48OnTGzdMu32f%2FsKe7aacdi4msVqU2Z4JwvyjewD4biHJ7lNUkAUf16Zz7SKkLvSvxO%2FwqV7HcxrEs&X-Amz-Signature=232f28f1b4392e185aadc27c50e96ba6101529f7a9cbca8eced582602a9a0e81&X-Amz-SignedHeaders=content-length%3Bhost%3Bx-amz-checksum-sha256&x-id=PutObject +https://carpark-prod-0.s3.us-west-2.amazonaws.com/bagbaiera5ommidzo57zza4ssqx2orgi25gcnfsucsry2nwmt2kd352baujoa/bagbaiera5ommidzo57zza4ssqx2orgi25gcnfsucsry2nwmt2kd352baujoa.car?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=ASIAXLN6VFMM665ZYAHR%2F20240314%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20240314T175533Z&X-Amz-Expires=86400&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEHEaCXVzLXdlc3QtMiJHMEUCIQCUXvGoqROkM7v%2FLVCivP8q5dqoiRe4dbXtqTCV3MP9TAIgKGTZvwjmQ365IcVthNaMXHytEQmm74ABJmUKxFSon14q0wMIehABGgw1MDU1OTUzNzQzNjEiDD3M45t0320FZ6VOriqwA422dYlN9z%2F89MSJjGZQitkOPlF8wdp4t4kU3I4wvjyThjkfSGeyBV20XS%2FfsF8Dcm1L3iuZwUwhMZImvX406MZDzEYLKlyejvoDPQLpGOsopDfLOygdFUUQUvsbZPXvgJwyZp5r5YSUUUEOhmafdGHztdQVzqkiKaQ%2BUrXA1KAHb7qEXlM3NvPAyDNO0GJD8cVDUlygg%2Bw6mZodsjeIt%2B250MRvDU0DL%2BRPIG1OhCM81GMgnkmGpyTh4pSuZPPUqINRm7qKCssk0Fg84isHOCMa7OEonwoAccIHBS%2BQB8XtuvQ%2FQWDZeQUYxFFhvXKQu5tze9ZOHxFC%2FOAvmpoAQklXXzWeySKplID5AihmbCiNqisqqjQ9MuTuJmNfHHEIqFfKx00Rdxkl42IWNp8gtzyjTSjytWbEd%2Bm6yStClUvFo3i4pwn3zvCgZ3aV5NjWuOufQpAnU7Tb7lK02Q0JikIOz5hHHtOFLh5%2F8ldnKGaBp0qs9Nj1E2dEFU%2B%2BuyZ3bx6l1MrVknVhPsWUqCoEmT9me%2BJXKGFUV0hPjDKJNrPEtMLDbFoaW3yIh3hwVR9MWzCX3syvBjqeAZDeWPQKO8Xv%2B%2Fti9cMOU6R7EtRbJ1hHmz95u9SJLKD6E%2F19RQnPFj5f4TcDyoqt4UT6p8fRcK9xkAVg2e1qamlLmtEMEhgavD6YgqSLbEPpRe9CitthIZ9jyPvZYtxccY%2BhUoQSeSAQ11JHTRtYzsjzOs%2BcgXTSDRAaj6ZnCcLpEYgRt7hRO%2BqIhDdUNxbUenM1edJhEW%2FyKBn%2BEfvm&X-Amz-Signature=5792f7d2847ab963c54e34c44c7f821ccded0baf951d2be262be1a2e9c59101f&X-Amz-SignedHeaders=content-length%3Bhost%3Bx-amz-checksum-sha256&x-id=PutObject ``` Once this is all done you'll be able to see your upload in `w3 ls` and through the gateway, eg at https://bafybeicq2urqdqum3ykblswqzzxf6wpy3mcbr6jg52ai2zlhds4l7ho5wa.ipfs.w3s.link