Skip to content

Oonirun v2.1 #301

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Oonirun v2.1 #301

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
7ea127b
Start adding details about dynamic inputs in OONI Run v2
hellais Feb 18, 2025
86c25ca
Add headers used to pass options to descriptor get request
hellais Feb 18, 2025
97552f0
Simplify headers for OONI Run v2.1
hellais Apr 14, 2025
ba441db
Update bk-005-ooni-run-v2.md
hellais May 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
191 changes: 152 additions & 39 deletions backends/bk-005-ooni-run-v2.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ Below are definitions for important components of the system:
with the OONI Probe app installed allows the user to instrument their probe to
run the nettests configured by the link creator. If the OONI Probe app is not
installed, it will display a web page asking them to download the app.

* **OONI Run descriptor** contains the metadata for a specific OONI Run link and
the nettest definitions: what nettests should be run and how they should be
configured (ex. what URLs should be tested)
Expand Down Expand Up @@ -125,12 +125,25 @@ An OONI Run link descriptor is a JSON file with the following semantics:
// `array` provides a JSON array of tests to be run.
"nettests":[{

// (optional) `array` provides a JSON array of tests to be run.
// (optional) `array` provides a JSON array of inputs for the specified test.
"inputs": [
"https://example.com/",
"https://ooni.org/"
],

// (optional) `array` provides a richer JSON array containing targets for the specified experiment.
// One of either `inputs` or `targets` can be specified, but not both.
"targets": [{
"input": "https://example.com/",
"extra": {
"category_code": "HUMR",
}
}
],

// (optional) string used to specify during creation what is the label for targets to use.
"targets_name": "websites_list_prioritized",
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will come from a fixed list of options right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we will have a predefined list of these. The client doesn't necessarily need to be aware of all the targets_name, but we will start off with just the websites target list.


// (optional) `map` options arguments provided to the specified test_name
//
// Note: this field is currently experimental. A future version of the specification
Expand All @@ -139,18 +152,6 @@ An OONI Run link descriptor is a JSON file with the following semantics:
"HTTP3Enabled": true
},

// (optional) `map` settings which are sent to probe_services for retrieving the inputs vector
// It's possible to reference user configured settings by accessing the
// $settings special variable.
// In particular the content of the backend_options will be sent to the /api/v1/check-in call nested
// under the relative test_name.
//
// Note: this field is currently experimental. A future version of the specification
// may modify the field name or its semantics if we discover it needs changes.
"backend_options": {
"category_codes": "$settings.category_codes"
},

// (optional) `bool` indicates if this test should be run as part of autoruns. Defaults to true.
//
// Note: this field is currently experimental. A future version of the specification
Expand All @@ -166,21 +167,33 @@ An OONI Run link descriptor is a JSON file with the following semantics:
"test_name": "web_connectivity"
}, {
"test_name": "openvpn",
"backend_options": {
"provider": "riseupvpn"
},
"targets": [{
"input": "https://foo.com/",
"extra": {
"provider": "riseupvpn",
}
}],
}]
}
```

In particular values starting with the `$settings` prefix should be mapped
based on user-configured preferences in the app. (Note also that this is
currently an experimental feature that we may change in the future.)
In reality there are two different views onto an OONI Run link descriptor. One
is the view from the perspective of the creator and owner of the link, the
second is from the perspective of measurement applications that need to consume
these links.

The reason for this is that certain target lists need to be generated
dynamically and at runtime, while the parameters for generating these dynamic
links are specified by the creator of the link.

Currently the following `$settings` are defined:
A prime example of this would be the OONI Run link for the stock OONI Websites
card. The OONI Run descriptor, as specified from the link creator, is saying
"run the test-lists with weights applied based on coverage", while the mobile
application will then receive a prioritized and sorted list which will change
every time a new run is performed.

* `$settings.category_codes` should be replaced with the list of category codes
the user has enabled (ex. `["HUMR", "NEWS"]`.
To achieve this we need 2 different views on the link, one for the creation
phase (`target_name`) and another for the retrieval phase (`targets`).

Based on the above specification it would be possible to re-implement the cards for the OONI Probe
dashboard as follows.
Expand All @@ -197,20 +210,17 @@ dashboard as follows.
"nettests":
[
{
"options": {
"MaxRuntime": "$settings.max_runtime",
},
"backend_settings": {
"category_codes": "$settings.category_codes"
},
"targets": [{
"input": "https://example.com/",
"extra": {
"category_code": "HUMR",
}
}],
"is_manual_run_enabled": true,
"is_background_run_enabled": false,
"test_name": "web_connectivity"
},
{
"backend_settings": {
"category_codes": "$settings.category_codes"
},
"is_manual_run_enabled": false,
"is_background_run_enabled": true,
"test_name": "web_connectivity"
Expand Down Expand Up @@ -358,7 +368,7 @@ authentication should be handled.
When you `CREATE` a new OONI RUN link, the client sends a HTTP `POST`
request conforming to the following:

`POST /api/v2/oonirun`
`POST /api/v2/oonirun/links`

```JavaScript
{
Expand Down Expand Up @@ -401,6 +411,11 @@ request conforming to the following:
}
```

Moreover it's possible to specify `targets` as rich JSON structure or
`target_name` indicating the name of the target list which will result in the
target list being generated dynamically.

`target`, `target_name` and `inputs` are mutually exclusive.

### Response status code

Expand Down Expand Up @@ -465,7 +480,7 @@ To update an OONI Run Link, the client issues a request compliant the same as th
Below we list the extra fields that are settable from the edit request that are
not settable during CREATE.

`PUT /api/v2/oonirun/{ooni_run_link_id}`
`PUT /api/v2/oonirun/links/{ooni_run_link_id}`

```JavaScript
{
Expand Down Expand Up @@ -510,16 +525,16 @@ following JSON body:

## 4.3 GET the OONI Run descriptor

This operation is performed by OONI Probe clients to retrieve the descriptor of
a certain OONI Run link given the ID.
This operation is performed by OONI Probe clients to retrieve the latest
revision for a descriptor of a certain OONI Run link given the ID.

As such, this request does not require any authentication.

### Request

To retrieve an OONI Run link descriptor, the client issues a request compliant with:

`GET /api/v2/oonirun/{oonirun_link_id}`
`GET /api/v2/oonirun/links/{oonirun_link_id}`

### Response status code

Expand All @@ -541,7 +556,105 @@ following JSON body:
}
```

## 4.4 LIST the OONI Run descriptors
Note: for dynamically generated run links, this view will only return the
`target_name` and not the `targets` list.

## 4.4 GET the OONI Run full descriptor by revision

This operation is performed by OONI Probe clients to retrieve the descriptor of
a certain OONI Run link given the ID and revision

As such, this request does not require any authentication.

### Request

To retrieve an OONI Run link descriptor, the client issues a request compliant with:

`GET /api/v2/oonirun/links/{oonirun_link_id}/full-descriptor/{revision}`

### Response status code

Same as 4.3 GET the OONI Run descriptor

### Response body

Same as 4.3 GET the OONI Run descriptor

Note: for dynamically generated run links, this view will only return the
`target_name` and not the `targets` list.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This way, the users never see the full list of URLs to be tested before installing a Run Link right? Maybe on the mobile clients we need to add a button to see the full list on the web...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this is by design though, given that for dynamic test lists (which are currently only derived from the test lists), we cannot know the full list ahead of time as it will change every time you run it.

From the perspective of the UI, we can either show a label that explains these are dynamic and refer to the the test-lists repo to view them. Alternatively we can fetch a batch of URLs from the API endpoint and display that, but make it clear it's just a sample and might change.


## 4.4 GET the OONI Run engine descriptor revision

This operation is performed by OONI Probe clients to retrieve the engine descriptor of
a certain OONI Run link given the ID and revision

As such, this request does not require any authentication.

This method is used to return just the nettests, revision and date_created
sections of a descriptor to be used by the measurement engine.

When the specified OONI Run link contains dynamic targets, it will return the
generated `targets` list and hence every request might return a different target
list.

### Request

To retrieve an OONI Run link descriptor, the client issues a request compliant with:

`GET /api/v2/oonirun/links/{oonirun_link_id}/engine-descriptor/{revision}?run_type={timed|manual}&is_charging={true|false}`

Upon receiving this request, the OONI Run backend:

1. SHOULD check whether the `${oonirun_link_id}` exists and return 404 if it does
not.

2. if everything is okay, returns `200` to the client (see below).

A client should also include the following headers to allow the server to
properly generate dynamic target lists:
* `X-OONI-NetworkInfo`: `<probe_asn>,<probe_cc> (<network_type>)`, eg `AS1234,IT (wifi)`
* `X-OONI-WebsiteCategoryCodes`: comma separated list of category codes that user has chosen to test (eg. NEWS,HUMR)
* `X-OONI-Credentials`: base64 encoded OONI anonymous credentials

The the `platform`, `software_name`, `software_name`, `engine_name` and
`engine_version` are encoded inside of the `User-Agent` string using the following
format:
```
<software_name>/<software_version> (<platform>) <engine_name>/<engine_version> (<engine_version_full>)
```

### Response body

In case of success (i.e. `200` response), the OONI Run Service MUST return the
following JSON body:

```JavaScript
{
"revision": "1",
"date_created": ""
"nettests": [
{
// See CREATE response format for other fields
"targets": [{
"input": "https://example.com/",
"extra": {
"category_code": "HUMR",
}
}],
"test_name": "web_connectivity"
}
]
}
```

Additionally, the `Vary` header should specify the list of headers that affect
the response body caching, which are all headers starting with the `X-OONI-`
prefix.

The server might also return an updated version of the submitted anonymous
credentials using the `X-OONI-Credentials` header.

## 4.7 LIST the OONI Run descriptors

This operation is performed by users of the OONI Run platform to list all the existing OONI Run links.

Expand All @@ -551,7 +664,7 @@ Authentication for this endpoint is optional.

To retrieve an OONI Run link descriptor, the client issues a request compliant with:

`GET /api/v2/oonirun_links?only_latest=true&only_mine=true&include_archived=true`
`GET /api/v2/oonirun/links?only_latest=true&only_mine=true&include_archived=true`

- `only_latest`, boolean flag to filter only by the latest revision of an OONI
Run link. If unset or set to false, it will instead include all revisions as
Expand Down