Skip to content

Commit

Permalink
📝(project) update ralph v4 upgrade guide and changelog
Browse files Browse the repository at this point in the history 
We update the upgrade guide to help users to migrate to
ralph v4.
We also update the changelog with recent changes made to the
data backends.
  • Loading branch information
@SergioSim
SergioSim committed Dec 14, 2023
1 parent c9ea22c commit 6e6df94
Show file tree
Hide file tree
Showing 2 changed files with 196 additions and 36 deletions.
86 changes: 50 additions & 36 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,66 +10,80 @@ and this project adheres to

### Added

- Implement Pydantic model for LRS Statements resource query parameters
- Implement xAPI LMS Profile statements validation
- `EdX` to `xAPI` converters for enrollment events
- Backends: Add `Writable` and `Listable` interfaces to distinguish supported
functionalities among `data` backends
- Backends: Add `max_statements` option to data backends `read` method
- Backends: Add `prefetch` option to async data backends `read` method
- Backends: Add `concurrency` option to async data backends `write` method
- Backends: Add `get_backends` function to automatically discover backends
for CLI and LRS usage
- Add aliases for `ralph-malph` extra dependencies: `backends` and `full`
- Backends: Add client options for WSDataBackend
- Backends: Add `READ_CHUNK_SIZE` and `WRITE_CHUNK_SIZE` data backend settings
- Models: Implement Pydantic model for LRS Statements resource query parameters
- Models: Implement xAPI LMS Profile statements validation
- Models: Add `EdX` to `xAPI` converters for enrollment events
- Project: Add aliases for `ralph-malph` extra dependencies: `backends` and
`full`

### Changed

- Upgrade base python version to 3.12 for the development stack and Docker
image
- Upgrade `cachetools` to `5.3.2`
- Refactor `database` and `storage` backends under the unified `data` backend
interface [BC]
- Refactor LRS `query_statements` and `query_statements_by_ids` backends
methods under the unified `lrs` backend interface [BC]
- Refactor LRS Statements resource query parameters defined for `ralph` API
- User credentials must now include an "agent" field which can be created
using the cli
- `GET /statements` now has "mine" option which matches statements that
have an authority field matching that of the user
- CLI: change `push` to `write` and `fetch` to `read` [BC]
- Upgrade `bcrypt` to `4.1.1`
- Upgrade `fastapi` to `0.105.0`
- Upgrade `sentry_sdk` to `1.38.0`
- Upgrade `uvicorn` to `0.24.0.post1`
- Arnold: Add variable to override PVC name in arnold deployment
- API: `GET /statements` now has "mine" option which matches statements that
have an authority field matching that of the user
- API: Invalid parameters now return 400 status code
- API: Forwarding PUT now uses PUT (instead of POST)
- Models: The xAPI `context.contextActivities.category` field is now mandatory
in the video and virtual classroom profiles. [BC]
- Backends: `LRSHTTP` methods must not be used in `asyncio` events loop (BC)
- Add variable to override PVC name in arnold deployment
- Backends: add `max_statements` option to `AsyncLRSHTTP`
- API: Incoming statements are enriched with `id`, `timestamp`, `stored`
and `authority`
- Backends: update `statementId` and `voidedStatementId` to snake_case,
with camelCase alias, in `LRSStatementsQuery`
- API: Add `RALPH_LRS_RESTRICT_BY_AUTHORITY` option making `?mine=True`
implicit
- CLI: list cli usage strings in alphabetical order
- API: Add `RALPH_LRS_RESTRICT_BY_SCOPE` option enabling endpoint access
control by user scopes
- Backends: Replace reference to a JSON column in ClickHouse with
function calls on the String column [BC]
- API: enhance 'limit' query parameter's validation
- API: Enhance 'limit' query parameter's validation
- API: Variable `RUNSERVER_AUTH_BACKEND` becomes `RUNSERVER_AUTH_BACKENDS`, and
multiple authentication methods are supported simultaneously
- Backends: Refactor LRS Statements resource query parameters defined for
`ralph` API
- Backends: Refactor `database`, `storage`, `http` and `stream` backends under
the unified `data` backend interface [BC]
- Backends: Refactor LRS `query_statements` and `query_statements_by_ids`
backends methods under the unified `lrs` backend interface [BC]
- Backends: Update `statementId` and `voidedStatementId` to snake_case,
with camelCase alias, in `LRSStatementsQuery`
- Backends: Replace reference to a JSON column in ClickHouse with
function calls on the String column [BC]
- CLI: User credentials must now include an "agent" field which can be created
using the cli
- CLI: Change `push` to `write` and `fetch` to `read` [BC]
- CLI: Change `-c --chunk-size` option to `-s --chunk-size` [BC]
- CLI: Change websocket backend name `-b ws` to `-b async_ws` along with it's
uri option `--ws-uri` to `--async-ws-uri` [BC]
- CLI: List cli usage strings in alphabetical order
- CLI: Change backend configuration environment variable prefixes from
`RALPH_BACKENDS__{{DATABASE|HTTP|STORAGE|STREAM}}__{{BACKEND}}__{{OPTION}}`
to `RALPH_BACKENDS__DATA__{{BACKEND}}__{{OPTION}}`
- Models: The xAPI `context.contextActivities.category` field is now mandatory
in the video and virtual classroom profiles. [BC]
- Upgrade base python version to 3.12 for the development stack and Docker
image
- Upgrade `cachetools` to `5.3.2`
- Upgrade `bcrypt` to `>=4.0.0`
- Upgrade `fastapi` to `0.105.0`
- Upgrade `sentry_sdk` to `1.38.0`
- Upgrade `uvicorn` to `0.24.0.post1`

### Fixed

- API: Fix a typo ('attachements' -> 'attachments') to ensure compliance with
the LRS specification and prevent potential silent bugs.
the LRS specification and prevent potential silent bugs

### Removed

- `school`, `course`, `module` context extensions in Edx to xAPI base converter
- `name` field in `VideoActivity` xAPI model mistakenly used in `video` profile
- drop support for python 3.7
- Project: Drop support for Python 3.7
- Models: Remove `school`, `course`, `module` context extensions in Edx to xAPI
base converter
- Models: Remove `name` field in `VideoActivity` xAPI model mistakenly used in
`video` profile
- CLI: Remove `DEFAULT_BACKEND_CHUNK_SIZE` environment variable configuration

## [3.9.0] - 2023-07-21

Expand Down
146 changes: 146 additions & 0 deletions UPGRADE.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,55 @@ For example here is a valid `auth.json` file:
]
```

#### Upgrade Ralph CLI usage

If you are using Ralph's CLI, the following changes may affect you:

- The `ralph fetch` command changed to `ralph read`
- The `-b ws` backend option changed to `-b async_ws`
- The corresponding `--ws-uri` option changed to `--async-ws-uri`
- The `-c --chunk-size` option changed to `-s --chunk-size`
- The `DEFAULT_BACKEND_CHUNK_SIZE` environment variable configuration is removed in
favor of allowing each backend to define their own defaults:

| Backend | Environment variable for default (read) chunk size |
|-------------------|-------------------------------------------------------|
| async_es/es | RALPH_BACKENDS__DATA__ES__READ_CHUNK_SIZE=500 |
| async_lrs/lrs | RALPH_BACKENDS__DATA__LRS__READ_CHUNK_SIZE=500 |
| async_mongo/mongo | RALPH_BACKENDS__DATA__MONGO__READ_CHUNK_SIZE=500 |
| clickhouse | RALPH_BACKENDS__DATA__CLICKHOUSE__READ_CHUNK_SIZE=500 |
| fs | RALPH_BACKENDS__DATA__FS__READ_CHUNK_SIZE=4096 |
| ldp | RALPH_BACKENDS__DATA__LDP__READ_CHUNK_SIZE=4096 |
| s3 | RALPH_BACKENDS__DATA__S3__READ_CHUNK_SIZE=4096 |
| swift | RALPH_BACKENDS__DATA__SWIFT__READ_CHUNK_SIZE=4096 |

- The `ralph push` command changed to `ralph write`
- The `-c --chunk-size` option changed to `-s --chunk-size`
- The `DEFAULT_BACKEND_CHUNK_SIZE` environment variable configuration is removed in
favor of allowing each backend to define their own defaults:

| Backend | Environment variable for default (write) chunk size |
|-------------------|--------------------------------------------------------|
| async_es/es | RALPH_BACKENDS__DATA__ES__WRITE_CHUNK_SIZE=500 |
| async_lrs/lrs | RALPH_BACKENDS__DATA__LRS__WRITE_CHUNK_SIZE=500 |
| async_mongo/mongo | RALPH_BACKENDS__DATA__MONGO__WRITE_CHUNK_SIZE=500 |
| clickhouse | RALPH_BACKENDS__DATA__CLICKHOUSE__WRITE_CHUNK_SIZE=500 |
| fs | RALPH_BACKENDS__DATA__FS__WRITE_CHUNK_SIZE=4096 |
| ldp | RALPH_BACKENDS__DATA__LDP__WRITE_CHUNK_SIZE=4096 |
| s3 | RALPH_BACKENDS__DATA__S3__WRITE_CHUNK_SIZE=4096 |
| swift | RALPH_BACKENDS__DATA__SWIFT__WRITE_CHUNK_SIZE=4096 |

- Environment variables used to configure backend options for CLI usage
(read/write/list commands) changed their prefix:
`RALPH_BACKENDS__{{DATABASE or HTTP or STORAGE or STREAM}}__{{BACKEND}}__{{OPTION}}`
changed to
`RALPH_BACKENDS__DATA__{{BACKEND}}__{{OPTION}}`
- Environment variables used to configure backend options for LRS usage
(runserver command) changed their prefix:
`RALPH_BACKENDS__{{DATABASE}}__{{BACKEND}}__{{OPTION}}`
changed to
`RALPH_BACKENDS__LRS__{{BACKEND}}__{{OPTION}}`

#### Upgrade history syntax

CLI syntax has been changed from `fetch` & `push` to `read` & `write` affecting the command history. You must replace the command history after updating:
Expand All @@ -82,6 +131,103 @@ sed -i 's/"fetch"/"read"/g' { my_history_file_path }
sed -i 's/"push"/"write"/g' { my_history_file_path }
```
#### Upgrade Ralph library usage (backends)
If you use Ralph's backends in your application, the following changes might affect you:
Backends from `ralph.backends.database`, `ralph.backends.http`, `ralph.backends.stream`,
and `ralph.backends.storage` packages have moved to a single `ralph.backends.data`
package.
| Ralph v3 (database/http/storage/stream) backends | Ralph v4 data backends |
|---------------------------------------------------------|--------------------------------------------------------|
| `ralph.backends.database.clickhouse.ClickHouseDatabase` | `ralph.backends.data.clickhouse.ClickHouseDataBackend` |
| `ralph.backends.database.es.ESDatabase` | `ralph.backends.data.es.ESDataBackend` |
| `ralph.backends.database.mongo.MongoDatabase` | `ralph.backends.data.mongo.MongoDataBackend` |
| `ralph.backends.http.async_lrs.AsyncLRSHTTP` | `ralph.backends.data.async_lrs.AsyncLRSDataBackend` |
| `ralph.backends.http.lrs.LRSHTTP` | `ralph.backends.data.lrs.LRSDataBackend` |
| `ralph.backends.storage.fs.FSStorage` | `ralph.backends.data.fs.FSDataBackend` |
| `ralph.backends.storage.ldp.LDPStorage` | `ralph.backends.data.ldp.LDPDataBackend` |
| `ralph.backends.storage.s3.S3Storage` | `ralph.backends.data.s3.S3DataBackend` |
| `ralph.backends.storage.swift.SwiftStorage` | `ralph.backends.data.swift.SwiftDataBackend` |
| `ralph.backends.stream.ws.WSStream` | `ralph.backends.data.async_ws.AsyncWSDataBackend` |
LRS-specific `query_statements` and `query_statements_by_ids` database backend methods
have moved to a dedicated `ralph.backends.lrs.BaseLRSBackend` interface that extends the
data backend interface with these two methods.
The `query_statements_by_ids` method return type changed to `Iterator[dict]`.
| Ralph v3 database backends for lrs usage | Ralph v4 LRS data backends |
|---------------------------------------------------------|--------------------------------------------------------|
| `ralph.backends.database.clickhouse.ClickHouseDatabase` | `ralph.backends.lrs.clickhouse.ClickHouseLRSBackend` |
| `ralph.backends.database.es.ESDatabase` | `ralph.backends.lrs.es.ESLRSBackend` |
| `ralph.backends.database.mongo.MongoDatabase` | `ralph.backends.lrs.mongo.MongoLRSBackend` |
**Backend interface differences**
- Data backends are read-only by default
- Data backends that support write operations inherit from the
`ralph.backends.data.base.Writable` interface
- Data backends that support list operations inherit from the
`ralph.backends.data.base.Listable` interface
- Data backends that support LRS operations
(`query_statements`/`query_statements_by_ids`) inherit from the
`ralph.backends.lrs.BaseLRSBackend` interface
- `__init__(self, **kwargs)` changed to `__init__(self, settings: DataBackendSettings)`
where each DataBackend defines it's own Settings object
For example the `FSDataBackend` uses `FSDataBackendSettings`
- `stream` and `get` methods changed to `read`
- `put` methods changed to `write`
**Backend usage migration example**
Ralph v3 using `ESDatabase`:
```python
from ralph.conf import ESClientOptions
from ralph.backends.database.es import ESDatabase, ESQuery

# Instantiate the backend.
backend = ESDatabase(
hosts="localhost",
index="statements"
client_options=ESClientOptions(verify_certs=False)
)
# Read records from backend.
query = ESQuery(query={"query": {"term": {"modulo": 0}}})
es_statements = list(backend.get(query))

# Write records to backend.
backend.put([{"id": 1}])
```
Ralph v4 using `ESDataBackend`:
```python
from ralph.backends.data.es import (
ESClientOptions,
ESDataBackend,
ESDataBackendSettings,
ESQuery,
)

# Instantiate the backend.
settings = ESDataBackendSettings(
HOSTS="localhost",
INDEX="statements",
CLIENT_OPTIONS=ESClientOptions(verify_certs=False)
)
backend = ESDataBackend(settings)

# Read records from backend.
query = ESQuery(query={"term": {"modulo": 0}})
es_statements = list(backend.read(query))

# Write records to backend.
backend.write([{"id": 1}])
```
#### Upgrade ClickHouse schema
If you are using the ClickHouse backend, schema changes have been made
Expand Down

0 comments on commit 6e6df94

Please sign in to comment.