Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add task runner documentation #2671

Merged
merged 12 commits into from
Dec 4, 2024
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
60 changes: 60 additions & 0 deletions docs/hosting/configuration/environment-variables/task-runners.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
---
#https://www.notion.so/n8n/Frontmatter-432c2b8dff1f43d4b1c8d20075510fe4
title: Task runner environment variables
description: Environment variables to confgure task runners your self-hosted n8n instance.
contentType: reference
tags:
- environment variables
hide:
- toc
- tags
---

# Task runner environment variables

--8<-- "_snippets/self-hosting/file-based-configuration.md"

[Task runners](/hosting/configuration/task-runners/) execute code defined by the [Code node](/integrations/builtin/core-nodes/n8n-nodes-base.code/).

## n8n instance environment variables

| Variable | Type | Default | Description |
| :------- | :---- | :------- | :---------- |
| `N8N_RUNNERS_ENABLED` | Boolean | `false` | Are task runners enabled. |
| `N8N_RUNNERS_MODE` | Enum string: `internal`, `external` | `internal` | How to launch and run the task runner. `internal` means n8n will launch a task runner as child process. `external` means an external orchestrator will launch the task runner. |
| `N8N_RUNNERS_AUTH_TOKEN` | String | Random string | Shared secret used by a task runner to authenticate to n8n. Required when using `external` mode. |
| `N8N_RUNNERS_SERVER_PORT` | Number | `5679` | Port the task broker listens on for task runner connections. |
| `N8N_RUNNERS_SERVER_LISTEN_ADDRESS` | String | `127.0.0.1` | Address the task broker listens on. |
| `N8N_RUNNERS_MAX_PAYLOAD` | Number | `1 073 741 824` | Maximum payload size in bytes for communication between a task broker and a task runner. |
| `N8N_RUNNERS_MAX_OLD_SPACE_SIZE` | String | | The `--max-old-space-size` option to use for a task runner (in MB). By default, Node.js will set this based on available memory. |
| `N8N_RUNNERS_MAX_CONCURRENCY` | Number | `5` | The number of concurrent tasks a task runner can execute at a time. |
| `N8N_RUNNERS_TASK_TIMEOUT` | Number | `60` | How long (in seconds) a task can take to complete before the task aborts and the runner restarts. Must be greater than 0. |
| `N8N_RUNNERS_HEARTBEAT_INTERVAL` | Number | `30` | How often (in seconds) the runner must send a heartbeat to the broker, else the task aborts and the runner restarts. Must be greater than 0. |


## Task runner launcher environment variables

| Variable | Type | Default | Description |
| :------- | :---- | :------- | :---------- |
| `N8N_RUNNERS_LAUNCHER_LOG_LEVEL` | Enum string: `debug`, `info`, `warn`, `error` | `info` | Which log messages to show. |
| `N8N_RUNNERS_AUTH_TOKEN` | String | - | Shared secret used to authenticate to n8n. |
| `N8N_RUNNERS_AUTO_SHUTDOWN_TIMEOUT` | Number | `15` | The number of seconds to wait before shutting down an idle runner. |
| `N8N_RUNNERS_TASK_BROKER_URI` | String | `http://127.0.0.1:5679` | The URI of the task broker server (n8n instance). |
| `N8N_RUNNERS_LAUNCHER_HEALTH_CHECK_PORT` | Number | `5680` | Port for the launcher's health check server. |
| `N8N_RUNNERS_MAX_PAYLOAD` | Number | `1 073 741 824` | Maximum payload size in bytes for communication between a task broker and a task runner. |
| `N8N_RUNNERS_MAX_CONCURRENCY` | Number | `5` | The number of concurrent tasks a task runner can execute at a time. |
| `NODE_OPTIONS` | String | - | [Options](https://nodejs.org/api/cli.html#node_optionsoptions) for Node.js. |


## Task runner environment variables

| Variable | Type | Default | Description |
| :------- | :---- | :------- | :---------- |
| `N8N_RUNNERS_GRANT_TOKEN` | String | Random string | Token the runner uses to authenticate with the task broker. This is automatically provided by the launcher. |
| `N8N_RUNNERS_AUTO_SHUTDOWN_TIMEOUT` | Number | `15` | The number of seconds to wait before shutting down an idle runner. |
| `N8N_RUNNERS_TASK_BROKER_URI` | String | `http://127.0.0.1:5679` | The URI of the task broker server (n8n instance). |
| `N8N_RUNNERS_LAUNCHER_HEALTH_CHECK_PORT` | Number | `5680` | Port for the launcher's health check server. |
| `N8N_RUNNERS_MAX_PAYLOAD` | Number | `1 073 741 824` | Maximum payload size in bytes for communication between a task broker and a task runner. |
| `N8N_RUNNERS_MAX_CONCURRENCY` | Number | `5` | The number of concurrent tasks a task runner can execute at a time. |
| `NODE_FUNCTION_ALLOW_BUILTIN` | String | - | Permit users to import specific built-in modules in the Code node. Use * to allow all. n8n disables importing modules by default. |
| `NODE_FUNCTION_ALLOW_EXTERNAL` | String | - | Permit users to import specific external modules (from `n8n/node_modules`) in the Code node. n8n disables importing modules by default. |
83 changes: 83 additions & 0 deletions docs/hosting/configuration/task-runners.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
---
#https://www.notion.so/n8n/Frontmatter-432c2b8dff1f43d4b1c8d20075510fe4
title: Task runners
description: How to configure task runners to execute tasks using internal or external runner processes.
contentType: howto
---

# Task runners

Task runners are a generic mechanism to execute tasks in a secure and performant way. They're used to execute user-provided JavaScript code in the [Code node](/integrations/builtin/core-nodes/n8n-nodes-base.code/).

This document describes how task runners work and how you can configure them.

## How it works

The task runner feature consists of three components: a task runner, a task broker, and a task requester.

![Task runner overview](/_images/hosting/configuration/task-runner-concept.png)

Task runners connect to the task broker using a websocket connection. A task requester submits a task request to the broker where an available task runner can pick it up for execution.

The runner executes the task and submits the results to the task requester. The task broker coordinates communication between the runner and the requester.

The n8n instance (main and worker) acts as the broker. The Code node in this case is the task requester.

## Task runner modes

You can use task runners in two different modes: internal and external.

### Internal mode

In internal mode, the n8n instance launches the task runner as a child process. The n8n process monitors and manages the life cycle of the task runner. The task runner process shares the same `uid` and `gid` as n8n.

### External mode

In external mode, an external orchestrator (for example, Kubernetes) launches the task runner instead of n8n. Typically, this means you would configure the task runner to run as a side-car container next to n8n.

![Task runner deployed as a side-car container](/_images/hosting/configuration/task-runner-external-mode.png)

In this mode, the orchestrator monitors and manages the life cycle of the task runner container. The task runner is fully isolated from the n8n instance.

When using the [Queue mode](/hosting/scaling/queue-mode), each n8n container (main and workers) needs to have its own task runner.

## Setting up external mode

Use the following details to configure task runners in external mode

### Configuring n8n instance in external mode

You can configure n8n to use external task runners by setting the following environment variables:

| Environment variables | Description |
|--------------------------------------------------------|------------------------------------------------------------|
| `N8N_RUNNERS_ENABLED=true` | Enables task runners. |
| `N8N_RUNNERS_MODE=external` | Use task runners in external mode. |
| `N8N_RUNNERS_AUTH_TOKEN=<random secure shared secret>` | A shared secret task runners use to connect to the broker. |

For full list of environment variables see [task runner environment variables](/hosting/configuration/environment-variables/task-runners/).

### Configuring task runners in external mode

The task runner comes bundled within the n8n Docker image. The Docker image also includes the task runner launcher.

The launcher can start the runner on-demand, which means lower memory usage when there's no work needed, but a short delay (few hundred ms) in cold-start. The launcher also monitors the launcher and restarts it in case of infinite loops or other issues.

Check warning on line 64 in docs/hosting/configuration/task-runners.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [from-write-good.Weasel] 'few' is a weasel word! Raw Output: {"message": "[from-write-good.Weasel] 'few' is a weasel word!", "location": {"path": "docs/hosting/configuration/task-runners.md", "range": {"start": {"line": 64, "column": 125}}}, "severity": "WARNING"}
tomi marked this conversation as resolved.
Show resolved Hide resolved

Run a task runner container from the n8n Docker image by setting the following properties:

| Configuration | Description |
|-----------------|---------------------------------------------------------|
| `command` | `["/usr/local/bin/task-runner-launcher", "javascript"]` |
| `livenessProbe` | `GET /healthz`, port `5680` |

Set the following environment variables for the container, adjusted to fit your needs:

| Environment variables | Description |
| ------ | ----- |
| `N8N_RUNNERS_AUTH_TOKEN=<random secure shared secret>` | The shared secret the task runner uses to connect to the broker. |
| `N8N_RUNNERS_MAX_CONCURRENCY=5` | The number of concurrent tasks the runner can execute. |
| `N8N_RUNNERS_TASK_BROKER_URI=localhost:5679` | The address of the task broker server within the n8n instance. |
| `N8N_RUNNERS_AUTO_SHUTDOWN_TIMEOUT=15` | Number of seconds of inactivity to wait before shutting down the task runner process. The launcher will automatically start the runner again when there are new tasks to execute. Set to `0` to disable automatic shutdown. |
| `NODE_OPTIONS=--max-old-space-size=<limit>` | The memory limit for the task runner Node.js process. This should be lower than the limit for container so that the runner runs out of memory before the container. That way, the launcher is able to monitor the runner. |

For full list of environment variables see [task runner environment variables](/hosting/configuration/environment-variables/task-runners/).
14 changes: 14 additions & 0 deletions docs/hosting/securing/hardening-task-runners.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
---
#https://www.notion.so/n8n/Frontmatter-432c2b8dff1f43d4b1c8d20075510fe4
title: Hardening task runners
description: "Harden task runners for better isolation for your self-hosted n8n instance."
contentType: howto
---

# Hardening task runners

[Task runners](/hosting/configuration/task-runners/) are responsible for executing code from the [Code node](/integrations/builtin/core-nodes/n8n-nodes-base.code/). While Code node executions are secure, you can follow these recommendations to further harden your task runners.

## Run task runners as sidecars in external mode

To increase the isolation between the core n8n process and code in the Code node, run task runners in [external mode](/hosting/configuration/task-runners#setting-up-external-mode). External task runners launch as separate containers, providing a fully isolated environment to execute the JavaScript defined in the Code node.
15 changes: 9 additions & 6 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1292,24 +1292,25 @@ nav:
- Configuration:
- Environment variables:
- Environment variables overview : hosting/configuration/environment-variables/index.md
- Binary data: hosting/configuration/environment-variables/binary-data.md
- Credentials: hosting/configuration/environment-variables/credentials.md
- Database: hosting/configuration/environment-variables/database.md
- Deployment: hosting/configuration/environment-variables/deployment.md
- Binary data: hosting/configuration/environment-variables/binary-data.md
- User management and 2FA: hosting/configuration/environment-variables/user-management-smtp-2fa.md
- Endpoints: hosting/configuration/environment-variables/endpoints.md
- External hooks: hosting/configuration/environment-variables/external-hooks.md
- Executions: hosting/configuration/environment-variables/executions.md
- Logs: hosting/configuration/environment-variables/logs.md
- External data storage: hosting/configuration/environment-variables/external-data-storage.md
- External hooks: hosting/configuration/environment-variables/external-hooks.md
- External secrets: hosting/configuration/environment-variables/external-secrets.md
- Logs: hosting/configuration/environment-variables/logs.md
- License: hosting/configuration/environment-variables/licenses.md
- Nodes: hosting/configuration/environment-variables/nodes.md
- Queue mode: hosting/configuration/environment-variables/queue-mode.md
- Security: hosting/configuration/environment-variables/security.md
- Source control: hosting/configuration/environment-variables/source-control.md
- External secrets: hosting/configuration/environment-variables/external-secrets.md
- Task runners: hosting/configuration/environment-variables/task-runners.md
- Timezone and localization: hosting/configuration/environment-variables/timezone-localization.md
- User management and 2FA: hosting/configuration/environment-variables/user-management-smtp-2fa.md
- Workflows: hosting/configuration/environment-variables/workflows.md
- License: hosting/configuration/environment-variables/licenses.md
- Configuration methods: hosting/configuration/configuration-methods.md
- Configuration examples:
- hosting/configuration/configuration-examples/index.md
Expand All @@ -1325,6 +1326,7 @@ nav:
- Configure webhook URLs with reverse proxy: hosting/configuration/configuration-examples/webhook-url.md
- Enable Prometheus metrics: hosting/configuration/configuration-examples/prometheus.md
- Supported databases and settings: hosting/configuration/supported-databases-settings.md
- Task runners: hosting/configuration/task-runners.md
- User management: hosting/configuration/user-management-self-hosted.md
- Logging and monitoring:
- Logging: hosting/logging-monitoring/logging.md
Expand All @@ -1347,6 +1349,7 @@ nav:
- Disable the API: hosting/securing/disable-public-api.md
- Opt out of data collection: hosting/securing/telemetry-opt-out.md
- Blocking nodes: hosting/securing/blocking-nodes.md
- Hardening task runners: hosting/securing/hardening-task-runners.md
- Starter Kits:
- AI Starter Kit: hosting/starter-kits/ai-starter-kit.md
- Architecture:
Expand Down
4 changes: 4 additions & 0 deletions styles/config/vocabularies/default/accept.txt
Original file line number Diff line number Diff line change
Expand Up @@ -197,6 +197,8 @@ Ortto
Oura
PDFs
Peekalink
Performant
performant
PGVector
PhantomBuster
Pinterest
Expand Down Expand Up @@ -226,6 +228,8 @@ Rekognition
requester
Rundeck
Salesmate
Sandboxed
sandboxed
Sekoia
Sendy
Serp
Expand Down
Loading