Skip to content

task env: add NOMAD_UNIX_ADDR var #25598

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .changelog/25598.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
```release-note:improvement
task environment: new NOMAD_UNIX_ADDR env var points to the task API unix socket, for use with workload identity
```
6 changes: 6 additions & 0 deletions client/taskenv/env.go
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,10 @@ const (

HostAddrPrefix = "NOMAD_HOST_ADDR_"

// UnixAddr is the task api unix socket, in the appropriate format
// for use in a NOMAD_ADDR (i.e. prefixed with "unix://")
UnixAddr = "NOMAD_UNIX_ADDR"

// IpPrefix is the prefix for passing the host IP of a port allocation
// to a task.
IpPrefix = "NOMAD_IP_"
Expand Down Expand Up @@ -620,10 +624,12 @@ func (b *Builder) buildEnv(allocDir, localDir, secretsDir string,
// Build the Nomad Workload Token
if b.workloadTokenDefault != "" {
envMap[WorkloadToken] = b.workloadTokenDefault
envMap[UnixAddr] = "unix://" + filepath.Join(secretsDir, "api.sock")
}

for name, token := range b.workloadTokens {
envMap[WorkloadToken+"_"+name] = token
envMap[UnixAddr] = "unix://" + filepath.Join(secretsDir, "api.sock")
}

// Copy and interpolate task meta
Expand Down
8 changes: 6 additions & 2 deletions client/taskenv/env_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -214,7 +214,7 @@ func TestEnvironment_AsList(t *testing.T) {
}
env := NewBuilder(n, a, task, "global").SetDriverNetwork(
&drivers.DriverNetwork{PortMap: map[string]int{"https": 443}},
)
).SetDefaultWorkloadToken("test-wi-token")

act := env.Build().List()
exp := []string{
Expand Down Expand Up @@ -263,6 +263,8 @@ func TestEnvironment_AsList(t *testing.T) {
fmt.Sprintf("NOMAD_ALLOC_ID=%s", a.ID),
fmt.Sprintf("NOMAD_SHORT_ALLOC_ID=%s", a.ID[:8]),
"NOMAD_ALLOC_INDEX=0",
"NOMAD_TOKEN=test-wi-token",
"NOMAD_UNIX_ADDR=unix://api.sock",
}
sort.Strings(act)
sort.Strings(exp)
Expand Down Expand Up @@ -342,7 +344,7 @@ func TestEnvironment_AllValues(t *testing.T) {
}
env := NewBuilder(n, a, task, "global").SetDriverNetwork(
&drivers.DriverNetwork{PortMap: map[string]int{"https": 443}},
)
).SetDefaultWorkloadToken("test-wi-token")

// Setting the network status ensures we trigger the addNomadAllocNetwork
// for the test.
Expand Down Expand Up @@ -453,6 +455,8 @@ func TestEnvironment_AllValues(t *testing.T) {
"NOMAD_ALLOC_INTERFACE_admin": "eth0",
"NOMAD_ALLOC_IP_admin": "172.26.64.19",
"NOMAD_ALLOC_ADDR_admin": "172.26.64.19:9000",
"NOMAD_TOKEN": "test-wi-token",
"NOMAD_UNIX_ADDR": "unix://api.sock",

// Env vars from the host.
"LC_CTYPE": "C.UTF-8",
Expand Down
32 changes: 32 additions & 0 deletions e2e/workload_id/input/api-nomad-cli.nomad.hcl
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# Copyright (c) HashiCorp, Inc.
# SPDX-License-Identifier: BUSL-1.1

job "task-api-nomad-cli" {
type = "batch"

group "grp" {
restart { attempts = 0 }
reschedule { attempts = 0 }
constraint {
attribute = "${attr.kernel.name}"
value = "linux"
}

task "tsk" {
driver = "raw_exec"
config {
command = "bash"
// "|| true" because failure to get a var makes nomad cli exit 1,
// but for this test, "Variable not found" actually indicates successful
// API connection.
args = ["-xc", "echo $NOMAD_ADDR; nomad var get nothing || true"]
}
env {
NOMAD_ADDR = "${NOMAD_UNIX_ADDR}"
}
identity { # creates unix addr
env = true # provides NOMAD_TOKEN
}
}
}
}
17 changes: 17 additions & 0 deletions e2e/workload_id/taskapi_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ import (
"testing"

"github.com/hashicorp/nomad/e2e/e2eutil"
"github.com/hashicorp/nomad/e2e/v3/cluster3"
"github.com/hashicorp/nomad/e2e/v3/jobs3"
"github.com/hashicorp/nomad/helper/uuid"
"github.com/shoenig/test"
"github.com/shoenig/test/must"
Expand All @@ -26,6 +28,7 @@ func TestTaskAPI(t *testing.T) {

t.Run("testTaskAPI_Auth", testTaskAPIAuth)
t.Run("testTaskAPI_Windows", testTaskAPIWindows)
t.Run("testTaskAPI_NomadCLI", testTaskAPINomadCLI)
}

func testTaskAPIAuth(t *testing.T) {
Expand Down Expand Up @@ -129,3 +132,17 @@ func testTaskAPIWindows(t *testing.T) {

must.StrHasSuffix(t, `"ok":true}}`, logs)
}

func testTaskAPINomadCLI(t *testing.T) {
cluster3.Establish(t,
cluster3.LinuxClients(1),
)
sub, _ := jobs3.Submit(t,
"./input/api-nomad-cli.nomad.hcl",
jobs3.WaitComplete("grp"),
)
logs := sub.TaskLogs("grp", "tsk")
test.StrContains(t, logs.Stdout, "unix:/") // from `echo $NOMAD_ADDR`
test.StrContains(t, logs.Stdout, "secrets/api.sock")
test.StrContains(t, logs.Stderr, "Variable not found") // api success
}
7 changes: 5 additions & 2 deletions website/content/docs/commands/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,8 @@ $ nomad -autocomplete-uninstall
Nomad's CLI commands have implied contexts in their naming convention. Because
the CLI is most commonly used to manipulate or query jobs, you can assume that
any given command is working in that context unless the command name implies
otherwise. For example, the `nomad job run` command runs a new job and the `nomad status` command queries information about existing jobs. Conversely,
otherwise. For example, the `nomad job run` command runs a new job and the
`nomad status` command queries information about existing jobs. Conversely,
commands with a prefix in their name likely operate in a different context.
Examples include the `nomad agent-info` or `nomad node drain` commands,
which operate in the agent or node contexts respectively.
Expand Down Expand Up @@ -64,7 +65,8 @@ may override these environment variables with individual flags.
#### Connection environment variables

- `NOMAD_ADDR` - The address of the Nomad server. Defaults to
`http://127.0.0.1:4646`.
`http://127.0.0.1:4646`. For unix sockets, as with the [task API][],
the format is either `unix:/path/to/api.sock` or `unix:///path/to/api.sock`.

- `NOMAD_REGION` - The region of the Nomad server to forward commands to.
Defaults to the Agent's local region
Expand Down Expand Up @@ -118,4 +120,5 @@ may override these environment variables with individual flags.
- `NOMAD_LICENSE` - The Nomad Enterprise license file contents as a string.


[task API]: /nomad/api-docs/task-api
[`tls` block in agent configuration]: /nomad/docs/configuration/tls
2 changes: 2 additions & 0 deletions website/content/partials/envvars.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
| `NOMAD_PARENT_CGROUP` | The parent cgroup used to contain task cgroups (Linux only) |
| `NOMAD_NAMESPACE` | Namespace in which the allocation is running |
| `NOMAD_REGION` | Region in which the allocation is running |
| `NOMAD_UNIX_ADDR` | Use this value as your `NOMAD_ADDR` to use `nomad` CLI with the [task API][]'s unix socket. The value is equivalent to `"unix://${NOMAD_SECRETS_DIR}/api.sock"`
| `NOMAD_META_<key>` | The metadata value given by `key` on the task's metadata. Any character in a key other than `[A-Za-z0-9_.]` will be converted to `_`. <br/> **Note:** this is different from [`${meta.<key>}`](/nomad/docs/runtime/interpolation#node-variables-) which are keys in the node's metadata. |
| `CONSUL_HTTP_TOKEN` | The tasks' Consul token. See [Consul Integration][consul] documentation for more details. |
| `CONSUL_TOKEN` | The tasks' Consul token. See [Consul Integration][consul] documentation for more details. This variable is deprecated and exists only for backwards compatibility. |
Expand Down Expand Up @@ -68,6 +69,7 @@ names such as `NOMAD_ADDR_<task>_<label>`
| `CONSUL_CLIENT_KEY` | Specifies the path to the Client Key certificate used for Consul communication. Will be automatically set if Nomad is configured with the `consul.share_ssl` option. |
| `CONSUL_TLS_SERVER_NAME` | Specifies the server name to use as the SNI host for Consul communication. Will be automatically set if Consul is configured to use TLS and the task is in a group using bridge networking mode. |

[task API]: /nomad/api-docs/task-api
[upstream]: /nomad/docs/job-specification/upstreams
[taskdirs]: /nomad/docs/runtime/environment#task-directories
[network-block]: /nomad/docs/job-specification/network
Expand Down