doc: add systemd chapter #142282

happysalada · 2021-10-20T02:06:21Z

Motivation for this change

Systemd has many features, some useful to create modules, but most features are hard to discover and use if you don't already know about them. This is an attempt at covering the basics that are useful in nixpkgs
see #102397 for more details.

Please note that this is still an overview, I would like feedback on

Potential secctions that you think are important but you feel are missing
Better order of the section to create a more natural flow.
Better place to put that documentation ?

I will populate each sections shortly, ideally there is consensus on the sections and their order before that.

Things done

rnhmjoj · 2021-10-20T12:32:52Z

Note that there are already a couple chapters about systemd in the manual. So, you may want to merge or move them.

jtojnar · 2021-10-20T15:15:15Z

Should not this rather go to NixOS manual?

doc/using/systemd.chapter.md

davidak · 2021-10-20T15:23:01Z

@happysalada thanks for starting this!

Can you make the PR a draft until it is ready to merge?

happysalada · 2021-10-20T16:43:37Z

This should indeed go to nixos manual.
let me also have a look at merging existing docs on it.
ryan also mentioned adding a chapter on tmpfiles on matrix.

Ma27 · 2021-10-23T14:07:13Z

doc/using/systemd.chapter.md

+
+From the [project's webpage](https://systemd.io/)
+```
+systemd is a suite of basic building blocks for a Linux system. It provides a system and service manager that runs as PID 1 and starts the rest of the system.


I guess, quoting (i.e. > systemd is [...]) is more correct here semantically, right?

oh instead of a triple backtick, sure, works for me.

Ma27 · 2021-10-23T14:08:05Z

doc/using/systemd.chapter.md

+  - Initialisation of state and dependents (e.g. database)
+    Talk about adding a systemd service to handle database initialisation
+  - Database management
+    Talk about connection via a unix socket for more performance and reduced secret management


Perhaps you should link these to the actual resource, correct?

I was thinking about adding links to examples in nixpkgs. What do you mean by actual resource ? Perhaps the article that explains why that is ? I don't think I've ever seen it explained all together, more like pieces of the puzzles everywhere. Some place tells you about default authentication method on postgres, and then I've seen somewhere the fact that performances are actually improved with a unix socket. I can probably find 2 references for each of those phenomenon.

What do you mean by actual resource

Anything where I can read up how to use the mentioned features :)

Also, while I appreciated the overall idea, a general suggestion: we shouldn't duplicate too much systemd docs here because we'll risk having outdated information here in that case, but for now this document seems quite OK (just thought I'd mention this) :)

Agreed on this. I'm going to try to just describe how things should be used, without giving too much details. Please comment if you see things that you think can potentially rot. It's important to not have documentation that requires too much maintenance.

happysalada · 2021-10-26T10:07:42Z

I implemented changes in a separate commit, but it's just for people involved in the PR to track changes more easily.
I'll squash all the commits when this is ready to merge.

I've moved the systemd chapter under nixos/doc/manual/development and put the systemd chapter right after the module chapter.

I've only populated 2 chapters and a half. The style is a little clumsy, I've tried to add the content I want to describe. Feel free to suggest better sentences, or just signal if you feel a section deserves more attention.

I tried to add examples to each section, but all the example basically refer to the one module that I wrote. I'm sure there are better examples, feel free to suggest any if a better modules comes to mind.

I've left the original section on systemd in the manual and added a little it. Basically the original section was describing how to do service management, so on the point of view of the cli. I added the logs section. I'm hoping that this section is more about systemd services options and how to use them.

jian-lin · 2021-11-07T19:17:38Z

nixos/doc/manual/development/systemd-environment-and-secrets.section.md

+To pass secret data to a systemd service, systemd has the `LoadCredential` mechanism.
+By passing a file path to systemd, it will make sure that the file is available to the process under a specific path.
+The load credential attribute takes an id a colon and a file path
+let's have a look at an [example](https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/web-apps/lemmy.nix#L182)


Maybe a permalink like this https://github.com/NixOS/nixpkgs/blob/3b0bff383f8e894b1ee7a30b62c8b727a68511df/nixos/modules/services/web-apps/lemmy.nix#L182 is better?

Thinking about those, I thought it would be better to just include the code-snippet rather than a link.
My next iteration will just have the relevant snippets just copied.

Let's at lest do a permalink for now so that the line doesn't get lost and we can do snippets later.

domenkozar · 2021-12-16T10:53:42Z

nixos/doc/manual/development/systemd-environment-and-secrets.section.md

+
+Note that this requires your application to read secrets from a file.
+
+If your service can only read secrets through environment variables, the best alternative is to ask the end user to pass an `environmentFile` containing the secrets.


Would be nice to document how that's done or link to an example

kevincox · 2021-12-16T13:28:05Z

nixos/doc/manual/development/systemd-environment-and-secrets.section.md

+To pass secret data to a systemd service, systemd has the `LoadCredential` mechanism.
+By passing a file path to systemd, it will make sure that the file is available to the process under a specific path.
+The load credential attribute takes an id a colon and a file path
+let's have a look at an [example](https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/web-apps/lemmy.nix#L182)


Let's at lest do a permalink for now so that the line doesn't get lost and we can do snippets later.

kevincox · 2021-12-16T13:30:52Z

nixos/doc/manual/development/systemd-environment-and-secrets.section.md

+### Secret management {#sec-systemd-secrets}
+
+To pass secret data to a systemd service, systemd has the `LoadCredential` mechanism.
+By passing a file path to systemd, it will make sure that the file is available to the process under a specific path.


It is probably a good idea to explicitly highlight that any path in the Nix store is world-readable so in order for this secret to be secret it needs to be managed outside of NixOS.

happysalada · 2021-12-16T13:47:44Z

Thanks a lot for commenting to both of you!

I have this task for the holidays to finish this PR.

I need to replace the links with the actual code block
I need to add 2 examples of
- How to pass an environment file
- How to use genJqSecretsReplacementSnippet (an alternative to replace a secret. There is an example in es7 (in a PR currently elk7: 7.11.1 -> 7.16.1, 6.8.3 -> 6.8.21 + add filebeat module and tests #150879).

roberth · 2021-12-16T23:05:34Z

nixos/doc/manual/development/systemd-state-management.section.md

+
+Most databases have 2 modes of connection, over TCP or over Unix socket. Connecting over a unix socket can provide 2 advantages
+- 30% performance improvement as packets don't need to be encoder over TCP
+- The default authentication mode postgres on unix sockets is just with the user name. This will remove the need to manage the password secret


Suggested change

- The default authentication mode postgres on unix sockets is just with the user name. This will remove the need to manage the password secret

- The default authentication mode postgres on unix sockets relies on the client's uid provided by the kernel, rather than a password that you need to be care for.

Though it being based on uid, this may be tricky across uid namespaces and therefore containers. I vaguely recall a nixpkgs issue or stackoverflow question about this, but I couldn't find it.

roberth · 2021-12-16T23:07:23Z

nixos/doc/manual/development/systemd.chapter.md

+Systemd is a building block of most nixos modules.
+
+From the [project's webpage](https://systemd.io/)
+```


This isn't code but a quote. You should probably use > syntax instead.

colemickens · 2021-12-17T00:34:56Z

A react emoji isn't enough -- thank you for this. this will be invaluable to have.

evils

some style points that caused me to hesitate while reading

it may be a bit early to start nitpicking
please ignore all of this if it'll take away time from substantive work
(especially the spell checking, which is non-exhaustive)
i'd love to do a more in-depth pass when this is closer to completion (or after it's merged)

thanks for doing this, i look forward to having it in the manual

evils · 2021-12-17T08:36:27Z

nixos/doc/manual/development/systemd-additional-resources.section.md

+[Systemd official website](https://systemd.io/)
+[Wikipedia](https://en.wikipedia.org/wiki/Systemd)
+[Archlinux Wiki](https://wiki.archlinux.org/title/Systemd)
+[Systemd announcement from Lennart Poettering (one of the 2 original developpers of systemd)](http://0pointer.de/blog/projects/systemd.html)


Suggested change

[Systemd announcement from Lennart Poettering (one of the 2 original developpers of systemd)](http://0pointer.de/blog/projects/systemd.html)

[Systemd announcement from Lennart Poettering (one of the 2 original developers of systemd)](http://0pointer.de/blog/projects/systemd.html)

evils · 2021-12-17T08:41:33Z

nixos/doc/manual/development/systemd-environment-and-secrets.section.md

+}
+```
+
+A file can be passed as well with `environmentFile`. If both are passed, `environmentFile` takes precedence over `environment`


Suggested change

A file can be passed as well with `environmentFile`. If both are passed, `environmentFile` takes precedence over `environment`

A file can be passed in with `environmentFile`. If both are used, `environmentFile` takes precedence over `environment`

evils · 2021-12-17T08:44:30Z

nixos/doc/manual/development/systemd-environment-and-secrets.section.md

+
+To pass secret data to a systemd service, systemd has the `LoadCredential` mechanism.
+By passing a file path to systemd, it will make sure that the file is available to the process under a specific path.
+The load credential attribute takes an id a colon and a file path


Suggested change

The load credential attribute takes an id a colon and a file path

The load credential attribute takes an id, a colon and a file path

evils · 2021-12-17T08:45:59Z

nixos/doc/manual/development/systemd-environment-and-secrets.section.md

+```
+LoadCredential = "jwt_secret:${cfg.jwtSecretPath}";
+```
+Inside your application, the jwt secret will be available under `$CREDENTIALS_DIRECTORY/jwt_secret` environment variable.


Suggested change

Inside your application, the jwt secret will be available under `$CREDENTIALS_DIRECTORY/jwt_secret` environment variable.

Inside your application, the jwt secret will be available under the `$CREDENTIALS_DIRECTORY/jwt_secret` environment variable.

evils · 2021-12-17T08:47:04Z

nixos/doc/manual/development/systemd-environment-and-secrets.section.md

+
+Note that this requires your application to read secrets from a file.
+
+If your service can only read secrets through environment variables, the best alternative is to ask the end user to pass an `environmentFile` containing the secrets.


Suggested change

If your service can only read secrets through environment variables, the best alternative is to ask the end user to pass an `environmentFile` containing the secrets.

If your service can only read secrets through environment variables, the best alternative is to ask the end user to pass in an `environmentFile` containing the secrets.

evils · 2021-12-17T08:59:44Z

nixos/doc/manual/development/systemd-handling-dependencies.section.md

+PartOf:
+- use `partOf = [ "B" ];`, if service A needs to re-started when service B is.
+
+Note that `wants,requires,bindsTo` are orthogonal to `after`. The most common occurence in nixpkgs is `requires` and `after`.


"and" makes it sound (to me) like the 2 most common occurrences are being listed

Suggested change

Note that `wants,requires,bindsTo` are orthogonal to `after`. The most common occurence in nixpkgs is `requires` and `after`.

Note that `wants,requires,bindsTo` are orthogonal to `after`. The most common occurrence in nixpkgs is `requires` with `after`.

evils · 2021-12-17T09:04:01Z

nixos/doc/manual/development/systemd-handling-dependencies.section.md

+### Examples {#sec-systemd-handling-dependencies-examples}
+
+- Web-app dependency on a database. If the web-app cannot start without the database, use `requires` and `after`. If you would rather have it operate even without a database use `wants` and `after`. [example in nixpkgs](https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/web-apps/lemmy.nix#L169)
+- Database migration and setup script for a web-app. It is common to make a service for database setup and migration. This service should be `partOf` the webapp service as everytime the webapp is restarted, this service should be restarted as well.


Suggested change

- Database migration and setup script for a web-app. It is common to make a service for database setup and migration. This service should be `partOf` the webapp service as everytime the webapp is restarted, this service should be restarted as well.

- Database migration and setup script for a web-app. It is common to make a service for database setup and migration. This service should be `partOf` the web-app service as every time the web-app restarts, this service should be restarted as well.

evils · 2021-12-17T09:15:01Z

nixos/doc/manual/development/systemd-handling-dependencies.section.md

+- Web-app dependency on a database. If the web-app cannot start without the database, use `requires` and `after`. If you would rather have it operate even without a database use `wants` and `after`. [example in nixpkgs](https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/web-apps/lemmy.nix#L169)
+- Database migration and setup script for a web-app. It is common to make a service for database setup and migration. This service should be `partOf` the webapp service as everytime the webapp is restarted, this service should be restarted as well.
+[example in nixpkgs](https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/web-apps/lemmy.nix#L214)
+- A web-app has two separate services, a UI and a server. The ui `requires` `after` the server. [example in nixpkgs](https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/web-apps/lemmy.nix#L205)


this is hard to parse as a sentence, also hard to keep it short :)

Suggested change

- A web-app has two separate services, a UI and a server. The ui `requires` `after` the server. [example in nixpkgs](https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/web-apps/lemmy.nix#L205)

- A web-app with two separate services, a UI and a server. The UI `requires` the server and starts `after` it. [example in nixpkgs](https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/web-apps/lemmy.nix#L205)

evils · 2021-12-17T09:25:03Z

nixos/doc/manual/development/systemd-state-management.section.md

+- 30% performance improvement as packets don't need to be encoder over TCP
+- The default authentication mode postgres on unix sockets is just with the user name. This will remove the need to manage the password secret
+
+Therefore the default for database connection for services is recommended to be with a unix socket. For postgres for example, the `host` needs to start with a `/`.


Suggested change

Therefore the default for database connection for services is recommended to be with a unix socket. For postgres for example, the `host` needs to start with a `/`.

Therefore the recommended default for database connections for services is a unix socket. With postgres for example, `host` needs to start with a `/`.

evils · 2021-12-17T09:26:31Z

nixos/doc/manual/development/systemd-state-management.section.md

+The initialisation service should be run by the database superuser so as not to require sudo.
+Typically it will involve creating the database and the user.
+There will be a step to see if the database is already created.
+That check is best implemented by going throug the created tables and exiting if the table already exists.


Suggested change

That check is best implemented by going throug the created tables and exiting if the table already exists.

That check is best implemented by going through the created tables and exiting if the table already exists.

con-f-use · 2023-04-03T21:06:08Z

Any reason this hasn't been merged?

happysalada · 2023-04-03T21:49:16Z

This was too early in the process its just incomplete. If anyone ever wants to take it up, they are more than welcome to take any part they want

ibizaman · 2024-07-31T11:03:26Z

FYI I'm working on #328472 and will pick this PR up afterwards if no one beats me to it, at least for the credentials part.

doc: add systemd chapter

3ef1377

github-actions bot added the 8.has: documentation This PR adds or changes documentation label Oct 20, 2021

happysalada mentioned this pull request Oct 20, 2021

Can Systemd 247 "credentials" logic solve secret management for us? #102397

Closed

ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux labels Oct 20, 2021

davidak reviewed Oct 20, 2021

View reviewed changes

doc/using/systemd.chapter.md Outdated Show resolved Hide resolved

happysalada marked this pull request as draft October 20, 2021 16:41

Ma27 reviewed Oct 23, 2021

View reviewed changes

move systemd chapter to nixos/development

692b73c

github-actions bot added the 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS label Oct 26, 2021

happysalada added 2 commits October 26, 2021 19:04

generated xml

cce5450

editor config checks

7e0abb6

add missing ids

f871c04

ofborg bot added 10.rebuild-linux: 1-10 and removed 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux labels Oct 27, 2021

jian-lin reviewed Nov 7, 2021

View reviewed changes

domenkozar mentioned this pull request Dec 16, 2021

Provide options for storing secrets outside the Nix store #24288

Open

75 tasks

domenkozar reviewed Dec 16, 2021

View reviewed changes

kevincox approved these changes Dec 16, 2021

View reviewed changes

roberth reviewed Dec 16, 2021

View reviewed changes

evils reviewed Dec 17, 2021

View reviewed changes

stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jun 19, 2022

stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Apr 3, 2023

asymmetric added the 6.topic: systemd label Apr 11, 2023

jian-lin mentioned this pull request Oct 10, 2023

nixos/synapse: doc: suggest LoadCredential before nixops/sops #260212

Open

1 task

wegank added the 12.approvals: 1 This PR was reviewed and approved by one reputable person label Mar 9, 2024

wegank added 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md 2.status: merge conflict This PR has merge conflicts with the target branch labels Mar 19, 2024

stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Mar 20, 2024

wegank added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jul 4, 2024

stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jul 31, 2024

wegank added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jan 1, 2025


		Note that this requires your application to read secrets from a file.

		If your service can only read secrets through environment variables, the best alternative is to ask the end user to pass an `environmentFile` containing the secrets.

	- The default authentication mode postgres on unix sockets is just with the user name. This will remove the need to manage the password secret
	- The default authentication mode postgres on unix sockets relies on the client's uid provided by the kernel, rather than a password that you need to be care for.

	[Systemd announcement from Lennart Poettering (one of the 2 original developpers of systemd)](http://0pointer.de/blog/projects/systemd.html)
	[Systemd announcement from Lennart Poettering (one of the 2 original developers of systemd)](http://0pointer.de/blog/projects/systemd.html)

	A file can be passed as well with `environmentFile`. If both are passed, `environmentFile` takes precedence over `environment`
	A file can be passed in with `environmentFile`. If both are used, `environmentFile` takes precedence over `environment`

	The load credential attribute takes an id a colon and a file path
	The load credential attribute takes an id, a colon and a file path

	Inside your application, the jwt secret will be available under `$CREDENTIALS_DIRECTORY/jwt_secret` environment variable.
	Inside your application, the jwt secret will be available under the `$CREDENTIALS_DIRECTORY/jwt_secret` environment variable.

	Note that `wants,requires,bindsTo` are orthogonal to `after`. The most common occurence in nixpkgs is `requires` and `after`.
	Note that `wants,requires,bindsTo` are orthogonal to `after`. The most common occurrence in nixpkgs is `requires` with `after`.

	- Database migration and setup script for a web-app. It is common to make a service for database setup and migration. This service should be `partOf` the webapp service as everytime the webapp is restarted, this service should be restarted as well.
	- Database migration and setup script for a web-app. It is common to make a service for database setup and migration. This service should be `partOf` the web-app service as every time the web-app restarts, this service should be restarted as well.

	- A web-app has two separate services, a UI and a server. The ui `requires` `after` the server. [example in nixpkgs](https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/web-apps/lemmy.nix#L205)
	- A web-app with two separate services, a UI and a server. The UI `requires` the server and starts `after` it. [example in nixpkgs](https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/web-apps/lemmy.nix#L205)

	Therefore the default for database connection for services is recommended to be with a unix socket. For postgres for example, the `host` needs to start with a `/`.
	Therefore the recommended default for database connections for services is a unix socket. With postgres for example, `host` needs to start with a `/`.

	That check is best implemented by going throug the created tables and exiting if the table already exists.
	That check is best implemented by going through the created tables and exiting if the table already exists.

doc: add systemd chapter #142282

Are you sure you want to change the base?

doc: add systemd chapter #142282

Conversation

happysalada commented Oct 20, 2021

Motivation for this change

Things done

rnhmjoj commented Oct 20, 2021 • edited Loading

jtojnar commented Oct 20, 2021

davidak commented Oct 20, 2021

happysalada commented Oct 20, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

happysalada commented Oct 26, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

happysalada commented Dec 16, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

colemickens commented Dec 17, 2021

evils left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

con-f-use commented Apr 3, 2023

happysalada commented Apr 3, 2023

ibizaman commented Jul 31, 2024 • edited Loading

rnhmjoj commented Oct 20, 2021 •

edited

Loading

happysalada commented Oct 26, 2021 •

edited

Loading

ibizaman commented Jul 31, 2024 •

edited

Loading