Add support for config file (#135)

* First implemtation of a flag flag/config handling

* Adding an example config file

* Remove unused config package

* Move config stuff to a separate package

* Get rid of manually maintained flag list

* Load parses os args by default

* Add tests for config loading

* Updating readme and adding comments

README.md

@@ -24,8 +24,11 @@ You can also use Docker to install b7s. See the [Docker documentation](/docker/R
 
 ## Usage
 
+For a more detailed overview of the configuration options, see the [b7s-node Readme](/cmd/node/README.md#usage).
+
 | Flag                      | Short Form | Default Value           | Description                                                                                   |
 | ------------------------- | ---------- | ----------------------- | --------------------------------------------------------------------------------------------- |
+| config                    | N/A        | N/A                     | Specifies the config file to load.
 | log-level                 | -l         | "info"                  | Specifies the level of logging to use.                                                        |
 | peer-db                   | N/A        | "peer-db"               | Specifies the path to database used for persisting peer data.                                 |
 | function-db               | N/A        | "function-db"           | Specifies the path to database used for persisting function data.                             |

cmd/node/README.md

@@ -19,29 +19,70 @@ Head Nodes also serve a REST API that can be used to query or trigger certain ac
 
 ## Usage
 
+There are two ways of specifying configuration options - the CLI flags and a config file.
+CLI flags override any configuration options in the config file.
+
+List of supported CLI flags is listed below.
+
 ```console
-Usage of node:
-  -l, --log-level string               log level to use (default "info")
+Usage of b7s-node:
+      --config string                  path to a config file
   -r, --role string                    role this note will have in the Blockless protocol (head or worker) (default "worker")
+  -c, --concurrency uint               maximum number of requests node will process in parallel (default 10)
+      --boot-nodes strings             list of addresses that this node will connect to on startup, in multiaddr format
+      --workspace string               directory that the node can use for file storage
+      --attributes                     node should try to load its attribute data from IPFS
       --peer-db string                 path to the database used for persisting peer data (default "peer-db")
       --function-db string             path to the database used for persisting function data (default "function-db")
-  -c, --concurrency uint               maximum number of requests node will process in parallel (default 10)
-      --rest-api string                address where the head node REST API will listen on
-      --workspace string               directory that the node can use for file storage (default "./workspace")
-      --runtime string                 runtime address (used by the worker node)
-      --private-key string             private key that the b7s host will use
+      --topics strings                 topics node should subscribe to
+  -l, --log-level string               log level to use (default "info")
   -a, --address string                 address that the b7s host will use (default "0.0.0.0")
   -p, --port uint                      port that the b7s host will use
-      --boot-nodes strings             list of addresses that this node will connect to on startup, in multiaddr format
+      --private-key string             private key that the b7s host will use
+  -w, --websocket                      should the node use websocket protocol for communication
+      --websocket-port uint            port to use for websocket connections
       --dialback-address string        external address that the b7s host will advertise (default "0.0.0.0")
       --dialback-port uint             external port that the b7s host will advertise
       --websocket-dialback-port uint   external port that the b7s host will advertise for websocket connections
-  -w, --websocket                      should the node use websocket protocol for communication
-      --websocket-port uint            port to use for websocket connections
+      --runtime-path string            Blockless Runtime location (used by the worker node)
+      --runtime-cli string             runtime CLI name (used by the worker node) (default "bls-runtime")
       --cpu-percentage-limit float     amount of CPU time allowed for Blockless Functions in the 0-1 range, 1 being unlimited (default 1)
       --memory-limit int               memory limit (kB) for Blockless Functions
+      --rest-api string                address where the head node REST API will listen on
 ```
 
+Alternatively to the CLI flags, you can create a YAML file and specify the parameters there.
+All of the CLI flags have a corresponding config option in the YAML file.
+In the config file, parameters are grouped based on the functionality they impact.
+
+Example configuration for a worker node:
+
+```yaml
+role: worker
+concurrency: 10
+workspace: /tmp/workspace
+attributes: false
+
+log:
+  level: debug
+
+connectivity:
+  address: 127.0.0.1
+  port: 9000
+  private-key: ~/.b7s/path/to/priv/key.bin
+  websocket: true
+
+
+worker:
+  runtime-path: ~/.local/blockless-runtime/bin
+  cpu-percentage-limit: 0.8
+
+```
+
+You can find a more complete reference in [example.yaml](/cmd/node/example.yaml).
+
+### Configuration Option Details
+
 You can find more information about `multiaddr` format for network addresses [here](https://github.com/multiformats/multiaddr) and [here](https://multiformats.io/multiaddr/).
 
 Private key path relates to the private key created by the [keygen](/cmd/keygen/README.md) utility.

cmd/node/example.yaml

@@ -0,0 +1,66 @@
+# role this node will have in the network
+# role: head
+
+# how many requests should the node process in parallel
+# concurrency: 10
+
+# directory where node will keep files needed for operation
+# workspace: workspace
+
+# directory where node will maintain its peer database
+# peer-db: pdb
+
+# directory where node will maintain its function database
+# function-db: fdb
+
+# multiaddresses of nodes this node will try to connect to on boot
+# boot-nodes: []
+
+# should the node load its attributes from IPFS/IPNS
+# attributes: false
+
+# log information
+# log:
+  # level: debug
+
+# connectivity information
+# connectivity:
+
+  # address: 0.0.0.0
+  # port: 9000
+
+  # private key this node will use for its operation.
+  # this determines how this node is identified on the network
+  # private-key: /path/to/private/key
+
+  # external address that the node will advertise
+  # dialback-address: 10.10.10.10
+
+  # external port that the node will advertise
+  # dialback-port: 9000
+
+  # use websocket protocol for communication
+  # websocket: false
+
+  # port to use for websocket communication
+  # websocket-port: 9010
+
+  # external port the node will advertise for websocket communication
+  # websocket-dialback-port: 9010
+
+
+# head node configuration
+# head:
+  # where will the head node serve the REST API
+  # rest-api: localhost:8888
+
+# worker node configuration
+# worker:
+  # local path to Blockless Runtime
+  # runtime-path: /path/to/blockless/runtime
+
+  # max percentage of CPU time Blockless will use for execution (1.0 for 100%)
+  # cpu-percentage-limit: 1.0
+
+  # max amount of memory (in kB) Blockless will use for execution (0 is unlimited)
+  # memory-limit: 0

cmd/node/internal/config/config.go

@@ -0,0 +1,57 @@
+package config
+
+// Config type is tightly coupled with the config options defined in flags.go.
+// Flag name should be the same as the value in the `koanf` tag here (flag is `--dialback-address`, the koanf tag is `dialback-address`).
+// This is needed so the two ways of loading config are correctly merged.
+//
+// The `group` of the config option defines in which section of the config file it lives.
+// Examples:
+//		connectivity => address, port, private-key...
+//		worker => runtime-path, runtime-cli, cpu-percentage-limit...
+//
+
+// Config describes the Blockless configuration options.
+type Config struct {
+	Role           string   `koanf:"role"`
+	Concurrency    uint     `koanf:"concurrency"`
+	BootNodes      []string `koanf:"boot-nodes"`
+	Workspace      string   `koanf:"workspace"`  // TODO: Check - does a head node ever use a workspace?
+	LoadAttributes bool     `koanf:"attributes"` // TODO: Head node probably doesn't need attributes..?
+	Topics         []string `koanf:"topics"`
+
+	PeerDatabasePath     string `koanf:"peer-db"`
+	FunctionDatabasePath string `koanf:"function-db"` // TODO: Head node doesn't need a function database.
+
+	Log          Log          `koanf:"log"`
+	Connectivity Connectivity `koanf:"connectivity"`
+	Head         Head         `koanf:"head"`
+	Worker       Worker       `koanf:"worker"`
+}
+
+// Log describes the logging configuration.
+type Log struct {
+	Level string `koanf:"level"`
+}
+
+// Connectivity describes the libp2p host that the node will use.
+type Connectivity struct {
+	Address               string `koanf:"address"`
+	Port                  uint   `koanf:"port"`
+	PrivateKey            string `koanf:"private-key"`
+	DialbackAddress       string `koanf:"dialback-address"`
+	DialbackPort          uint   `koanf:"dialback-port"`
+	Websocket             bool   `koanf:"websocket"`
+	WebsocketPort         uint   `koanf:"websocket-port"`
+	WebsocketDialbackPort uint   `koanf:"websocket-dialback-port"`
+}
+
+type Head struct {
+	API string `koanf:"rest-api"`
+}
+
+type Worker struct {
+	RuntimePath        string  `koanf:"runtime-path"`
+	RuntimeCLI         string  `koanf:"runtime-cli"`
+	CPUPercentageLimit float64 `koanf:"cpu-percentage-limit"`
+	MemoryLimitKB      int64   `koanf:"memory-limit"`
+}