-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy pathHACKING
250 lines (213 loc) · 13.5 KB
/
HACKING
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
How to hack on bytemark-client
==============================
bytemark-client is a big go project with a lot of code (> 30,000 SLOC at last
count) and a lot of packages. It's divided up into two main big chunks -
the command-line client in cmd/bytemark, and the API client library in lib.
The API client library is based around go's built-in http package with very
little exciting about it. We have an interface called PrettyPrinter which all
of our wire types implement, and is used by the command-line client for output.
The command-line client is based around github.com/urfave/cli - it's very
useful to understand urfave/cli before you start working on bytemark-client.
In particular, focus on understanding the App and Command types within it.
==============
Control flow
==============
Here is an approximate description of what goes on during an invocation of the
client:
1. Before cmd/bytemark.main is run, all the init functions are run, which
compile slices of cli.Commands in the commands/* packages and cmd/bytemark.
2. cmd/bytemark.main() is run, which runs some other functions in
cmd/bytemark to set up a cmd/bytemark/config.Manager and flag.FlagSet
3. cmd/bytemark.Commands() combines the commands slices together, with
cmd/bytemark/commands/admin if the --admin flag is set, then creates a
urfave/cli.App using cmd/bytemark/app.BaseAppSetup
4. cmd/bytemark.main creates a new lib.Client, then attaches that and the
config.Manager to the urfave/cli.App.
5. cmd/bytemark.main runs the cli.App, which calls the relevant command's Action
or returns help, whatever is needed.
6. Most command actions are made using cmd/bytemark/app.Action, chaining together
functions from cmd/bytemark/app/args and cmd/bytemark/app/with - which are
used to collect and parse data from the command line and to get data from the
API.
7. The final function in the command's Action chain then performs whatever task
the purpose of the command is.
8. Any errors in the cmd/bytemark/app.Action chain bubble back up through the
original invocation of urfave/cli.App.Run to cmd/bytemark.main()
9. cmd/bytemark.main() then calls cmd/bytemark/util.ProcessError, which attempts
to understand and present the error (if there was one) in a nice,
human-readable form.
Steps 1, 6 and 7 kind of gloss over a lot of detail. Let's look at a more
specific example.
`bytemark show account` is a command which shows the end-user details about
their account and the groups and servers it contains. It's a pretty simple and
pretty normal command, in terms of its implementation. It's defined in show.go
Here's the source for its Action. (app here being cmd/bytemark/app)
Action: app.Action(args.Optional("account"), with.Account("account"), func(c *app.Context) error {
return c.OutputInDesiredForm(c.Account)
}),
app.Action returns a `func(c *cli.Context) error`, and takes
`(func(c *app.Context) error)...` as arguments. The function returned by
app.Action wraps the cli.Context in an app.Context, then executes each function
in order, stopping at the first error. args.Optional("account") reads the next
command line argument if there is one and sets the flag named "account" with
it. with.Account("account") takes the "account" flag and attaches it to the
app.Context which is being passed to each of the providers - this sets c.Account
which is then used by the final function in this chain, which runs
c.OutputInDesiredForm.
app.Context.OutputInDesiredForm is a complicated function which looks at the
--output-type global flag to determine how to output the account, and then does
so.
==============
Code overview
==============
Here's a tree of the folders in this repo with descriptions of what each is for
.
├── cmd - empty package
│ └── bytemark - the main package - glue code to stick the commands together
│ │ and run an app. At time of writing, still also contains some
│ │ implementations of commands which ought to be refactored into
│ │ the commands subfolder.
│ ├── app - setup functions for our cliApp
│ │ │ github.com/urfave/cli, plus our Context which is full of
│ │ │ useful methods for dealing with flags & output
│ │ ├── args - functions for chaining together using app.Action to parse
│ │ │ arguments into flags
│ │ ├── auth - Authentication routine for the client, called via
│ │ │ with.EnsureAuth
│ │ ├── flags - new flag types that are used in the client (e.g.
│ │ │ │ VirtualMachineName, SizeSpec, ResizeSpec, AccountName,
│ │ │ │ Privilege)
│ │ │ └── gen - packages to generate flag code
│ │ │ └── slice_flags - generator for SliceFlags - currently
│ │ │ AccountName, GroupName and VirtualMachineName.
│ │ │ More can be added by altering
│ │ │ app/flags/slice_flags.go and you may need to add
│ │ │ more preparation code to
│ │ │ gen/slice_flags/template_test.go.tmpl
│ │ │ Whenever you alter anything in slice_flags, run
│ │ │ `go generate ./...` from the bytemark-client dir
│ │ │ to make sure the slice flags are kept up-to-date.
│ │ ├── flagsets - common sets of flags & code to support reading complex
│ │ │ objects from them. e.g. VirtualMachineSpec & ImageInstall
│ │ ├── wait - functions which wait for a condition to be true before
│ │ │ returning
│ │ └── with - functions for chaining together using app.Action to
│ │ attach things to the Context by parsing flag values.
│ ├── cliutil - workarounds for deficiencies in urfave/cli
│ ├── commands - contains implementations for all commands, either
│ │ │ directly for commands with no subcommands, or indirectly
│ │ │ through its subpackages for commands with subcommands
│ │ ├── add - implementations of all the "add X" commands
│ │ ├── admin - contains all the admin commands in admin.Commands
│ │ │ ├── add - contains all the admin add commands for adding
│ │ │ │ resources that only an admin would have permission to
│ │ │ ├── migrate - contains all the migrate commands for moving
│ │ │ │ things around on bigv
│ │ │ └── show - contains admin show commands
│ │ ├── delete - implementations of all the "delete X" commands
│ │ ├── show - implementations of all the "show X" commands
│ │ └── update - implementations of all the "update X" commands
│ ├── config - manages configuration from the environment, global flags
│ │ and config dir.
│ ├── testutil - helpers for cmd/bytemark tests
│ └── util - various utilities which should be their own packages.
│ │ including: CallBrowser - a function for opening a browser
│ │ to a given page on any OS
│ │ Prompter - utility interface for prompting
│ │ for input including passwords
│ │ Some additional parsing functions and flag types
│ │ that don't require access to an app.Context
│ └── sizespec - Parser for size specifications (e.g 35GiB, +3MiB)
├── doc - folder of documentation stuff.
│ Also includes the manpage source (bytemark.asciidoc)
├── gen - scripts to aid the building/distributing process. In particular, has
│ │ a changelog.sh script to generate a new changelog entry with all the
│ │ changes since the last commit to master.
│ └─ list_types - package to generate code for a list of types using sprintf.
├── lib - the main API client object, which makes requests.
│ │ At the moment, loads of requests are defined as methods on lib.Client,
│ │ but we want to move away from that and towards requests being
│ │ functions which take a lib.Client.
│ ├── billing - types for use with the bmbilling server
│ ├── brain - types for use with the Bytemark Cloud Servers brain
│ ├── output - Functions and types related to human-readable output of the
│ │ │ other types in lib.
│ │ │ DefaultFieldsHaver interface defined and tested-for here
│ │ │ (required for table & list output types)
│ │ ├── morestrings - extra string manipulation functions used by multiple
│ │ │ packages
│ │ └── prettyprint - PrettyPrinter interface and a helper function for
│ │ writing PrettyPrinters
│ │ All the types in lib/billing, lib/brain and lib/spp
│ │ should implement PrettyPrinter
│ ├── pather - Types whose exclusive purpose is to identify API objects and
│ │ provide URLs for them. Key types include AccountName,
│ │ GroupName, and VirtualMachineName.
│ ├── requests - empty package
│ │ ├── billing - functions that make requests against the billing endpoint
│ │ └── brain - functions that make requests against the brain endpoint
│ ├── spp - types & requests for use with the spp server
│ ├── testutil - helpers for HTTP tests in lib
│ │ ├── assert - assertion helpers for tests. with emphasis on HTTP tests
│ │ │ from testutil.
│ │ └── helpers - Parameterised RequestTestSpecs to make writing
│ │ MutiRequestTestSpecs a bit more readable.
│ └── util - helpers for lib
├── mocks - mock types used for testing in various packages
└── util - empty package
└── log - mostly-deprecated logging functions. Needs some refactoring,
should still be used by lib but not by cmd/bytemark and
subpackages.
==============
Quality testing
==============
Some automated quality tests are performed by gitlab-ci. To run them yourself,
run the following commands:
go install github.com/BytemarkHosting/bytemark-client/...
go test -run 'TestQuality.*' -tags quality
The quality build tag is used to signify that the `go install` step is needed.
It's needed to ensure that compiled code ends up in the $GOPATH/pkg dir, where
it can be read by the `go/importer` package.
Here's a probably-not-comprehensive description of the quality tests that are
performed:
lib/interface_test.go ensures that the lib.Client interface doesn't have any
new methods added to it.
lib/output/types_test.go generates a test from lib/output/types_test.go.inc and
runs it. This test ensures that all types in lib/brain, lib/billing and lib/spp
implement DefaultFieldsHaver and PrettyPrinter, with a few exceptions which are
specified near the top of lib/output/types_test.go.inc
==============
Notes on the direction we're trying to take
==============
In the early days of bytemark-client we were simply trying to achieve feature-
parity with the panel & the old bigv-client.
2.0 wasn't a huge change - just a few small breaking changes in lib.
During 2.x we started adding a lot of admin commands, which exacerbated a
problem we already knew existed - that cmd/bytemark and lib were both getting
kinda bloated - many multiple-hundred-line files per package.
Between 2.x and 3.0 we split out a lot of the utility-functions from
cmd/bytemark to allow it to be split up. We also declared a moratorium on
adding more methods to the lib.Client interface and cleaned up the tests to
allow requests to be written in any package. See the 'update
billing-definition' command in update.go and the UpdateDefinition and
GetDefinition functions in lib/requests/billing, and the tests for those, for
examples on how to structure code in this new way.
3.0 was primarily about fixing our slightly wacky command naming.
The focus of 4.0 is going to be carving up cmd/bytemark and lib into multiple
smaller packages, and splitting the files within up into smaller ones - one per
request type.
Maybe in 4.0 we'll also split the bytemark-client repo up in two:
github.com/BytemarkHosting/client - which will contain the current cmd/bytemark
package and subpackages.
github.com/BytemarkHosting/bmapi - which will contain the current lib package.
==============
Making sure we update this document
==============
At the end of this file is a hash generated from the list of directories
that make up the client. This document should be updated whenever any
are added or removed. The hash is tested every time gitlab-ci runs.
To generate the hash, use the following command.
find . -type d \! -path './.*' \! -path './vendor/*' | sort | sha256sum
or on macos
find . -type d \! -path './.*' \! -path './vendor/*' | sort | shasum -a 256
# sha256sum: 97818f7e5ab27fa5c6e37840ee1159f0cac168c80e699e1f7f262738e2fea563