RFC structure of test inputs

[tiborsimko]

Note: This RFC stems from live discussions around #79 (comment)

Test inputs living as strings

Currently, the test suite mostly relies on example REST API outputs (which are reana-client-go inputs), living as strings. These are usually denoted successResponse, for example in info_test.go we have:

func TestInfo(t *testing.T) {
	successResponse := `{
	"compute_backends": {
		"title": "List of supported compute backends",
		"value": [
			"kubernetes",
			"slurmcern"
		]
	},
	"default_kubernetes_jobs_timeout": {
		"title": "Default timeout for Kubernetes jobs",
		"value": "124"
	},
	"default_kubernetes_memory_limit": {
		"title": "Default memory limit for Kubernetes jobs",
		"value": "248"
	},
	"default_workspace": {
		"title": "Default workspace",
		"value": "/var/reana"
	},
	"kubernetes_max_memory_limit": {
		"title": "Maximum allowed memory limit for Kubernetes jobs",
		"value": "1000"
	},
	"maximum_kubernetes_jobs_timeout": {
		"title": "Maximum timeout for Kubernetes jobs",
		"value": "500"
	},
	"maximum_workspace_retention_period": {
		"title": "Maximum retention period in days for workspace files",
		"value": "250"
	},
	"workspaces_available": {
		"title": "List of available workspaces",
		"value": [
			"/var/reana",
			"/var/cern"
		]
	}
}
`
	minimalResponse := `{
	"kubernetes_max_memory_limit": {
		"title": "Maximum allowed memory limit for Kubernetes jobs"
	},
	"maximum_workspace_retention_period": {
		"title": "Maximum retention period in days for workspace files"
	}
}
`

This has an advantage that the developer sees exactly what kind of input is being sent by the REST API endpoint, which facilitates comparing to real-life responses obtained from real servers like:

$ curl "https://reana.cern.ch/api/info?access_token=$REANA_ACCESS_TOKEN"

It is nice for debugging of corner-case situations not covered by "unit test" style of code coverage.

Test inputs co-generated by Swagger

Moreover, for a small number of tests, using strings to denote example inputs was not very practical due to length etc. We have called for a help from Swagger to generate part of the server response structure, as it were, such as in logs_test.go:

	successResponseRaw, _ := json.Marshal(operations.GetWorkflowLogsOKBody{
		Logs:         fmt.Sprintf(logsTemplate, "workflow_1", "job1", "workflow 1 logs"),
		User:         "user",
		WorkflowID:   "my_workflow_id",
		WorkflowName: "my_workflow",
	})
	successResponse := string(successResponseRaw)

This simplifies coding, but the developer does not really see the REST API endpoint output very clearly here. And, when the OpenAPI specs get updated and the server might send a different response, then the test case could get updated "silently" via updated Swagger client, without the developer even noticing. This may make it harder to test the behaviour of the client against various plethora of older and newer servers, say Client 1.2 with Servers 1.0, 1.1, 1.3, and 2.3 in the future.

This also means that we have a certain discrepancy in our testing strategy, the most tests using string inputs, and some tests don't. Which may cause some unnecessary confusion.

We may perhaps want to standardise on one approach. (Each has its pros and cons!)

RFC Test inputs stored independently as JSON files

Perhaps we should think about another option alongside harmonising the two successResponse-like technique with the successResponseRaw-like technique described above, and aim going beyond consistency. I'll describe one proposal that comes to mind.

For example, we could store JSON responses from servers in an example test data folder as independent files, such as:

• testdata/inputs/v0.8/info.json
• testdata/inputs/v0.8/list.json
• testdata/inputs/v0.8/list_include_workspace_size.json
• ...

for each concrete test case, and our test code would then read inputs and compare results against expected outputs (that could also be perhaps living as files in testdata/outputs/... directories).

This may have an advantage that if we as developers would need to deal with debugging some external REANA server, we can store its response into a file using the curl command above, and then go through it using our regular code base debugging techniques. And, after we fix the bug, we can retain this file in the test suite so that this bug won't reappear again, as a kind of new regression test case.

Note that storing inputs independently from Go files may also facilitate sharing of real-life test cases amongst Python and Go clients. (I'm looking in the direction of #8.)

So, if we go for something like that, we would have basically several different kind of tests in the suite: (a) typical Go langugae unit tests in the client, testing various helper functions we have developed, living as Go files; (b) various modelled REST endpoint outputs serving as the client real-life inputs, living as JSON files; and perhaps (c) expected good outputs matching those inputs, living as textual output files.

This was just an example to ease the brainstorming for the discussions happening live around #79 (comment) We can harmonise or one or the other, or choose a completely different proposal based on various inherent pros/cons that each approach has.

P.S. Note that some Go client projects are using a similar "test data" approach, e.g. see https://github.com/kubernetes/kubectl/tree/master/testdata