Finished test suite documentation

TESTING.md

@@ -26,7 +26,7 @@ You obviously need to build and install GixSQL, or use one of the binary package
 
 You can also build and install a version of GixSQL in a separate directory (e.g. `/opt/gixsql`) just for testing.
 
-#### Building the test runner
+### Building the test runner
 
 Both for Windows and Linux install the .Net 6.0 SDK from [here](https://dotnet.microsoft.com/en-us/download/dotnet/6.0). Depending on your preferences, on Linux you might install the distribution-supplied packages, but beware that these might not be up to date.
 
@@ -64,6 +64,7 @@ Here you will have to check and edit some parameters for the test runner:
 - **gixsql-install-base**:this is the location where GixSQL is installed
 - **keep-temps** if set to "1", temporary test files will be kept around (this is recommended)
 - **verbose**: generate (sometimes a lot) more output while running the tests. Normally leave it to "0"
+- **test-filter**: a comma-separated list of test case IDs. If left blank, all the tests in the suite will be run, unless they are filtered out by specific conditions on the individual test case
 - **dbtype-filter**: a comma-separated list of DB "types" for which the tests will be run. For example, if you are only intested in running tests for PostgreSQL and MySQL, use `<dbtype-filter>pgsql,mysql</dbtype-filter>`. The identifiers (also used in all the configuration elements where a "DB driver type" is needed) are `mysql`, `pgsql`, `odbc`, `oracle`, `sqlite`
 - **mem-check**: if set, the tests will be run under a memory checker (usually valgrind on Linux and DrMemory on Windows) using the command line supplied. here you can use several macros:
 
@@ -284,5 +285,144 @@ You only need to define the data sources for the DBMS type that you will test ag
 
 The `options` field contains the additional settings that are optionally passed with a standard GixSQL connection string, to enable, disable or configure specific options in the drivers. If you are using the standard test suite, you should change the connection information but leave the `options` field unchanged, to ensure that the standard tests will run successfully.
 
-(to be continued)
+### Test configuration
 
+If you only intend to run the standard test suite you can actually skip this sections and jump directly to *Running the tests*, since there is probably not a lot you would need to modify here. Nevertheless it is a good idea to have an idea of how the single test cases are configured
+
+Each run of a single test follows a pre-determined series of steps, some of which are optional:
+
+- Preparation of the environment: the test database is prepared by running scripts, payloads are generated, source files are extracted
+- Preprocessing: the source files (i.e. `gixpp`) are preprocessed
+- Compilation: the source files are compiled with the selected GnuCOBOL compiler
+- Run: the program is run and the output checked
+
+The standard test suite is defined in an XML file (`gixsql_test_data.xml`) that is embedded in the test runner when you compile it, there is no need to refer to it anywhere else. 
+
+*Note: future versions of the test runner will have the possibility to use a separate test definition file.*
+
+Each test case is defined inside a `test` XML element (this is a possibly non-working example, to illustrate the various features and parameters):
+
+		<test name="TSQL001A" enabled="true" applies-to="pgsql">
+			<description>Open connection, cursor</description>
+			<issue-coverage>#976</issue-coverage>
+			<group></group>
+			<architecture>all</architecture>
+			<compiler-type>all</compiler-type>
+
+			<cobol-sources>
+				<src name="TSQL001A.cbl" deps="EMPREC.cpy" />
+			</cobol-sources>
+
+			<data-sources count="1" />
+
+			<environment>
+				<variable key="DATASRC" value="${datasource1-noauth-url}" />
+				<variable key="DATASRC_USR" value="${datasource1-username}" />
+				<variable key="DATASRC_PWD" value="${datasource1-password}" />
+			</environment>
+			
+			<generate-payload id=":p1" type="random-bytes" length="256" />
+
+			<pre-run-drop-table data-source-index="1">bintest</pre-run-drop-table>
+
+			<pre-run-sql-statement data-source-index="1">create table bintest (id number GENERATED BY DEFAULT as IDENTITY, data BLOB)</pre-run-sql-statement>
+
+			<pre-run-sql-statement data-source-index="1" params=":p1">insert into bintest (id, data) values (1, :p1)</pre-run-sql-statement>			
+
+			<pre-run-sql-file data-source-index="1">dbdata.sql</pre-run-sql-file>
+
+			<preprocess value="true" />
+			<compile value="true" />
+			<run value="true" />
+
+			<expected-output>
+				<line>CONNECT SQLCODE: +0000000000</line>
+				<line>SELECT SQLCODE : +0000000000</line>
+				<line>RES: 003</line>
+			</expected-output>
+		</test>
+
+
+
+- **test (attribute: name)**: the name/ID of the test case
+- **description**: the description of the test case
+- **group**: the group the test case belongs to (currently not used)
+- **architecture**: the architecture this test is enabled for
+- **compiler-type**: the compiler type this test is enabled for
+- **cobol-sources/src**: a single COBOL source module that will be compiled
+- **cobol-sources/src (attribute: deps)**: a comma separated list of dependencies, usually COPY or data files
+- **data sources (attribute: count)**: the number of data sources of the same type that will be used by the test case, defaults to 1
+- **environment**: a series of variable definitions in the format indicated. In this element it is possibile to use a few macros to populate the value that will be assigned to the variables. These values are taken from those defined in the local configuration for each data source:
+    -  `${datasource1-url}`: the complete connection string (in standard GixSQL format, e.g. `pgsql://[email protected]/mydb?optiion=value`) to a given data source. This macro embeds the authentication credentials in the connection string.
+    -  `{datasource1-noauth-url}`: same as above without the authentication credentials
+    -  `${datasource1-type}`: the database type defined for a data source
+    -  `${datasource1-username}`: the username defined for a data source
+    -  `${datasource1-password}`: the password defined for a data source
+    -  `${datasource1-dbname}`: the database/schema name defined for a data source
+    -  `${datasource1-host}`: the server hostname defined for a data source
+    -  `${datasource1-port}`: the server port defined for a data source
+    -  `${datasource1-options}`: the connection options defined for a data source
+
+   In all these macros the number after `datasource` identifies the specific data source being referred (in this case "1"), starting with 1 and up to the number of data sources defined in the  `count` attribute of the `data-sources` element.
+
+- **generate-payload**: a payload that will be automatically generated (usually for use in a parameter). This payload can be of different types:
+
+    - random-bytes: generates a series of random bytes, whose length is defined by the `length` attribute. You can also specify a `min` and `max` attributes to restrict the values that will be generated (normally from 0 to a 255)
+    - `random-string`: generates a random string in the range `[A-Z0-9]` whose length is defined by the `length` attribute.
+    - `byte-sequence`: generates a sequence of bytes whose start is defined by the `start` attribute (default is 0) and whose length is defined by the `length` attribute.
+
+- **pre-run-drop-table**: the name of a single table that will be dropped before running the test. The `DROP` command will be issued on the data source defined by the `data-source-index` attribute (defaults to 1)
+- **pre-run-sql-statement**: an SQL statement that will be executed before running the test. The statement text can use the parameters defined in the `generate-payload` elements. The statement will be executed on the data source defined by the `data-source-index` attribute (defaults to 1).
+- **pre-run-sql-file**: an SQL file that will be loaded and executed before running the test. The SQL file will be executed on the data source defined by the `data-source-index` attribute (defaults to 1). In the current versionof the test runner the SQL file itself must be located in the `data` folder inside the `gixsql-tests-nunit` project and will be emebedded in the test runner.  
+
+The `pre-run-drop-table`, `pre-run-sql-statement` and `pre-run-sql-file` elements can have an optional `type` attribute that restricts the operation to a specific DB type. This may be useful to account for the differences in syntax between DBMSs, especially with regard to DDL statements.
+
+- **preprocess**: set to `true` to perform the process phase
+- **compile**: set to `true` to perform the compile phase
+- **run**: set to `true` to perform the run phase
+
+Some tests are only meant to check for syntax problem, so there is no need to run or even compile them, you can disable the correspopnding phases here.
+
+- **expected-output**: this section is made of `line entries` that will be matched against the program output to assert a successful program execution. The test runner normally checks the whole content of the `line` element, unless one of the following modifiers are present:
+
+    -  `{{RX}}`: the `line` element will be treated as a regular expression (e.g. `<line>{{RX}}^ABC.*$</line>`)
+    -  `{{SW}}`: the check will test if there is a line from the output that *starts with* the supplied text (e.g. `<line>{{SW}}ABC</line>`)
+    -  `{{NOT}}`: the check will test if there is NOT a line from the output that matches the supplied text (e.g. `<line>{{NOT}}ABC</line>`)
+
+
+### Running the tests
+
+After you have built the test runner and edited your local configuration, you should define where the local configuration can be found:
+
+On Linux:
+
+    export GIXTEST_LOCAL_CONFIG=/path/to/my_test_config.xml
+
+On Windows:
+
+    set GIXTEST_LOCAL_CONFIG=c:\my_test_dir\my_test_config.xml
+
+
+Then, assuming you are in the directory with the GixSQL source:
+
+    dotnet gixsql-tests-nunit/bin/Debug/net6.0/gixsql-tests-nunit.dll 
+
+The test runner should perform all the tests and output a report that should end like this:
+
+``` 
+...
+(#128) - TSQL039A/x64/gcc/odbc - Parser error on semicolon and error continuation does not work                                         : OK
+(#000) - TSQL040A/x64/gcc/odbc - Parser error on DECLARE TABLE statements                                                               : OK
+(#125) - TSQL041A/x64/gcc/odbc - Not possible to insert binary data (including low-value) or UTF16                                      : OK
+Run: 262 - Success: 0 - Failed: 
+```
+
+In case any of the tests failed,there will be a section of the report listing only the tests that didn't pass.
+
+In both cases you can examine the test results in the test temporary directory (the one you have supplied in the local configuration in the global/temp-dir element). There you will find:
+
+- the source file(s) for you test and its prepreocessed version (e.g `TSQL001A.cbl` and `TSQL001A.cbsql`) including any dependency
+- the compiled executable that was used for the test (e.g. `TSQL001A.exe`)
+- `stdout` and `stderr` files containing standard and error output from each of the three phases of the test run (preprocess, compile run)
+- a file named `gixsql-<test id>-<arch>-<db type>-<compiler type>.log` (e.g. `gixsql-TSQL001A-x64-mysql-msvc.log`) that contains the log output from the GixSQL library (tests are always run at `trace` level)
+- if the `mem-check` option was used, you should also find the output from your memory checker (e.g. `valgrind-TSQL001A-...`)