Automated functional tests

[vepatel]

As the product becomes more established, it'll become more and more important to make sure all the features are behaving in a defined way, as we add more features and tweak existing ones it'll be of utmost importance that we don't break anything from earlier unexpectedly. This suite will be different from conformance test suite as the focus is more on the NKG functionality(config etc) not Gateway API itself.

Here's a initial proposal for that:

• Create a test library which can setup the product by the means of API call not kubectl,
• Create a common framework which can be then used to write tests (eg. create common resources like namespaces, secrets etc.)
• Automates the deployment of current examples as the initial test suite.

Let me know what community thinks :)

[pleshakov]

Create a test library which can setup the product by the means of API call not kubectl,

What would be the benefits of using the API for that?

I wonder if should try to use helm, since it's going to be our primary way to install NKG

[vepatel]

Benefits of API imo is the ability to install components independently like secrets, deployments etc. but product can be setup by means of helm completely then its fine 👍🏼
Still be requiring k8s api based functions for other resources

[pleshakov]

Create a common framework which can be then used to write tests (eg. create common resources like namespaces, secrets etc.)

I wonder if it makes sense to reuse the framework included with the Gateway API?

[pleshakov]

Automates the deployment of current examples as the initial test suite.

I don't think it will bring much value to write tests based on current examples - because the conformance tests is a superset of those examples

However, what if we have a way to make examples runnable - some way to use those source files and the README as the test cases and run them in the pipeline? This way if break examples or they stop working for some other reason, we will know quickly

[pleshakov]

List of proposed tests:

[pleshakov]

#944

• App upgrade via Deployment rolling upgrade
• App upgrade via blue green service selector update
• App upgrade via backendRef change
• App upgrade via canary release with fully promoting canary at the end

For those cases, consider:

• high traffic volume (thousands of RPS)
• variety of requests (different sizes (POST with body and GET), client latency)
• variety of responses (different sizes (from KBs to MBs), different server latency (example, from ms to seconds))

[pleshakov]

Test websocket support
#962