Test Automation

Test automation

Components

1. Trigger the execution
2. Run the tests
3. Publish the results

Trigger the tests

Expected: Should trigger the execution of the tests on iOS, Android and some browsers after each commit in the cocos2d-js-tests

Current Status: Not started

Run the tests

Expected: All the tests should be run

Current Status:

1. "Automated Test" button should be pressed
2. Different test files should be executed manually
3. Once each test file is opened, all the tests defined in that module are executed automatically

Publish Results

Expected: Results of the tests should be published in... TBD

Current Status: Not started

How to add new tests

BaseTestLayer class

In order to reuse the current cocos2d tests, an ad-hoc automation test was developed. This is a simple framework that iterates all over the tests that are defined in each test file.

The requirement is that each test should be a subclass of BaseTestLayer, and the following functions and properties shall be overridden:

    getExpectedResult:function() {
        return "Expected value";
    },
    getCurrentResult:function() {
        return "Current value";
    }

If the expected result is equal to the current result, the test will be marked as Ok, otherwise as a Fail.

You can override the compare function too, in case you to customize the comparison:

    // Default comparison. Override to have a comparison.
    // return true if current and expected are the same
    //  return false otherwise
    compareResults:function(current, expected) {
        return (current == expected);
    },

Duration of each test

By default, each test will be stopped 0.25 seconds after it was started. getExpectedResult() and getCurrentResult() will be called right after the test is stopped.

To change the duration of each test, you have to override the following property:

    // How many seconds should this test run
    testDuration:0.25,

Tests that do not depend on time, can have a testDuration of 0.1. Tests that depends on reading a property after an action finished should last as long as the action, plus a small value. eg:

    // Assuming that the action will last 2 seconds
    testDuration: 2 + 0.1,

Expected result

In order to be useful, the test should return a good expected result. Examples of "good" expected values:

• When testing actions, a "good" expected value is the final property (position, rotation, scale, color, opacity, etc...) of the target node.
• When testing rendering components (OpenGL tests, label alignment tests), capturing bytes from the screen is a "good" expected value.

In order to capture bytes, BaseTestLayer has a helpful API:

// returns an Uint8Array of w * h * 4 elements
readPixels:function(x,y,width,height) {}

// example
getCurrentResult:function() {
    // reads 4 * 4 pixels. Each pixel contains to 4 bytes (RGBA).
    var ret = this.readPixels(winSize.width/2-2, winSize.height/2-2,  4, 4);
    return JSON.stringify(ret);
}

Multiplatform

In order to be useful, the test should be compatible with iOS, Android and Web. When adding a new test make sure that at least it work on one mobile platform and Web. Tips:

• Do not use absolute coordinates for reading properties like position
• Reading pixels could be a challenge. OpenGL does not guarantee the same rendering results. So try to read only the needed pixels and no more than that. Use gl.NEAREST filter if needed
• Timed-precision tests could be very difficult to implement. When needed add "delays" in the test.

Working examples

• Ease Actions Test: Uses "delays" in order to be compatible between JSB and Web
• OpenGL Test: Uses readPixels for most of its test

Collaborate

• Go here, put your name in "owner" and start adding "expected results".