Unit Testing with scripts in EASE

In the company I work for we do lots of regression testing, mostly for interface testing of hardware devices. Therefore we needed a unit testbench similar to JUnit without the complexity of writing java code for our tests. So we implemented a simple Unit test framework for EASE.

Source code for this tutorial is available in the EASE scripts repository.

Step 1: Create a simple test script

Create a new Project in your workspace named JavaScript Unit Testing Tutorial. Create a new file Tests/Simple/01 Valid tests.js

// starts a simple test
startTest("empty", "an empty test case");
// ends a test
endTest();

// start another test
startTest("no test code", "a test containing no assertions");
print("Hi from valid test");
endTest();

// third test
startTest("valid assertions", "a test containing valid assertions");
// check
assertTrue(true);
endTest();

// code outside of a testcase
print('"' + getTestFile() + '" completed');

You might ask where these test functions come from as we do not load any modules. The unit test framework comes with a module called Unittest. It is hidden by default but will be loaded automatically when we execute EASE UnitTests. To unhide go to the Preferences/Scripting/Modules (Note: this is not necessary to use the module, it will only display it in UI components).

If you are familiar with unit testing in general you also know assertions and according methods like assertTrue(). In general these methods perform a check for validity. In case an expected result does not match with a current result, an error is generated for the containing test case.

So test cases are encapsulated between a startTest() and an endTest(). It is recommended to not use assertions outside test cases, yet not mandatory.

Step 2: Create and execute a test suite

To execute test files, we need to create a Test Suite, that sets up a test environment for us. Create a new Scripting/Script Testsuite named Testsuite.suite.  The editor will allow you to select test files you want to execute. Therefore we look for script files in the same project where the suite is located.

Select our previously created test file and hit the Run Test Suite button in the top right corner.

 A new view is opened that displays unit test results.

The top tree view reflects the file structure of our test files and displays decorators indicating the test execution result. If you select a test file, detailed results are displayed in the bottom section.

Step 3: Test errors

Create a new file Tests/Simple/02 Test errors.js.

// test containing assertions that fail
startTest("invalid assertions", "a test containing invalid assertions");
assertTrue(true);
assertTrue(false);
assertFalse(true);
assertFalse(false);
endTest();

// manually create a failure
startTest("failure", "test throwing a failure");
failure("test broken, stop execution here");

// an exception would also create a failure
throw new java.lang.Exception("code exception");
endTest();

startTest("never to be reached");
// not being reached as the failure above terminates test file execution
endTest();

Save the test, add it to your suite and run the suite right away. When an assertion fails, error markers are generated on your script files on the according code location. You will also see the most critical assertion reflected in the Script Unit Test view. Double clicking on errors will open the source editor on the corresponding location.

Similar to JUnit there exist two different types of things that can go wrong: errors and failures. Errors are assertions that do not meet their criteria. They are recoverable, meaning script execution of the test file continues.

Failures on the other hand are not recoverable, so script execution of that test file is terminated immediately.

Step 4: Test Suite configuration

Typically tests need to be adapted to a certain environment. In general you do not want test engineers to modify your script code. So the test suite supports setting of Variables that are automatically available in all test files. The Setup section allows to execute arbitrary code on the Test Suite start, Test File start, startTest() call and also for the corresponding teardown sections. As you may provide multiple suite files within a project, these secions are a good place to adapt tests according to your environment.

Optional: Test automation

Often tests need to be executed automatically, eg for a nightly test run. The Unittest module allows to run a testsuite from a script. Together with the headless application from EASE you can run a suite from the commandline and create a report (we currently support the JUnit xml format that can be picked up from Hudson/Jenkins).

Optional: Additional assertions

If you want to provide your own assertion methods, simply create your own module and let your wrapped method return an IAssertion. If you call your method from within a test script it will automatically validate your assertion method.

So why do I want to write scripted tests?

In the company I work for we use these kind of tests for interface verification of hardware devices. Therefore we provide our own modules that allow us to send stimuli and validate expected responses. We also use it in combination with analog measurement devices to verify that certain hardware parameters are within valid ranges.

Currently Script Unit Testing is supported for the Rhino script engine only.