StressTestTool Tutorial

The JImmutableStressTestTool is a test program designed to weed out hard-to-find bugs in the library. It runs an infinite loop that feeds data to every implementation of every JImmutable Collection type, queries the data, and then deletes the data to verify that the collection always contains what it should. The goal of the StressTest is to find any problems that were not caught in the unit tests.

The StressTestTool has a StressTest for every JImmutable data type: JImmutableArray, JImmutableListMap, JImmutableList, JImmutableMap, JImmutableMultiset, JImmutableRandomAccessList, JImmutableSetMap, JImmutableSet, and JImmutableStack. Each StressTestable then tests each implementation of its corresponding data type. For example, JImmutableMapListStressTest tests the JImmutableHashListMap, JImmutableInsertOrderListMap, and the JImmutableTreeListMap.

Some base classes are run by two different StressTests. For example, JImmutableHashMultiset is tested by the JImmutableSetStressTest and the JImmutableMultisetStressTest. This is because JImmutableMultiset implements JImmutableSet. It is both a set and a multiset, so it is tested as both.

Every StressTest follows this same general structure:

int size = random.nextInt(100000);
for (int loops = 1; loops <= 6; ++loops) {
	growing phase
	updating phase*
	shrinking phase
	contains phase*
}
cleanup phase

*not in every StressTest

By the end of the for loop, the size of the JImmutable collection being tested should match the int size. During each loop, the collection will be grown by a sixth of size. The growing phase calls the collections insert methods randomly until it has grown by one third of size. This phase is in every StressTest. The updating phase is only used by StressTests for collections such as JImmutableMap, whose values can be changed without effecting the size of the collection. In this phase, existing values are replaced with new ones. For example, a key already in the JImmutableMap being tested would be picked at random and assigned a new value. The shrining phase calls the collections delete methods randomly until it has shrunk by one sixth of size. This phase is in every StressTest. The contains phase tests methods that do not change the size of the collection (ex. contains, containsAll, get, find, etc.). The StressTest randomly tests these methods to ensure that the expected result is returned. The cleanup phase runs once the for loop has completed and the collection has reached its goal size. This phase empties the collection of all values by randomly calling methods that reduce its size.

For convenience, the StressTestTool can be run from the command line. Options can be used to change the initial seed for the Random object used in the tests, to tell the StressTestTool what file to take tokens from, and to tell the Tool which StressTests to run. Jopt Simple was used for the command line parsing.

Commands:

--seed <long>
Used to set the seed used by Random in the first test. Useful when reproducing errors.

--file <filepath>
Adds a new file for the StressTestTool to tokenize. These tokens are then used to build Strings to store in the collections during the tests.

--list <optional>
Runs every JImmutableListStressTest. An optional argument may be added to run only a specific ListTest. These arguments are: arraylist, btreelist, treelist

--ralist <optional>
Runs every JImmutableRandomAccessListStressTest. An optional argument may be added to run only a specific RandomAccessListTest. These arguments are: btreelist, treelist

--set <optional>
Runs every JImmutableSetStressTest. An optional argument may be added to run only a specific SetTest. These arguments are: hashset, insertorderset, treeset, hashmultiset, insertordermultiset, treemultiset

--multiset, --mset <optional>
Runs every JImmutableMultisetStressTest. An optional argument may be added to run only a specific MultisetTest. These arguments are: hashmultiset, insertordermultiset, treemultiset

--map <optional>
Runs every JImmutableMapStressTest. An optional argument may be added to run only a specific MapTest. These arguments are: hashmap, comparablehashmap, badhashmap, comparablebadhashmap, insertordermap, treemap

--setmap, --smap <optional>
Runs every JImmutableSetMapStressTest. An optional argument may be added to run only a specific SetMapTest. These arguments are: hashsetmap, insertordersetmap, treesetmap

--listmap, --lmap <optional>
Runs every JImmutableListMapStressTest. An optional argument may be added to run only a specific ListMapTest. These arguments are: hashlistmap, insertorderlistmap, treelistmap.

--array <optional>
Runs every JImmutableArrayStressTest. An optional argument may be added to run only a specific ArrayTest. These arguments are: triearray, bit32array

--stack <optional>
Runs every JImmutableStack StressTest. Because there is only one implementation of JImmutableStack, this command has no optional arguments.

Additionally, any of the above arguments may also be entered as a command to run all of the corresponding StressTests for that implementation. For example, using the command --btreelist would run a JImmutableListStressTest and a JImmutableRandomAccessListStressTest on the JImmutableBtreeList. To run only one of these tests, a command and argument must be used. For example, --list btreelist would run only a JImmutableListStressTest on a JImmutableBtreeList.

Generally, command line options are not necessary. They are quite useful when try to reproduce errors, however. If a problem shows up with a specific test and seed, we can reproduce the problem so it can be fixed. For example, say a RuntimeException occurred for a JImmutableSetStressTest on a JImmutableInsertOrderMultiset with seed 1438368061318. The error could be duplicated with the following commands:

--seed 1438368061318 --set insertordermultiset

Which would give us the output:

Loaded 325756 tokens from 2 files

Starting with seed 1438368061318
JImmutableSetStressTest on JImmutableInsertOrderMultiset of size 83248
growing 0
checking contents with size 27701
shrinking 27701
checking contents with size 13814
contains 13814
checking cursor with size 13814
......

Before releasing a new version of JImmutable Collections, this StressTestTool is left running for an extended period of time to verify there are no errors that were not caught by the unit tests. If anyone wishes to verify the reliability of JImmutable, they may also run the Tool.