Test Automation Framework

Testing Scope

What do we need to test? Everything possible. The testing framework needs to be able to test countless configurations of plugins, engines, configuration variables, performance scenarious, and more. So, the framework needs to abstract away these variations so that the test-writer can focus on writing the test and not making the test specific to one plugin or engine.

Types of Tests

Types of tests:

• Functional/Use Case: test of a series of actions which should have a determinate behaviour
  • For instance, test that a CREATE TABLE, followed by a DROP TABLE, followed by a TRUNCATE TABLE results in an error.
• Unit Test: a test of a code section or class. The test tests whether the publicly described interface works as described, and that inputs and outputs of the interface are correct
  • For example, a unit test of the Lex class would test all public methods for proper input and output, and verify that state changes are consistent with the published API
• Bug Fix test: a test which verifies a bug's behaviour and demonstrates the correct behaviour.
  • For example, if a bug is issued which says that GROUP_CONCAT() does not function as advertised, the test case is written to first verify the behaviour, and then, when the bug is fixed in the code, that the code fixes the bug.
• Performance test: a test which tracks the regression or increase in performance/scalability of the server over time
• Stress test: a test which attempts to overload the server or determine the breaking points of the server

Areas to Test

Which test suites should be built?

• Replication
• Client/Server Communication
• Transactions
• Information Schema

Test Automation Framework

For designing a framework, various elements need to be taken into consideration. Some of them are:

• What actions need to be commonly performed per test run
• Communication with additional automation tools
• Communicating between test client and various other clients and servers
• Logging of the test run
• How Errors and Warnings should be handled?
• Standardized input and output of tests
• How to deal with dependencies within tests?

Functional Test Case Input Format

The existing system of using separate files for tests and for results will be kept. After considering a single file approach, it was determined that testing multiple engines, plugins, and options would make the single file approach untenable, as too many differences would need to be tested in a single test file.

The test case format is proposed as follows:

aggregate_no_rows.test:

# Tests that COUNT(*), AVG(), MIN(), MAX() on a table 
# with no rows returns correct results

SETUP ()
{
	EXECUTE_SQL("DROP TABLE IF EXISTS t1");
	EXECUTE_SQL("CREATE TABLE t1 (id INT NOT NULL)");
}

TEARDOWN ()
{
	EXECUTE_SQL("TRUNCATE t1");
}

TEST (count)
{
	EXECUTE_SQL("SELECT COUNT(*) FROM t1");
}

TEST (max)
{
	EXECUTE_SQL("SELECT MAX(id) FROM t1");
}

TEST (min)
{
	EXECUTE_SQL("SELECT MIN(id) FROM t1");
}

TEST (avg)
{
	EXECUTE_SQL("SELECT AVG(id) FROM t1");
}

Each test case file shall contain zero or one "SETUP() {}" section, containing test commands run before each test in the test case file is run. Similarly, zero or one "TEARDOWN() {}" section may be included that is run after each test in the test case file is run.

Comments are any line which begins with a # symbol

A "TEST(test_name) {}" section indicates a single test in the test case. Inside the parentheses, you should put a descriptive name for the single test. The name of the test will default to testN where N is the ordinal position of the test in the test case file.

Within each test in a test case file, you may place one or more test commands for the runner to execute in the test. Test commands include the following:

EXECUTE_SQL(sql_string);           # Executes the statement inside the parentheses
EXECUTE_SQL_FROM_FILE(filename);   # Execute all statements in a supplied file

Other possible commands which may be issued in a test case:

REQUIRE_PLUGIN(plugin_name);       # Ensure that a plugin is loaded and available in order to run the test case

Function Test Case Result Format

The new results format will follow a more standard xUnit way of testing (asserting) expected results. Here is an example results file which would accompany the above test:

aggregate_no_rows.result:

RESULT (count)
{
	ASSERT_ROWS(1);
	ASSERT_DATA_EQUALS(0,0,0);
}

RESULT (max)
{
	ASSERT_ROWS(1);
	ASSERT_DATA_ISNULL(0,0);
}

RESULT (min)
{
	ASSERT_ROWS(1);
	ASSERT_DATA_ISNULL(0,0);
}

RESULT (avg)
{
	ASSERT_ROWS(1);
	ASSERT_DATA_ISNULL(0,0);
}

Required Actions of Test Runner

Framework development is facilitated using the same set of identified tools. Scripting language supported by the test automation tools is used to create the components. Tool extensibility utility/component can be developed using a different language. In addition to the re-usable components, driver scripts and worker scripts need to be created. The approach for developing re-usable utilities/components should include:

• Record/Replay
• Screen/Window/Transaction
• Action/Keyword
• Data Driven

Thoughts on a new Test Runner

Basically, the existing test runner (/tests/test-run.pl) is pretty good. A new test runner should built on its foundation and clean it up to make it more extensible. The existing framework does the following things, which should be kept/emulated:

• Spawn a pool of threads to run individual test cases
• Allow a developer to run a specific test or a suite of tests

New concepts for a new Test Runner

A "suite" is a collection of tests that check a related feature or functional unit. For instance, "replication" or "transactions"

A "config" is an input to the test runner which sets or unsets a variety of parameters in the test run.

A "type" is the type of test (functional, unit, stress, etc)

Calling the test runner in various formats

# Run the functional replication tests
./test-runner --suite=replication --type=functional

# Run the "slave-api" unit test in the replication suite
./test-runner --type=unit suite=replication slave-api

# Run all the functional tests
./test-runner --type=functional

# Run all functional tests for the "MyISAM" configuration
./test-runner --type=functional --config=MyISAM

Structure of the Testing Framework

/ # root source directory
  /tests # root testing directory
    /runner # location for the actual test runner and framework
    /functional # location of functional test
    /performance # location of performance tests
    /stress  # location of stress tests
    /var # runtime location for test runner data and files

Thought: Unit tests should be in a /unit directory within each directory? For instance, server unit tests should be in /drizzled/unit ?

Existing Test Runner Options

Logging: ./dtr --help

./dtr [ OPTIONS ] [ TESTCASE ]

Options to control what engine/variation to run

  compress              Use the compressed protocol between client and server
  bench                 Run the benchmark suite
  small-bench           Run the benchmarks with --small-tests --small-tables

Options to control directories to use
  benchdir=DIR          The directory where the benchmark suite is stored
                        (default: ../../mysql-bench)
  tmpdir=DIR            The directory where temporary files are stored
                        (default: ./var/tmp).
  vardir=DIR            The directory where files generated from the test run
                        is stored (default: ./var). Specifying a ramdisk or
                        tmpfs will speed up tests.
  mem                   Run testsuite in "memory" using tmpfs or ramdisk
                        Attempts to find a suitable location
                        using a builtin list of standard locations
                        for tmpfs (/dev/shm)
                        The option can also be set using environment
                        variable MTR_MEM=[DIR]

Options to control what test suites or cases to run

  force                 Continue to run the suite after failure
  do-test=PREFIX or REGEX
                        Run test cases which name are prefixed with PREFIX
                        or fulfills REGEX
  skip-test=PREFIX or REGEX
                        Skip test cases which name are prefixed with PREFIX
                        or fulfills REGEX
  start-from=PREFIX     Run test cases starting from test prefixed with PREFIX
  suite[s]=NAME1,..,NAMEN Collect tests in suites from the comma separated
                        list of suite names.
                        The default is: "main,binlog,rpl"
  skip-rpl              Skip the replication test cases.
  big-test              Set the environment variable BIG_TEST, which can be
                        checked from test cases.
  combination="ARG1 .. ARG2" Specify a set of "mysqld" arguments for one
                        combination.
  skip-combination      Skip any combination options and combinations files

Options that specify ports

  master_port=PORT      Specify the port number used by the first master
  slave_port=PORT       Specify the port number used by the first slave
  mtr-build-thread=#    Specify unique collection of ports. Can also be set by
                        setting the environment variable MTR_BUILD_THREAD.

Options for test case authoring

  record TESTNAME       (Re)genereate the result file for TESTNAME
  check-testcases       Check testcases for sideeffects
  mark-progress         Log line number and elapsed time to <testname>.progress

Options that pass on options

  mysqld=ARGS           Specify additional arguments to "mysqld"

Options to run test on running server

  extern                Use running server for tests
  user=USER             User for connection to extern server

Options for debugging the product

  client-ddd            Start drizzletest client in ddd
  client-debugger=NAME  Start drizzletest in the selected debugger
  client-gdb            Start drizzletest client in gdb
  ddd                   Start mysqld in ddd
  debug                 Dump trace output for all servers and client programs
  debugger=NAME         Start mysqld in the selected debugger
  gdb                   Start the mysqld(s) in gdb
  manual-debug          Let user manually start mysqld in debugger, before
                        running test(s)
  manual-gdb            Let user manually start mysqld in gdb, before running
                        test(s)
  manual-ddd            Let user manually start mysqld in ddd, before running
                        test(s)
  master-binary=PATH    Specify the master "mysqld" to use
  slave-binary=PATH     Specify the slave "mysqld" to use
  strace-client         Create strace output for drizzletest client
  max-save-core         Limit the number of core files saved (to avoid filling
                        up disks for heavily crashing server). Defaults to
                        5, set to 0 for no limit.

Options for coverage, profiling etc

  gcov                  FIXME
  gprof                 See online documentation on how to use it.
  valgrind              Run the "drizzletest" and "mysqld" executables using
                        valgrind with default options
  valgrind-all          Synonym for --valgrind
  valgrind-drizzletest    Run the "drizzletest" and "drizzle_client_test" executable
                        with valgrind
  valgrind-mysqld       Run the "mysqld" executable with valgrind
  valgrind-options=ARGS Deprecated, use --valgrind-option
  valgrind-option=ARGS  Option to give valgrind, replaces default option(s),
                        can be specified more then once
  valgrind-path=[EXE]   Path to the valgrind executable
  callgrind             Instruct valgrind to use callgrind

Misc options

  comment=STR           Write STR to the output
  notimer               Don't show test case execution time
  script-debug          Debug this script itself
  verbose               More verbose output
  start-and-exit        Only initialize and start the servers, using the
                        startup settings for the specified test case (if any)
  start-dirty           Only start the servers (without initialization) for
                        the specified test case (if any)
  fast                  Don't try to clean up from earlier runs
  reorder               Reorder tests to get fewer server restarts
  help                  Get this help text

  testcase-timeout=MINUTES Max test case run time (default 15)
  suite-timeout=MINUTES Max test suite run time (default 180)
  warnings | log-warnings Pass --log-warnings to mysqld

  sleep=SECONDS         Passed to drizzletest, will be used as fixed sleep time