DejaGnu is entirely written in expect, which uses Tcl as a
command language. expect serves as a very programmable shell;
you can run any program, as with the usual Unix command shells--but
once the program is started, your expect script has fully
programmable control of its input and output. This does not just apply
to the programs under test; expect can also run any auxiliary
program, such as diff or sh, with full control over its
input and output.
DejaGnu itself is merely a framework for the set of test suites distributed separately for each GNU tool. Future releases of GNU tools will include even more tests, developed throughout the free software community.
runtest is the glue to tie together and manage the test scripts.
The runtest program is actually a simple Bourne shell script that
locates a copy of the expect shell and then starts the main Tcl
code, runtest.exp. runtest.exp itself has these essential
functions:
runtest.exp locates the tests
by exploiting a straightforward naming convention based on the string
you specify with the `--tool' option.
expect.
DejaGnu uses `$tool', the name of the tool under test, to tie together the testing configuration in a straightforward but flexible way.
`$tool' is not used to invoke the tool, since sites that run
multiple configurations of a particular tool often call each
configuration by a different name. runtest uses the
configuration-dependent variables captured in `site.exp' to
determine how to call each tool.
runtest uses tool names to find directories containing tests.
runtest scans the source directory (specified with
--srcdir) for all directories whose names start with the tool
name followed by a period. For instance, directories that start with
`g++.' contain G++ tests. To add a new test, just put it in
any directory (create an entirely new directory, if you wish) whose name
follows this convention.
A test is any file in an appropriately named subdirectory whose name
ends in `.exp' (the conventional way of naming expect
scripts). These simple naming conventions make it as simple as possible
to install new tests: all you must do is put the test in the right
directory.
runtest sorts the tests in each subdirectory by name (using the
Tcl lsort command) and runs them in the resulting order.
The initialization module (or "init file") has two purposes: to provide tool and target dependent procedures, and to start up an interactive tool to the point where it is ready to operate. The latter includes establishing communications with the target. All the tests for interactive programs assume that the tool is already running and communicating. Initialization modules for non-interactive programs may only need to supply the support functions.
Each test suite directory must contain (in its `config'
subdirectory) a separate initialization module for each target. The
appropriate init file is identified by name: it must be named
`$target_abbrev.exp', where target_abbrev is the Tcl
variable recording the abbreviated configuration name for the target
operating system.
If the test suite contains tests for more than one tool, it can either
use a single init file for all the tools, or segregate the init files by
tool using a more elaborate naming convention:
`$target_abbrev-$tool.exp', where tool is the Tcl variable
identifying tests for a particular tool. It is better to avoid this
convention if possible, since it may lead to unacceptable constraints on
tool names due to systems with low limits on file-name length. It is more
portable to segregate tests for different tools into different test-suite
directories, instead.
At the beginning of the init file, you must first determine the proper executable name of the tool to execute, since the actual name of the tool to be tested my vary from system to system. Here's an example for the GNU C compiler.
global CC
if ![info exists CC] then {
set CC [transform gcc]
}
global CFLAGS
if ![info exists CFLAGS] then {
set CFLAGS ""
}
It is always a good idea to first check the variable, and only set it if
it has not yet been defined. Often the proper value of CC is set
on the command line that invokes `runtest'.
The transform procedure takes as its argument the native name of
a tool (such as `gcc' for the compiler), and returns the name as
configured for that tool in the current installation. (For example, a
cross-compiling version of GNU CC that generates MIPS code may be
installed with a name like mips-idt-ecoff-gcc.)
In a test running native, writing the Tcl code for initialization is usually quite simple. For cross configurations, however, more elaborate instructions are usually needed to describe how to talk to a remote target.
Each initialization module defines up to four procedures with standard
names and purposes. The names of these procedures begin with
`$tool', the string that identifies tests for a particular tool:
$tool_start, $tool_load, $tool_exit, and
$tool_version. For example, the start procedure for GDB is
called gdb_start. (Since start procedures are used differently
for batch and interactive tools, however, runtest itself never
calls the start procedure. Init files for interactive tools are
expected to end by running the start procedure.)
The initialization module is also a good place to call load_lib
to get any collections of utility procedures meant for a family of test
cases, and to set up default values for any additional Tcl variables
needed for a specific set of tests.
See section Target dependent procedures, for full descriptions of these procedures.
DejaGnu provides these Tcl procedures for use in test scripts.
You can also use any standard expect or Tcl function.
See section A POSIX conforming test framework, for more detailed explanations of the test outcomes (`FAIL', `PASS', `UNTESTED', `UNRESOLVED', `UNSUPPORTED').
perror "string"
perror writes in the log files a message beginning with
`ERROR', appending the argument string.
As a side effect, perror also changes the effect of the next
pass or fail command: the test outcome becomes
`UNRESOLVED', since an automatic `PASS' or `FAIL' cannot
be trusted after a severe error in the test framework.
warning "string"
warning writes in the log files a message beginning with
`WARNING', appending the argument string. Use warning
rather than error for cases (such as communication failure
to be followed by a retry) where the test case can recover from the
error.
As a side effect, three or more calls to warning in a single test
case also change the effect of the next pass or fail
command: the test outcome becomes `UNRESOLVED', since an automatic
`PASS' or `FAIL' may not be trustworthy after many warnings.
pass "string"
pass writes in the
log files a message beginning with `PASS' (or XPASS, if
failure was expected), appending the argument string.
fail "string"
fail writes in the
log files a message beginning with `FAIL' (or XFAIL, if
failure was expected), appending the argument string.
unresolved "string"
unresolved writes
in the log file a message beginning with `UNRESOLVED', appending
the argument string. This usually means the test did not execute
as expected, and a human being must go over results to determine if it
passed or failed (and to improve the test case).
untested "string"
untested writes in the log file a
message beginning with `UNTESTED', appending the argument
string. For example, you might use this in a dummy test whose
only role is to record that a test does not yet exist for some feature.
unsupported "string"
unsupported writes in the log file a
message beginning with `UNSUPPORTED', appending the argument
string.
transform "toolname"
runtest as
`m68k-vxworks-runtest', the result of ` transform "gcc" '
is `m68k-vxworks-gcc'.
ishost "host"
1;
otherwise the result is 0. host must be a full three-part
configure host name; in particular, you may not use the shorter
nicknames supported by configure (but you can use wildcard
characters, using shell syntax, to specify sets of names).
istarget "target"
1;
otherwise the result is 0. target must be a full
three-part configure target name; in particular, you may not use
the shorter nicknames supported by configure (but you can use
wildcard characters, using shell syntax, to specify sets of names).
isnative
1; otherwise it returns a 0.
load_lib "library-file"
runtest. If DejaGnu has been installed, it looks in a path
starting with the installed library directory. If you are running
DejaGnu directly from a source directory, without first running
`make install', this path defaults to the current directory. In
either case, it then looks in the current directory for a directory
called lib. If there are duplicate definitions, the last one
loaded takes precedence over the earlier ones.
setup_xfail "config [bugid]"
configure target name; in particular, you may not use
the shorter nicknames supported by configure (but you can use the
common shell wildcard characters to specify sets of names). The
bugid argument is optional, and used only in the logging file
output; use it as a link to a bug-tracking system such as GNATS
(see section `Overview' in Tracking Bugs With GNATS).
Once you use setup_xfail, the fail and pass
procedures produce the messages `XFAIL' and `XPASS'
respectively, allowing you to distinguish expected failures (and
unexpected success!) from other test outcomes.
Warning: you must clear the expected failure after using
setup_xfail in a test case. Any call to pass or
fail clears the expected failure implicitly; if the test has some
other outcome, e.g. an error, you can call clear_xfail to clear
the expected failure explicitly. Otherwise, the expected-failure
declaration applies to whatever test runs next, leading to surprising
results.
clear_xfail config
setup_xfail)
for a particular set of configurations. The config argument is a
list of configuration target names. It is only necessary to call
clear_xfail if a test case ends without calling either
pass or fail, after calling setup_xfail.
verbose "string" number
runtest command
line. It prints string if the value of the variable
verbose is higher than or equal to the optional number. The
default value for number is 1.
Certain other general-purpose procedures are kept as utility collections
in the `lib' subdirectory. These files are not used by all tests,
but are used (via the Tcl command source) often enough to warrant
a common definition.
`lib/remote.exp' defines these functions, for establishing and managing communications:
Procedures to establish a connection: Each of these procedures
tries to establish the connection up to three times before returning.
Warnings (if retries will continue) or errors (if the attempt is
abandoned) report on communication failures. The result for any of
these procedures is either -1, when the connection cannot be
established, or the spawn ID returned by the expect command
spawn.
telnet hostname
rlogin hostname
rsh hostname
tip port
tip.
port must be a name from the tip configuration file
`/etc/remote'. Often, this is called `hardwire', or something
like `ttya'. This file holds all the configuration data for
the serial port.
kermit port bps
kermit.
port is the device name, e.g. `/dev/ttyb'. bps is
the line speed to use (in bits per second) for the connection.
Procedures to manage a connection:
tip_download spawnid file
~put
command under tip. Most often used for single board computers
that require downloading programs in ASCII S-records. Returns
1 if an error occurs, 0 otherwise.
exit_remote_shell spawnid
download file [ spawnid ]
download reads in file (object code in
S-record format) and writes it to the device controlling this
spawnid. (From the point of view of the target, the S-record file
comes in via standard input.)
If you have more than one target active, you can use the optional argument
spawnid to specify an alternative target (the default is the most
recently established spawnid.)
`lib/utils.exp' defines these utility procedures:
getdirs dir
getdirs dir pattern
getdirs assumes `*'. You may use the common shell wildcard
characters in pattern.
find dir pattern
which binary
which utility. This procedure uses the shell
environment variable `PATH'. It returns 0 if the binary is
not in the path, or if there is no `PATH' environment variable. If
binary is in the path, it returns the full path to binary.
grep filename regexp
grep filename regexp line
grep.
Use the optional third argument `line' to start lines in the result
with the line number in filename. (This argument is simply an
option flag; type it just as shown---`line'.)
prune list pattern
Most tools also have a library file associated with the tool, containing procedures found useful by test developers. These are especially helpful when the tests span several subdirectories.
Each combination of target and tool requires some target-dependent procedures. The names of these procedures have a common form: the tool name, followed by an underbar `_', and finally a suffix describing the procedure's purpose. For example, a procedure to extract the version from GDB is called `gdb_version'. See section Initialization module, for a discussion of how DejaGnu arranges to find the right procedures for each target.
runtest itself calls only two of these procedures,
tool_exit and tool_version; these procedures use
no arguments.
The other two procedures, tool_start and
tool_load, are only called by the test suites themselves
(or by testsuite-specific initialization code); they may take arguments
or not, depending on the conventions used within each test suite.
tool_start
tool_start starts and initializes the tool, leaving the
tool up and running for the test cases; an example is gdb_start,
the start function for GDB. For a batch oriented tool,
tool_start is optional; the recommended convention is to
let tool_start run the tool, leaving the output in a
variable called comp_output. Test scripts can then analyze
`$comp_output' to determine the test results. An example of this
second kind of start function is gcc_start, the start function
for GCC.
runtest itself does not call tool_start. The
initialization module `tool_init.exp' must call
tool_start for interactive tools; for batch-oriented tools,
each individual test script calls tool_start (or makes
other arrangements to run the tool).
tool_load
gdb_load loads
a new executable file into the debugger. For batch oriented tools,
tool_load may do nothing--though, for example, the
GCC support uses gcc_load to load and run a binary on the
target environment. Conventionally, tool_load leaves the
output of any program it runs in a variable called `exec_output'.
Writing tool_load can be the most complex part of extending
DejaGnu to a new tool or a new target, if it requires much communication
coding or file downloading.
Test scripts call tool_load.
tool_exit
runtest exits. For interactive
tools, this usually ends the interactive session. You can also use
tool_exit to remove any temporary files left over from the
tests.
runtest calls tool_exit.
tool_version
runtest calls tool_version.
The usual convention for return codes from any of these procedures
(although it is not required by runtest) is to return 0 if
the procedure succeeded, 1 if it failed, and -1 if there
was a communication error.
The DejaGnu distribution includes support for the following remote
targets. You can set the target name and the connect mode in the
`site.exp' file (using the Tcl variables `targetname' and
`connectmode', respectively), or on the runtest command line
(using `--name' and `--connect').
configure also recognizes the abbreviation `udi29k'.) Then,
to run tests, use the runtest target name to specify whether you
want to use a simulator, or a particular hardware board. The particular
string to use with `--name' will depend on your UDI setup file,
`udi_soc' (if `udi_soc' is not in your working directory, the
environment variable `UDICONF' should contain a path to this file).
For example, if your UDI setup file includes these lines:
iss AF_UNIX * isstip -r /home/gnu/29k/src/osboot/sim/osboot mon AF_UNIX * montip -t serial -baud 9600 -com /dev/ttyb
mondfe is the only shell DejaGnu supports for UDI targets.
mondfe is an AMD specific monitor program freely available
from AMD.
Warning: This target requires GDB version 4.7.2 (or
greater). Earlier versions of GDB do not fully support the
load command on this target, so DejaGnu has no way to load
executable files from the debugger.
.text, .bss, and .data.
With this configuration, the default for `--connect' is `tip'.
`tip' is the only communications protocol supported for connecting
to `m68k-abug-*' targets. `tip' uses an ASCII downloader
(the ~put command) to load S-records into the target board. The
`--name' string must be a machine name that tip
understands (for example, on some tip implementations it must be
an entry from the initialization file for tip; this file is
sometimes called `/etc/remote').
See your system documentation for information on how to create new
entries in `/etc/remote'. (Some UNIX systems are distributed
with at least one default entry with a name resembling `hardwire';
if your system has one, you can edit it, or make a modified copy with a
new name.) When you have a working `/etc/remote' entry
abugtarget, you should be able to type `tip
abugtarget', and get the prompt `135ABUG>' from the board.
Use the same abugtarget string with `runtest --name'.
BUG boot monitor. Only the monitor commands and the addresses are
different.
The runtest program used to invoke DejaGnu is a short shell
script generated by make during the configuration process. Its
main task is to read the main test framework driver, `runtest.exp'.
`runtest.exp', in turn, reads expect code from certain other
files, in this order:
runtest defaults, for details.
runtest
itself rather than for general-purpose use in both runtest and
test suites.
expect in PostScript form as the file
`expect/tcl-debug.ps'.)
tool_init.exp. See section Initialization module, for more discussion of init files.
runtest always writes two kinds of output files: summary logs and
detailed logs. The contents of both of these are determined by your
tests.
For troubleshooting, a third kind of output file is useful: use
`--debug' to request an output file showing details of what
expect is doing internally.
runtest always produces a summary output file
`tool.sum'. This summary shows the names of all test files
run; for each test file, one line of output from each pass
command (showing status `PASS' or `XPASS') or fail
command (status `FAIL' or `XFAIL'); trailing summary
statistics that count passing and failing tests (expected and
unexpected); and the full pathname and version number of the tool
tested. (All possible outcomes, and all errors, are always reflected in
the summary output file, regardless of whether or not you specify
`--all'.)
If any of your tests use the procedures unresolved,
unsupported, or untested, the summary output also
tabulates the corresponding outcomes.
For example, after `runtest --tool binutils', look for a summary
log in `binutils.sum'. Normally, runtest writes this file
in your current working directory; use the `--output' option to
select a different directory.
Here is a short sample summary log:
Test Run By rob on Mon May 25 21:40:57 PDT 1992
=== gdb tests ===
Running ./gdb.t00/echo.exp ...
PASS: Echo test
Running ./gdb.all/help.exp ...
PASS: help add-symbol-file
PASS: help aliases
PASS: help breakpoint "bre" abbreviation
FAIL: help run "r" abbreviation
Running ./gdb.t10/crossload.exp ...
PASS: m68k-elf (elf-big) explicit format; loaded
XFAIL: mips-ecoff (ecoff-bigmips) "ptype v_signed_char" signed
C types
=== gdb Summary ===
# of expected passes 5
# of expected failures 1
# of unexpected failures 1
/usr/latest/bin/gdb version 4.6.5 -q
runtest also saves a detailed log file `tool.log',
showing any output generated by tests as well as the summary output.
For example, after `runtest --tool binutils', look for a detailed
log in `binutils.log'. Normally, runtest writes this file
in your current working directory; use the `--output' option to
select a different directory.
Here is a brief example showing a detailed log for G++ tests:
Test Run By rob on Mon May 25 21:40:43 PDT 1992
=== g++ tests ===
--- Running ./g++.other/t01-1.exp ---
PASS: operate delete
--- Running ./g++.other/t01-2.exp ---
FAIL: i960 bug EOF
p0000646.C: In function `int warn_return_1 ()':
p0000646.C:109: warning: control reaches end of non-void function
p0000646.C: In function `int warn_return_arg (int)':
p0000646.C:117: warning: control reaches end of non-void function
p0000646.C: In function `int warn_return_sum (int, int)':
p0000646.C:125: warning: control reaches end of non-void function
p0000646.C: In function `struct foo warn_return_foo ()':
p0000646.C:132: warning: control reaches end of non-void function
--- Running ./g++.other/t01-4.exp ---
FAIL: abort
900403_04.C:8: zero width for bit-field `foo'
--- Running ./g++.other/t01-3.exp ---
FAIL: segment violation
900519_12.C:9: parse error before `;'
900519_12.C:12: Segmentation violation
/usr/latest/bin/gcc: Internal compiler error: program cc1plus got
fatal signal
=== g++ Summary ===
# of expected passes 1
# of expected failures 3
/usr/ps/bin/g++ version cygnus-2.0.1
expect internal actions
With the `--debug' option, you can request a log file showing the
output from expect itself, running in debugging mode. This file
(`dbg.log', in the directory where you start runtest) shows
each pattern expect considers in analyzing test output.
This file reflects each send command, showing the string sent as
input to the tool under test; and each expect command, showing
each pattern it compares with the tool output.
The log messages for expect begin with a message of the form
expect: does {tool output} (spawn_id n) match pattern
{expected pattern}?
For every unsuccessful match, expect issues a `no' after
this message; if other patterns are specified for the same
expect command, they are reflected also, but without the first
part of the message (`expect...match pattern').
When expect finds a match, the log for the successful match ends
with `yes', followed by a record of the expect variables set
to describe a successful match. Here is an excerpt from the debugging
log for a GDB test:
send: sent {break gdbme.c:34\n} to spawn id 6
expect: does {} (spawn_id 6) match pattern {Breakpoint.*at.* file
gdbme.c, line 34.*\(gdb\) $}? no
{.*\(gdb\) $}? no
expect: does {} (spawn_id 0) match pattern {<return>}? no
{\(y or n\) }? no
{buffer_full}? no
{virtual}? no
{memory}? no
{exhausted}? no
{Undefined}? no
{command}? no
break gdbme.c:34
Breakpoint 8 at 0x23d8: file gdbme.c, line 34.
(gdb) expect: does {break gdbme.c:34\r\nBreakpoint 8 at 0x23d8:
file gdbme.c, line 34.\r\n(gdb) } (spawn_id 6) match pattern
{Breakpoint.*at.* file gdbme.c, line 34.*\(gdb\) $}? yes
expect: set expect_out(0,start) {18}
expect: set expect_out(0,end) {71}
expect: set expect_out(0,string) {Breakpoint 8 at 0x23d8: file
gdbme.c, line 34.\r\n(gdb) }
expect: set expect_out(spawn_id) {6}
expect: set expect_out(buffer) {break gdbme.c:34\r\nBreakpoint 8
at 0x23d8: file gdbme.c, line 34.\r\n(gdb) }
PASS: 70 0 breakpoint line number in file
This example exhibits three properties of expect and DejaGnu that
might be surprising at first glance:
expect begins attempting to match the patterns supplied
immediately; often, the first pass is against incomplete output (or
completely before all output, as in this case).
error procedure to
make the actions for fail-safe patterns produce messages starting with
`ERROR' on the runtest standard output, and in the detailed
log file.