3 Keith Seitz (keiths@cygnus.com)
8 The Insight testsuite is run in much the same way that gdb's testsuites
9 are run. The one big difference is the environment variable GDB_DISPLAY,
10 which governs what display should be used for the tests.
12 When GDB_DISPLAY is not set in the user's environment, the Insight testsuite
13 will attempt to run Xvfb, an X server with a virtual frame buffer. Using
14 Xvfb, the testsuite can run without interuppting the user.
16 When Xvfb is not available, the testsuite will mark the Insight tests
17 "untested" and print out some appropriate warning to the testsuite log
20 If GDB_DISPLAY is set in the user's environment, the testsuite will attempt
21 to use this display for the tests. If this display is a desktop display,
22 it is very likely that any interaction between the user and his desktop
23 will interfere with the tests. Some tests warp the cursor, i.e., they
24 force the mouse to move on the screen. If you don't want this to happen to
25 you, put Xvfb in your path.
27 On Cygwin systems, Xvfb is not supported. Only two choices are available in
28 this environment: run the testsuites using the desktop or do not run the
29 testsuite. To run the testsuite on Cygwin, just define the environment
30 variable GDB_DISPLAY to anything.
32 The examples below summarize the usage of the environment variable GDB_DISPLAY
33 on unix/X-Windows hosts and Cygwin hosts. In all examples, assume that DISPLAY
34 set to the local workstation's main display (:0).
36 To run the testsuite using Xvfb -- unix only (Xvfb must be in PATH):
39 To run the testsuite using a given display (either the desktop or a peviously
41 $ GDB_DISPLAY=$DISPLAY make check
43 To run the testsuite on Cygwin:
44 $ GDB_DISPLAY=foo make check
47 TESTSUITE INFRASTRUCTURE
49 The rest of this document deals with writing tests for Insight. This reading
50 is only noteworthy for developers and contributors.
52 The Insight testsuite consists of two large portions of code: code which is
53 run in dejagnu and code which runs in Insight's built-in Tcl interpreter. Files
54 containing dejagnu code (those files ending in ".exp" in the testsuite directory)
55 are "glue code" between gdb's dejagnu testsuite and Insight's Tcl testsuite.
57 Dejagnu testsuite files are considered "drivers" for any particular set of
58 tests, since they allow dejagnu to control Insight's Tcl testsuite.
61 Dejagnu Testsuite Infrastructure
63 The dejagnu code is responsible for doing several things. Some of the more
64 important responsibilities include:
66 o Initializing the display
67 o Determining if tests should be run
69 o Compiling testcases in various languages
70 o Reporting results to gdb's testsuite
72 There are various functions defined to facilitate the writing of tests. These
73 functions currently reside in gdb's gdb.exp (src/gdb/testsuite/lib/gdb.exp) and
77 proc gdbtk_initialize_display {}
79 gdbtk_initialize_display should be the first function called from the
80 (dejagnu) test file. It initializes the DISPLAY variable on unix systems
81 and determines (for all host systems) whether or not the testsuite should
82 run. It returns 1 if the test should run. If tests should not run, it
83 marks the test as "untested" and leaves a suitable message about why
84 the test should not run. If gdbtk_initialize_display returns zero, a test
87 proc gdbtk_start {test}
89 This function marks the start of a test and will execute Insight for
90 testing. The TEST parameter should be the file name of the Tcl test
91 file to load into Insight's Tcl interpreter. It returns a list of
92 test results suitable for passing to gdbtk_done or gdbtk_analyze_results.
93 See gdbtk_analyze_results for more information on the format of results.
95 gdbtk_start is responsible for communicating target startup information
96 to Insight, so that Insight's testsuite may be run on any target supported
97 by gdb. It does this by setting several environment variables just before
98 it starts Insight. These environment variables are:
101 The object directory of the dejagnu testsuite (i.e.,
102 objdir/gdb/testsuite).
104 The dejagnu source directory in which the tests are located (i.e,
107 The dejagnu testsuite subdirectory for the test (i.e., gdb.gdbtk)
109 The location of the testsuite definitions file (i.e.,
110 src/gdb/testsuite/gdb.gdbtk/defs)
112 Note that DEFS is converted to absolute tcl-style paths. On unix,
113 this means that DEFS would point to, for example,
114 /home/keiths/insight/src/gdb/testsuite/gdb.gdbtk/defs. On Cygwin it
115 would point to C:/cygwin/home/keiths/insight/src/gdb/testsuite/gdb.gdbtk/defs.
116 This is because of a descrepency between Cygwin's posix paths and Tcl's
117 not-quite-posix paths.
119 proc gdbtk_analyze_results {results}
120 This function translates the list of results in RESULTS into dejagnu
121 results, reporting the number of failures, errors, passes, and expected
122 failures and passes. It currently does not deal with "untested" and other
123 test statuses from dejagnu since Insight's tcl testsuite does not
126 The format of the results expected by gdbtk_analyze_results is simple:
127 it is a list of {status name description msg}. "status" is the execution
128 status of one of the tcl tests run. This can be "PASS", "FAIL", "ERROR",
131 "name" is the name of the test, and it is reported in all testsuite
132 results in dejagnu. This speeds location of the failing test. This
133 "name" may also be given to Insight's testsuite, telling it to
134 only run this test. See "do_tests" in Tcl Testsuite Infrastructure
135 for more information.
137 "description" is a textual description of the test given by "name".
139 "msg" is currently not used.
141 proc gdbtk_done {{results {}}}
142 gdbtk_done takes any RESULTS and passes it gdbtk_analyze_results for
143 outputting to the dejagnu part of the testsuite. It may be called
144 without an argument, in which case it will only terminate Xvfb if it
145 was started. Tests must call gdbtk_done _once_ at the end of their
149 proc _gdbtk_export_target_info
150 This function exports information about the target into the environment
151 so that Insight's testsuite may run programs on any supported gdb
152 target. This target information is passed into the Tcl testsuite
153 via an environment variable, TARGET_INFO, which is really a Tcl array,
154 i.e., the array is constructed in tcl and exported into the environment
155 with Tcl's "array get" command).
157 There are four elements to the array:
158 TARGET_INFO(init) - (optional) A list of commands to execute in gdb
159 to initialize the session. This usually includes
160 setting baud rates and remote protocol options.
161 TARGET_INFO(target) - (required) The complete "target" command to connect
163 TARGET_INFO(load) - (optional) The complete "load" command to load an
164 executable into a target.
165 TARGET_INFO(run) - (required) The complete "run" command, sans arguments,
166 to start a process on the target. For remote targets,
167 this is usually just "continue".
169 proc _gdbtk_xvfb_init
170 This procedure actually determines whether the an Insight test should
171 run and what DISPLAY it should use for that test. It is called by
172 gdbtk_initialize_display to do most of the dirty work.
174 It has a simple heuristic: If GDB_DISPLAY is not set and Xvfb is available
175 (on unix), it starts Xvfb, using the current process id as the screen number.
176 If Xvfb is not available and GDB_DISPLAY was not set, it skips the tests.
178 proc _gdbtk_xvfb_exit
179 _gdbtk_xvfb_exit will kill any previously started Xvfb.
182 global _xvfb_spawn_id
183 This variable holds the spawn_id of any Xvfb process started
184 by the testsuite (or it is left undefined).
186 global _using_windows
187 A global variable which indicates whether the testsuite is running
188 on cygwin. Unfortunately, as of writing, the global variable
189 tcl_platform(platform) is "unix" on Cygwin, so it is not possible
190 to rely on this for platform-dependent operations.
192 Instead, this variable is set by gdbtk_initialize_display. The test
193 it uses to determine if Cygwin is being used: it looks for the program
194 cygpath in the PATH. Therefore, cygpath is REQUIRED to run the testsuite
195 on Cygwin. (gdbtk_start also uses cygpath to determine Windows
196 pathnames for Cygwin.)
199 Testsuite Driver Basics
201 Given the above interfaces for connecting Insight's Tcl testsuite and
202 gdb's dejagnu testsuite, the basic testsuite driver file should look
203 (minimally) like this:
206 1 load_lib "insight-support.exp"
207 2 if {[gdbtk_initialize_display]} {
208 3 # We found a display to use
209 4 gdb_exit; # Make sure any previous gdb is gone
210 5 set results [gdbtk_start mytest.test]
213 8 gdbtk_done [split $results \n]
216 Line 1 loads the insight testsuite support library which contains definitions
217 for all the procedures used in dejagnu to initialize and run the Insight testsuite.
218 Line 2 calls gdbtk_initialize_display to ascertain whether there is a display
219 to use for the test. This could use an existing display (if GDB_DISPLAY is
220 set in the environment) or gdbk_initialize_display could startup an Xvfb
221 for use by the testsuite.
223 Line 4 forces any previously executing gdb to terminate.
225 Line 5 signals the start of the test run. "mytest.test" is the name of the
226 Tcl test file to execute in Insight's built-in Tcl interpreter. The output
227 of gdbtk_start_test is all of the results of the Tcl test from Insight, which
228 is subsequently passed to gdbk_analyze_results via gdbtk_done on Line 8.
230 Note how nothing happens if gdbtk_initialize_display returns false.
233 Tcl Testsuite Infrastructure
235 The heart of Insight's testsuite is its Tcl testsuite. It is these tests
236 which run directly in Insight's Tcl interpreter and allow test writers
237 access to Insight's internals. Tcl testsuite files have the filename suffix
238 ".test" to distinguish them from their driver files, which end in ".exp".
240 The design of the Insight Tcl testsuite parallels Tcl's testsuite. It has
241 many powerful features, including the ability to run ANY test in a given
242 Tcl test file. See the description of utility routines below for more
243 information about running any set of tests from a file.
245 The bulk of the code implementing the Tcl testsuite infrastructure in
246 Insight is contained in the testsuite definitions file, "defs", located
247 in src/gdb/testsuite/gdb.gdbtk. This file contains routines necessary
248 to write tests for Insight.
251 proc gdbtk_read_defs {}
252 This function, located in Insight's core Tcl library, attempts to load
253 the testsuite definitions file. If it fails, it will either pop up
254 a dialog box with the error (if running interactively) or it will
255 print the error to stderr and exit (if running non-interactively).
257 If successful, it will return true.
259 proc gdbtk_test_file {filename}
260 This function is used to load the file given by FILENAME into
261 Insight. It will automatically append ".exe" to any FILENAME
262 on Cygwin-hosted systems.
264 If successful, it will load the given file into Insight and
265 return the output of gdb's file command. It will call "error"
266 if it was succesful, therefore all calls to "gdbtk_test_file"
267 should be called using "catch".
269 Test authors should not use "gdb_cmd {file FILENAME}" to load
270 files into gdb unless they are testing interface code between
273 proc gdbtk_test_run {{prog_args {}}}
274 gdbtk_test_run runs the previously loaded executable, passing
275 the given arguments to the inferior. Like Insight's Run button,
276 it will do whatever is necessary to get the executable running,
277 including any target initialization (setting baud rate and remote
278 protocol options), downloading the executable to the target, and
279 finally starting execution.
281 Test authors should NEVER use "gdb_cmd {run ARGUMENTS}" to run an
282 executable. Doing so will insure that your tests will only run on
285 It returns true if successful or false otherwise. It will report
286 the error in a dialog box (if running interactively) or it will
287 print the error to stderr.
289 proc gdbtk_test {name description script answer}
290 This is Tcl testsuite equivalent of "expect". "name" is a canonical
291 name of the test, usually of the form "shortname-major.minor". This is
292 the name that is used when running selected tests from a given file.
293 If "name" starts with an asterisk (*), it designates that the test
296 "description" is a short textual description of the test to help
297 humans understand what it does.
299 "script" is the actual test script to run. The result of this script
300 will be compared against "answer" to determine if the test passed
303 It calls gdbtk_print_verbose to print out the results to the terminal
304 (if running interactively) or to the log file.
306 proc gdbtk_test_done {}
307 gdbtk_test_done is called at the very end of all tcl tests. It is used
308 to exit Insight and return control back to the dejagnu driver which
311 proc gdbtk_dotests {file args}
314 proc do_test {{file {}} {verbose {}} {tests {}}}
315 This procedure is used to invoke the Insight test(s) given in FILE
316 which match the regular expression(s) in TESTS. This is invoked
317 from Insight's console window to run tests interactively.
319 VERBOSE sets the verbosity of the test run. When set to one,
320 the testsuite will report all results in human readable form.
321 When set greater than one, it will print out results as a list,
322 i.e., for passing to gdbtk_analyze_results. If zero, it will only
323 print errors and failures in human readable form.
325 Public global variables:
326 objdir - The objdir from dejagnu. See gdbtk_start for more information.
327 srcdir - The srcdir from dejagnu. See gdbtk_start for more information.
328 test_ran - Indicates whether the last test ran or not. See example below.
331 proc gdbtk_test_error {desc}
332 An internal function used to report a framework error in the testsuite.
333 "desc" is a description of the error. It calls gdbtk_test_done.
335 proc gdbtk_print_verbose {status name description script code answer}
336 A helper procedure to gdbtk_test which prints out results to the terminal
337 or the logfile (or both or none).
339 Private global variables:
340 _test - An array used by the testsuite internals.
345 Armed with the basic interface described above, it is possible to test Insight's
346 GUI. Please do not write tests which attempt to imitate a user (moving the
347 mouse and clicking buttons), unless there is no other way to test the functionality.
349 The basic test file (with one test) looks like this (nonsensical one):
351 1 if {![gdbtk_read_defs]} {
355 5 global objdir test_ran
356 6 set program [file join $objdir mytest]
357 7 if {[catch {gdbtk_test_file $program} t]} {
358 8 gdbtk_test_error "loading \"$program\": $t"
360 10 if {![gdbtk_test_run]} { exit 1 }
365 15 # Test: mytest-1.1
366 16 # Desc: check if a source window was created
367 17 gdbtk_test mytest-1.1 {source window created} {
368 18 set window [ManagedWin::find SrcWin]
380 Line 1 calls the Inisght function gdbtk_read_defs to read in the testsuite
383 Line 6 then specifies the name of a file (mytest) in the object directory
384 which is loaded into gdb on Line 7. If loading the file into Insight
385 failed, gdbtk_test_error is called to publish the error (and terminate the
386 test run for this file).
388 Line 10 runs the executable on the target, and exits if it was unable
391 Line 13 simply sets a global variable foo to illustrate the purpose
392 of the global "test_ran". Before the test "mytest-1.1" runs, foo is set to
393 one. In order to support running specific tests, the state of the debugger
394 cannot be altered INSIDE gdbtk_test scripts, since the contents of the
395 script may not be run if the user requested only a specific test to run.
397 Line 20 in the middle of the test modifies the global foo. If subsequent
398 test relied on foo being one, we would have a state violation, since
399 mytest-1.1 may have (or may have not) run.
401 Therefore, we can check if a test ran and reset foo by checking the
402 global "test_ran". If set, we know that the previous test (mytest-1.1)
403 was run, and that foo is now thirteen. We reset the result back to one.
405 (Aside: Some tests do not follow this rule explicitly: they can assume
406 that all tests run sequentially. In these cases, running a specific
407 test in the file will probably fail, since the debugger is not brought
408 to a known state along the way.)
410 Lines 17-21 contain the actual test. The test's name is "mytest-1.1". It
411 is this name that may be referred to when asking the testsuite to run
412 a specific test. The description of this test is "source window created",
413 indicating that mytest-1.1's purpose is to check whether a source window
416 If gdbtk_test determines that this test is to run, it will execute the
417 next part, lines 18-20, and compare the output of that script (llength
418 $window) with "1". If the result is not "1", a failure is recorded. If
419 it is "1", a pass is recorded.
421 Finally, the test file is done and exits on line 28.