@item PASS
A test has succeeded. That is, it demonstrated that the assertion is true.
-@cindex XFAIL, avoiding for POSIX
-@item XFAIL
-@sc{posix} 1003.3 does not incorporate the notion of expected failures,
-so @code{PASS}, instead of @code{XPASS}, must also be returned for test
-cases which were expected to fail and did not. This means that
-@code{PASS} is in some sense more ambiguous than if @code{XPASS} is also
-used. For information on @code{XPASS} and @code{XFAIL}, see
-@ref{Invoking runtest,,Using @code{runtest}}.
-
@item FAIL
@cindex failure, POSIX definition
A test @emph{has} produced the bug it was intended to capture. That is,
message is based on the test case only. Other messages are used to
indicate a failure of the framework.
-As with @code{PASS}, @sc{posix} tests must return @code{FAIL} rather
-than @code{XFAIL} even if a failure was expected.
-
@item UNRESOLVED
@cindex ambiguity, required for POSIX
A test produced indeterminate results. Usually, this means the test
@end ftable
@noindent
-The only remaining output message left is intended to test features that
-are specified by the applicable @sc{posix} standard as conditional:
+The only remaining @sc{posix} output message left is intended to test
+features that are specified by the applicable @sc{posix} standard as
+conditional:
@ftable @code
@item UNSUPPORTED
@code{gethostname} would never work on a target board running only a
boot monitor.
@end ftable
-
+
DejaGnu uses the same output procedures to produce these messages for
all test suites, and these procedures are already known to conform to
@sc{posix} 1003.3. For a DejaGnu test suite to conform to @sc{posix}
-1003.3, you must avoid the @code{setup_xfail} procedure as described in
-the @code{PASS} section above, and you must be careful to return
+1003.3, you must avoid the @code{setup_xfail} and @code{setup_kfail}
+procedures (see below), and you must be careful to return
@code{UNRESOLVED} where appropriate, as described in the
@code{UNRESOLVED} section above.
+
+Besides the @sc{posix} messages, DejaGnu provides for variations of the
+PASS and FAIL messages that can be helpful for the tool maintainers.
+It must be noted, however, that this feature is not @sc{posix} 1003.3
+compliant, so its use should be avoided if compliance is necessary.
+
+The additional messages are:
+
+@ftable @code
+
+@item XFAIL
+A test is expected to fail in some environment(s) due to some bug
+in the environment that we hope is fixed someday (but we can't do
+nothing about as it is not a bug in the tool that we are testing).
+The procedure @code{setup_xfail} is used to indicate that a failure
+is expected.
+
+@cindex XFAIL, avoiding for POSIX
+@sc{posix} 1003.3 does not incorporate the notion of expected failures,
+so @sc{posix} tests must return @code{FAIL} rather
+than @code{XFAIL} even if a failure was expected.
+
+@item KFAIL
+A test is known to fail in some environment(s) due to a known bug
+in the tool being tested (identified by a bug id string). This
+exists so that, after a bug is identified and properly registered
+in a bug tracking database (Gnats, for instance), the count of
+failures can be kept as zero. Having zero as a baseline in all
+platforms allow the tool developers to immediately detect regressions
+caused by changes (which may affect some platforms and not others).
+The connection with a bug tracking database allows for automatic
+generation of the BUGS section of man pages or Release Notes, as
+well as a "Bugs Fixed this Release" section (by comparing to a
+previous release set of known failures).
+The procedure @code{setup_kfail} is used to indicate a failure is
+known to exist.
+
+@cindex KFAIL, avoiding for POSIX
+As with @code{XFAIL}, @sc{posix} tests must return @code{FAIL} rather
+than @code{KFAIL} even if a failure was due to a known bug.
+
+
+@item XPASS
+A test was expected to fail with @code{XFAIL} but passed instead.
+Whatever problem that used to exist in the environment was corrected
+The test may also be failing to detect the failure due to some
+environment or output changes, so this possibility must be investigated
+as well.
+
+@item KPASS
+A test was expected to fail with @code{KFAIL} but passed instead.
+Someone may have fixed the bug and failed to unmark the test.
+As for XPASS, the test may also be failing to detect the
+failure due to some environment or output changes, so this possibility
+must also be checked.
+
+@code{PASS}, instead of @code{XPASS} or @code{KPASS}, must also be
+returned for test cases which were expected to fail and did not,
+if @sc{posix} 1003.3 compliance is required.
+This means that @code{PASS} is in some sense more ambiguous than if
+@code{XPASS} and @code{KPASS} are also used.
+
+@end ftable
+
+See also @ref{Invoking runtest,,Using @code{runtest}}.
+For information on how to mark tests as expected/known to fail by using
+@code{setup_xfail} and @code{setup_kfail}, see
+@ref{framework.exp,,Core Internal Procedures}.
+
@node Future Directions
@section Future directions
This may indicate progress; inspect the test case to determine whether
you should amend it to stop expecting failure.
+@item KPASS
+@kindex KPASS
+@cindex successful test, unexpected
+@cindex unexpected success
+A pleasant kind of failure: a test was known to fail, but succeeded.
+This may indicate progress; inspect the test case to determine whether
+you should amend it to stop expecting failure.
+
@item FAIL
@kindex FAIL
@cindex failing test, unexpected
@cindex expected failure
@cindex failing test, expected
A test failed, but it was expected to fail. This result indicates no
-change in a known bug. If a test fails because the operating system
-where the test runs lacks some facility required by the test, the
-outcome is @code{UNSUPPORTED} instead.
+change in a known environment bug. If a test fails because the operating
+system where the test runs lacks some facility required by the test
+(i.e. failure is due to the lack of a feature, not the existence of a bug),
+the outcome is @code{UNSUPPORTED} instead.
+
+@item KFAIL
+@kindex KFAIL
+@cindex known failure
+@cindex failing test, known
+A test failed, but it was known to fail. This result indicates no
+change in a known bug. If a test fails because of a problem in the
+environment, not in the tool being tested, that is expected to be
+fixed one day, the outcome is @code{XFAIL} instead.
@item UNRESOLVED
@kindex UNRESOLVED
@cindex test output, displaying all
Display all test output. By default, @code{runtest} shows only the
output of tests that produce unexpected results; that is, tests with
-status @samp{FAIL} (unexpected failure), @samp{XPASS} (unexpected
-success), or @samp{ERROR} (a severe error in the test case itself).
+status @samp{FAIL} (unexpected failure), @samp{XPASS} or @samp{KPASS}
+(unexpected success), or @samp{ERROR} (a severe error in the test case itself).
Specify @samp{--all} to see output for tests with status @samp{PASS}
(success, as expected) @samp{XFAIL} (failure, as expected), or
@samp{WARNING} (minor error in the test case itself).
change how anything is actually configured unless --build is also
specified; it affects @emph{only} DejaGnu procedures that compare the
host string with particular values. The procedures @code{ishost},
-@code{istarget}, @code{isnative}, and @code{setup_xfail} are affected by
-@samp{--host}. In this usage, @code{host} refers to the machine that the
+@code{istarget}, @code{isnative}, @code{setup_xfail} and
+@code{setup_kfail} are affected by @samp{--host}.
+In this usage, @code{host} refers to the machine that the
tests are to be run on, which may not be the same as the @code{build}
machine. If @code{--build} is also specified, then @code{--host} refers
to the machine that the tests wil, be run on, not the machine DejaGnu is
@item pass "@var{string}"
@cindex test case, declaring success
Declares a test to have passed. @code{pass} writes in the
-log files a message beginning with @samp{PASS} (or @code{XPASS}, if
-failure was expected), appending the argument @var{string}.
+log files a message beginning with @samp{PASS}
+(or @code{XPASS}/@code{KPASS}, if failure was expected),
+appending the argument @var{string}.
@item fail "@var{string}"
@cindex test case, declaring failure
output; use it as a link to a bug-tracking system such as @sc{gnats}
(@pxref{Overview,, Overview, gnats.info, Tracking Bugs With GNATS}).
+See notes under setup_kfail (below).
+
+@item setup_kfail "@var{config} @r{[}@var{bugid}@r{]}"
+@c two spaces above to make it absolutely clear there's whitespace---a
+@c crude sort of italic correction!
+@cindex test case, known failure
+@cindex failure, known
+@cindex known failure
+Declares that the test is known to fail on a particular set of
+configurations. The @var{config} argument must be a list of full
+three-part @code{configure} target name; in particular, you may not use
+the shorter nicknames supported by @code{configure} (but you can use the
+common shell wildcard characters to specify sets of names). The
+@var{bugid} argument is mandatory, and used only in the logging file
+output; use it as a link to a bug-tracking system such as @sc{gnats}
+(@pxref{Overview,, Overview, gnats.info, Tracking Bugs With GNATS}).
+
@cindex @code{XFAIL}, producing
+@cindex @code{KFAIL}, producing
@cindex @code{XPASS}, producing
-Once you use @code{setup_xfail}, the @code{fail} and @code{pass}
-procedures produce the messages @samp{XFAIL} and @samp{XPASS}
-respectively, allowing you to distinguish expected failures (and
-unexpected success!) from other test outcomes.
-
-@emph{Warning:} you must clear the expected failure after using
-@code{setup_xfail} in a test case. Any call to @code{pass} or
-@code{fail} clears the expected failure implicitly; if the test has some
-other outcome, e.g. an error, you can call @code{clear_xfail} to clear
-the expected failure explicitly. Otherwise, the expected-failure
+@cindex @code{KPASS}, producing
+Once you use @code{setup_xfail} or @code{setup_kfail}, the @code{fail}
+and @code{pass} procedures produce the messages @samp{XFAIL} or @samp{KFAIL}
+and @samp{XPASS} or @samp{KPASS} respectively, allowing you to distinguish
+expected/known failures (and unexpected success!) from other test outcomes.
+
+If a test is marked as both expected to fail and known to fail for a
+certain configuration, a @samp{KFAIL} message will be generated.
+As @samp{KFAIL} messages are expected to draw more attention than
+the @samp{XFAIL} ones this will hopefuly ensure the test result is not
+overlooked.
+
+@emph{Warning:} you must clear the expected/known failure after using
+@code{setup_xfail} or @code{setup_kfail} in a test case. Any call to
+@code{pass} or @code{fail} clears the expectedknown failure implicitly;
+if the test has some other outcome, e.g. an error, you can call
+@code{clear_xfail} to clear the expected failure or @code{clear_kfail}
+to clear the known failure explicitly. Otherwise, the expected-failure
declaration applies to whatever test runs next, leading to surprising
results.
@code{clear_xfail} if a test case ends without calling either
@code{pass} or @code{fail}, after calling @code{setup_xfail}.
+@item clear_kfail @var{config}
+@cindex cancelling known failure
+@cindex known failure, cancelling
+Cancel a known failure (previously declared with @code{setup_kfail})
+for a particular set of configurations. The @var{config} argument is a
+list of configuration target names. It is only necessary to call
+@code{clear_kfail} if a test case ends without calling either
+@code{pass} or @code{fail}, after calling @code{setup_kfail}.
+
@item verbose @r{[}-log@r{]} @r{[}-n@r{]} @r{[}--@r{]} "@var{string}" @var{number}
@cindex @code{verbose} builtin function
Test cases can use this function to issue helpful messages depending on
@code{runtest} always produces a summary output file
@file{@var{tool}.sum}. This summary shows the names of all test files
run; for each test file, one line of output from each @code{pass}
-command (showing status @samp{PASS} or @samp{XPASS}) or @code{fail}
-command (status @samp{FAIL} or @samp{XFAIL}); trailing summary
-statistics that count passing and failing tests (expected and
+command (showing status @samp{PASS}, @samp{XPASS} or @samp{KPASS}) or
+@code{fail} command (status @samp{FAIL}, @samp{XFAIL} or @samp{KFAIL});
+trailing summary statistics that count passing and failing tests (expected and
unexpected); and the full pathname and version number of the tool
tested. (All possible outcomes, and all errors, are always reflected in
the summary output file, regardless of whether or not you specify
# Without this, all messages that start with a keyword are written only to the
# detail log file. All messages that go to the screen will also appear in the
# detail log. This should only be used by the framework itself using pass,
-# fail, xpass, xfail, warning, perror, note, untested, unresolved, or
-# unsupported procedures.
+# fail, xpass, xfail, kpass, kfail, warning, perror, note, untested, unresolved,
+# or unsupported procedures.
#
proc clone_output { message } {
global sum_file
regsub "^\[ \t\]*(\[^ \t\]+).*$" "$message" "\\1" firstword;
case "$firstword" in {
- {"PASS:" "XFAIL:" "UNRESOLVED:" "UNSUPPORTED:" "UNTESTED:"} {
+ {"PASS:" "XFAIL:" "KFAIL:" "UNRESOLVED:" "UNSUPPORTED:" "UNTESTED:"} {
if $all_flag {
send_user "$message\n"
return "$message"
if { $testcnt > 0 } {
set totlcnt 0;
# total all the testcases reported
- foreach x { FAIL PASS XFAIL XPASS UNTESTED UNRESOLVED UNSUPPORTED } {
+ foreach x { FAIL PASS XFAIL KFAIL XPASS KPASS UNTESTED UNRESOLVED UNSUPPORTED } {
incr totlcnt test_counts($x,$which);
}
set testcnt test_counts(total,$which);
}
}
}
- foreach x { PASS FAIL XPASS XFAIL UNRESOLVED UNTESTED UNSUPPORTED } {
+ foreach x { PASS FAIL XPASS XFAIL KPASS KFAIL UNRESOLVED UNTESTED UNSUPPORTED } {
set val $test_counts($x,$which);
if { $val > 0 } {
set mess "# of $test_counts($x,name)";
}
+#
+# Setup a flag to control whether it is a known failure
+#
+# A bug report ID _MUST_ be specified, and is the first argument.
+# It still must be a string without '-' so we can be sure someone
+# did not just forget it and we end-up using a taget triple as
+# bug id.
+#
+# Multiple target triplet patterns can be specified for targets
+# for which the test is known to fail.
+#
+#
+proc setup_kfail { args } {
+ global kfail_flag
+ global kfail_prms
+
+ set kfail_prms 0
+ set argc [ llength $args ]
+ for { set i 0 } { $i < $argc } { incr i } {
+ set sub_arg [ lindex $args $i ]
+ # is a prms number. we assume this is a string with no '-' characters
+ if [regexp "^\[^\-\]+$" $sub_arg] {
+ set kfail_prms $sub_arg
+ continue
+ }
+ if [istarget $sub_arg] {
+ set kfail_flag 1
+ continue
+ }
+ }
+
+ if {$kfail_prms == 0} {
+ perror "Attempt to set a kfail without specifying bug tracking id"
+ }
+}
+
+
# check to see if a conditional xfail is triggered
# message {targets} {include} {exclude}
#
}
#
+# Clear the kfail flag for a particular target
+#
+proc clear_kfail { args } {
+ global kfail_flag
+ global kfail_prms
+
+ set argc [ llength $args ]
+ for { set i 0 } { $i < $argc } { incr i } {
+ set sub_arg [ lindex $args $i ]
+ case $sub_arg in {
+ "*-*-*" { # is a configuration triplet
+ if [istarget $sub_arg] {
+ set kfail_flag 0
+ set kfail_prms 0
+ }
+ continue
+ }
+ }
+ }
+}
+
+#
# Record that a test has passed or failed (perhaps unexpectedly)
#
# This is an internal procedure, only used in this file.
global exit_status
global prms_id bug_id
global xfail_flag xfail_prms
+ global kfail_flag kfail_prms
global errcnt warncnt
global warning_threshold perror_threshold
global pf_prefix
set message [concat $message "\t(PRMS $xfail_prms)"]
}
}
+ KPASS {
+ set exit_status 1
+ if { $kfail_prms != 0 } {
+ set message [concat $message "\t(PRMS $kfail_prms)"]
+ }
+ }
+ KFAIL {
+ if { $kfail_prms != 0 } {
+ set message [concat $message "\t(PRMS: $kfail_prms)"]
+ }
+ }
UNTESTED {
- # The only reason we look at the xfail stuff is to pick up
+ # The only reason we look at the xfail/kfail stuff is to pick up
# `xfail_prms'.
- if { $xfail_flag && $xfail_prms != 0 } {
+ if { $kfail_flag && $kfail_prms != 0 } {
+ set message [concat $message "\t(PRMS $kfail_prms)"]
+ } elseif { $xfail_flag && $xfail_prms != 0 } {
set message [concat $message "\t(PRMS $xfail_prms)"]
} elseif $prms_id {
set message [concat $message "\t(PRMS $prms_id)"]
}
UNRESOLVED {
set exit_status 1
- # The only reason we look at the xfail stuff is to pick up
+ # The only reason we look at the xfail/kfail stuff is to pick up
# `xfail_prms'.
- if { $xfail_flag && $xfail_prms != 0 } {
+ if { $kfail_flag && $kfail_prms != 0 } {
+ set message [concat $message "\t(PRMS $kfail_prms)"]
+ } elseif { $xfail_flag && $xfail_prms != 0 } {
set message [concat $message "\t(PRMS $xfail_prms)"]
} elseif $prms_id {
set message [concat $message "\t(PRMS $prms_id)"]
}
}
UNSUPPORTED {
- # The only reason we look at the xfail stuff is to pick up
+ # The only reason we look at the xfail/kfail stuff is to pick up
# `xfail_prms'.
- if { $xfail_flag && $xfail_prms != 0 } {
+ if { $kfail_flag && $kfail_prms != 0 } {
+ set message [concat $message "\t(PRMS $kfail_prms)"]
+ } elseif { $xfail_flag && $xfail_prms != 0 } {
set message [concat $message "\t(PRMS $xfail_prms)"]
} elseif $prms_id {
set message [concat $message "\t(PRMS $prms_id)"]
set warncnt 0
set errcnt 0
set xfail_flag 0
+ set kfail_flag 0
set xfail_prms 0
+ set kfail_prms 0
}
#
# Record that a test has passed
#
proc pass { message } {
- global xfail_flag compiler_conditional_xfail_data
+ global xfail_flag kfail_flag compiler_conditional_xfail_data
# if we have a conditional xfail setup, then see if our compiler flags match
if [ info exists compiler_conditional_xfail_data ] {
unset compiler_conditional_xfail_data
}
- if $xfail_flag {
+ if $kfail_flag {
+ record_test KPASS $message
+ } elseif $xfail_flag {
record_test XPASS $message
} else {
record_test PASS $message
# Record that a test has failed
#
proc fail { message } {
- global xfail_flag compiler_conditional_xfail_data
+ global xfail_flag kfail_flag compiler_conditional_xfail_data
# if we have a conditional xfail setup, then see if our compiler flags match
if [ info exists compiler_conditional_xfail_data ] {
unset compiler_conditional_xfail_data
}
- if $xfail_flag {
+ if $kfail_flag {
+ record_test KFAIL $message
+ } elseif $xfail_flag {
record_test XFAIL $message
} else {
record_test FAIL $message
}
#
-# Record that a test has passed unexpectedly
+# Record that a test that was expected to fail has passed unexpectedly
#
proc xpass { message } {
record_test XPASS $message
}
#
-# Record that a test has failed unexpectedly
+# Record that a test that was expected to fail did indeed fail
#
proc xfail { message } {
record_test XFAIL $message
}
+
+#
+# Record that a test for a known bug has passed unexpectedly
+#
+proc kpass { bugid message } {
+ global kfail_flag kfail_prms
+ set kfail_flag 1
+ set kfail_prms $bugid
+ record_test KPASS $message
+}
+
+#
+# Record that a test has failed due to a known bug
+#
+proc kfail { bugid message } {
+ global kfail_flag kfail_prms
+ set kfail_flag 1
+ set kfail_prms $bugid
+ record_test KFAIL $message
+}
\f
#
# Set warning threshold
set test_counts(FAIL,name) "unexpected failures"
set test_counts(XFAIL,name) "expected failures"
set test_counts(XPASS,name) "unexpected successes"
+ set test_counts(KFAIL,name) "known failures"
+ set test_counts(KPASS,name) "unknown successes"
set test_counts(WARNING,name) "warnings"
set test_counts(ERROR,name) "errors"
set test_counts(UNSUPPORTED,name) "unsupported tests"
OSDN Git Service
