<Chapter Id="regress">
-<Title>Regression Test</Title>
+<Title id="regress-title">Regression Test</Title>
<Abstract>
<Para>
</Para>
<Para>
+ There are two different ways in which the regression tests can be run:
+ the "sequential" method and the "parallel" method. The sequential method
+ runs each test script in turn, whereas the parallel method starts up
+ multiple server processes to run groups of tests in parallel. Parallel
+ testing gives confidence that interprocess communication and locking
+ are working correctly. Another key difference is that the sequential
+ test procedure uses an already-installed postmaster, whereas the
+ parallel test procedure tests a system that has been built but not yet
+ installed. (The parallel test script actually does an installation into
+ a temporary directory and fires up a private postmaster therein.)
+</Para>
+
+<Para>
+ Some properly installed and fully functional PostgreSQL installations
+ can "fail" some of these regression tests due to artifacts of floating point
+ representation and time zone support. The tests are currently evaluated
+ using a simple <application>diff</application> comparison against the
+ outputs generated on a reference system, so the results are sensitive to
+ small system differences.
+ When a test is reported as "failed", always examine the differences
+ between expected and actual results; you may well find that the differences
+ are not significant.
+</Para>
+
+<Para>
The regression tests were originally developed by Jolly Chen and Andrew Yu,
and were extensively revised/repackaged by Marc Fournier and Thomas Lockhart.
From <ProductName>PostgreSQL</ProductName> v6.1 onward
the regression tests are current for every official release.
</Para>
-<Para>
- Some properly installed and fully functional PostgreSQL installations
- can fail some of these regression tests due to artifacts of floating point
- representation and time zone support. The current tests are evaluated
- using a simple "diff" algorithm, and are sensitive to small system
- differences. For apparently failed tests, examining the differences
- may reveal that the differences are not significant.
-</Para>
+<Sect1>
+<Title>Regression Environment</Title>
<Para>
The regression testing notes below assume the following (except where noted):
</ItemizedList>
</Para>
-<Sect1>
-<Title>Regression Environment</Title>
-
-<Para>
- To prepare for regression testing, do <Command>make all</Command> in the regression test
- directory. This compiles a <Acronym>C</Acronym> program with PostgreSQL extension functions
- into a shared library. Localized SQL scripts and output-comparison
- files are also created for the tests that need them. The localization
- replaces macros in the source files with absolute pathnames and user names.
-</Para>
-
<Para>
Normally, the regression tests should be run as the postgres user since
the 'src/test/regress' directory and sub-directories are owned by the
postgres user. If you run the regression test as another user the
- 'src/test/regress' directory tree must be writeable to that user.
+ 'src/test/regress' directory tree must be writeable by that user.
</Para>
<Para>
results/ .. .out files that contain what the results *actually* look
like. Also used as temporary storage for table copy testing.
+
+ tmp_check/ temporary installation created by parallel testing script.
</ProgramListing>
</Para>
</Sect1>
like <FileName>ps</FileName> and <FileName>tar</FileName> vary wildly on what options you should use on each
platform. <Emphasis>Use common sense</Emphasis> before typing in these commands.
</Para>
-
- <Para>
- For a fresh install or upgrading from previous releases of
- <ProductName>Postgres</ProductName>:
- </Para>
<Procedure>
- <Title><ProductName>Postgres</ProductName> Regression Configuration</Title>
-
+ <Title><ProductName>Postgres</ProductName> Regression Test</Title>
+
<Step Performance="required">
<Para>
- The file /usr/src/pgsql/src/test/regress/README has detailed
- instructions for running and interpreting the regression tests.
- A short version follows here:
+ Prepare the files needed for the regression test with:
+ <ProgramListing>
+ cd /usr/src/pgsql/src/test/regress
+ gmake clean
+ gmake all
+ </ProgramListing>
+ You can skip "gmake clean" if this is the first time you
+ are running the tests.
+ </para>
+ <Para>
+ This step compiles a <Acronym>C</Acronym>
+ program with PostgreSQL extension functions into a shared library.
+ Localized SQL scripts and output-comparison files are also created
+ for the tests that need them. The localization replaces macros in
+ the source files with absolute pathnames and user names.
</Para>
-
+ </step>
+
+ <Step Performance="optional">
<Para>
- If the postmaster is not already running, start the postmaster on an
- available window by typing
+ If you intend to use the "sequential" test procedure, which tests
+ an already-installed postmaster, be sure that the postmaster
+ is running. If it isn't already running,
+ start the postmaster in an available window by typing
<ProgramListing>
postmaster
</ProgramListing>
-
or start the postmaster daemon running in the background by typing
<ProgramListing>
cd
nohup postmaster > regress.log 2>&1 &
</ProgramListing>
- </Para>
-
- <Para>
- Run postmaster from your <ProductName>Postgres</ProductName> super user account (typically
- account postgres).
+ The latter is probably preferable, since the regression test log
+ will be quite lengthy (60K or so, in
+ <ProductName>Postgres</ProductName> 7.0) and you might want to
+ review it for clues if things go wrong.
<Note>
<Para>
</Para>
</Step>
- <Step Performance="optional">
- <Para>
- If you have previously invoked the regression test, clean up the
- working directory with:
-
- <ProgramListing>
- cd /usr/src/pgsql/src/test/regress
- gmake clean
- </ProgramListing>
- </para>
- <Para>
- You do not need to type "gmake clean" if this is the first time you
- are running the tests.
- </Para>
- </step>
-
-
<Step Performance="required">
<Para>
- Build the regression test. Type
+ Run the regression tests. For a sequential test, type
<ProgramListing>
cd /usr/src/pgsql/src/test/regress
- gmake all
+ gmake runtest
</ProgramListing>
- </Para>
- </Step>
-
- <Step Performance="required">
- <Para>
- Run the regression tests. Type
+ For a parallel test, type
<ProgramListing>
cd /usr/src/pgsql/src/test/regress
- gmake runtest
+ gmake runcheck
</ProgramListing>
+ The sequential test just runs the test scripts using your
+ already-running postmaster.
+ The parallel test will perform a complete installation of
+ <ProductName>Postgres</ProductName> into a temporary directory,
+ start a private postmaster therein, and then run the test scripts.
+ Finally it will kill the private postmaster (but the temporary
+ directory isn't removed automatically).
</Para>
</Step>
<Step Performance="required">
<Para>
-
You should get on the screen (and also written to file ./regress.out)
a series of statements stating which tests passed and which tests
failed. Please note that it can be normal for some of the tests to
- "fail". For the failed tests, use diff to compare the files in
- directories ./results and ./expected. If float8 failed, type
- something like:
- <ProgramListing>
- cd /usr/src/pgsql/src/test/regress
- diff -w expected/float8.out results
- </ProgramListing>
+ "fail" due to platform-specific variations. See the next section
+ for details on determining whether a "failure" is significant.
+ </Para>
+ <Para>
+ Some of the tests, notably "numeric", can take a while, especially
+ on slower platforms. Have patience.
</Para>
</Step>
<Para>
After running the tests and examining the results, type
<ProgramListing>
- dropdb regression
cd /usr/src/pgsql/src/test/regress
gmake clean
</ProgramListing>
to recover the temporary disk space used by the tests.
+ If you ran a sequential test, also type
+ <ProgramListing>
+ dropdb regression
+ </ProgramListing>
</Para>
</Step>
</procedure>
<Title>Regression Analysis</Title>
<Para>
- The results are in files in the ./results directory. These results
- can be compared with results in the ./expected directory using 'diff'.
- (The test script does this for you, and leaves the differences
- in ./regression.diffs.)
+ The actual outputs of the regression tests are in files in the
+ <filename>./results</filename> directory. The test script
+ uses <application>diff</application> to compare each output file
+ against the reference outputs stored in the
+ <filename>./expected</filename> directory. Any differences are
+ saved for your inspection in
+ <filename>./regression.diffs</filename>. (Or you can run
+ <application>diff</application> yourself, if you prefer.)
</Para>
<Para>
When comparing the results from different platforms, differences occur
in the 2nd or 3rd place to the right of the decimal point. The SQL
- statements where these problems occur are the folowing:
+ statements where these problems occur are the following:
<ProgramListing>
QUERY: SELECT * from street;
<Title>Random differences</Title>
<Para>
- There is at least one test case in random.out which is intended to produce
- random results. This causes random to fail the regression testing
- once in a while.
+ There is at least one case in the "random" test script that is
+ intended to produce
+ random results. This causes random to fail the regression test
+ once in a while (perhaps once in every five to ten trials).
Typing
<ProgramListing>
diff results/random.out expected/random.out
</ProgramListing>
-
- should produce only
- one or a few lines of differences for this reason, but other floating
- point differences on dissimilar architectures might cause many more
- differences. See the release notes below.
+ should produce only one or a few lines of differences. You need
+ not worry unless the random test always fails in repeated attempts.
+ (On the other hand, if the random test is <emphasis>never</emphasis>
+ reported to fail even in many trials of the regress tests, you
+ probably <emphasis>should</emphasis> worry.)
</Para>
</Sect2>
system using the <FileName>postgres5-1.02a5.tar.gz</FileName> source tree. It was compared
with a file created on an I386 Solaris 2.4 system and the differences
were only in the floating point polygons in the 3rd digit to the right
- of the decimal point. (see below)
+ of the decimal point.
The original <FileName>sample.regress.out</FileName> file was from the postgres-1.01 release
- constructed by Jolly Chen and is included here for reference. It may
+ constructed by Jolly Chen. It may
have been created on a DEC ALPHA machine as the <FileName>Makefile.global</FileName>
in the postgres-1.01 release has PORTNAME=alpha.
</Para>
OSDN Git Service
