From 5521658e3077f0ec7c6bfbeb5a7b9e12a71a170e Mon Sep 17 00:00:00 2001 From: Tom Lane Date: Sun, 26 Mar 2000 07:01:19 +0000 Subject: [PATCH] Add description of new parallel regression testing method; update procedure to cover running either test script; other minor improvements. --- doc/src/sgml/regress.sgml | 190 ++++++++++++++++++++++++---------------------- 1 file changed, 100 insertions(+), 90 deletions(-) diff --git a/doc/src/sgml/regress.sgml b/doc/src/sgml/regress.sgml index ed211bf823..bf7162bb38 100644 --- a/doc/src/sgml/regress.sgml +++ b/doc/src/sgml/regress.sgml @@ -1,5 +1,5 @@ -Regression Test +Regression Test @@ -14,20 +14,39 @@ Regression test instructions and analysis. + There are two different ways in which the regression tests can be run: + the "sequential" method and the "parallel" method. The sequential method + runs each test script in turn, whereas the parallel method starts up + multiple server processes to run groups of tests in parallel. Parallel + testing gives confidence that interprocess communication and locking + are working correctly. Another key difference is that the sequential + test procedure uses an already-installed postmaster, whereas the + parallel test procedure tests a system that has been built but not yet + installed. (The parallel test script actually does an installation into + a temporary directory and fires up a private postmaster therein.) + + + + Some properly installed and fully functional PostgreSQL installations + can "fail" some of these regression tests due to artifacts of floating point + representation and time zone support. The tests are currently evaluated + using a simple diff comparison against the + outputs generated on a reference system, so the results are sensitive to + small system differences. + When a test is reported as "failed", always examine the differences + between expected and actual results; you may well find that the differences + are not significant. + + + The regression tests were originally developed by Jolly Chen and Andrew Yu, and were extensively revised/repackaged by Marc Fournier and Thomas Lockhart. From PostgreSQL v6.1 onward the regression tests are current for every official release. - - Some properly installed and fully functional PostgreSQL installations - can fail some of these regression tests due to artifacts of floating point - representation and time zone support. The current tests are evaluated - using a simple "diff" algorithm, and are sensitive to small system - differences. For apparently failed tests, examining the differences - may reveal that the differences are not significant. - + +Regression Environment The regression testing notes below assume the following (except where noted): @@ -60,22 +79,11 @@ The runtime path is /usr/local/pgsql (other paths are possible). - -Regression Environment - - - To prepare for regression testing, do make all in the regression test - directory. This compiles a C program with PostgreSQL extension functions - into a shared library. Localized SQL scripts and output-comparison - files are also created for the tests that need them. The localization - replaces macros in the source files with absolute pathnames and user names. - - Normally, the regression tests should be run as the postgres user since the 'src/test/regress' directory and sub-directories are owned by the postgres user. If you run the regression test as another user the - 'src/test/regress' directory tree must be writeable to that user. + 'src/test/regress' directory tree must be writeable by that user. @@ -131,6 +139,8 @@ The runtime path is /usr/local/pgsql (other paths are possible). results/ .. .out files that contain what the results *actually* look like. Also used as temporary storage for table copy testing. + + tmp_check/ temporary installation created by parallel testing script. @@ -144,39 +154,48 @@ The runtime path is /usr/local/pgsql (other paths are possible). like ps and tar vary wildly on what options you should use on each platform. Use common sense before typing in these commands. - - - For a fresh install or upgrading from previous releases of - Postgres: - - <ProductName>Postgres</ProductName> Regression Configuration - + <ProductName>Postgres</ProductName> Regression Test + - The file /usr/src/pgsql/src/test/regress/README has detailed - instructions for running and interpreting the regression tests. - A short version follows here: + Prepare the files needed for the regression test with: + + cd /usr/src/pgsql/src/test/regress + gmake clean + gmake all + + You can skip "gmake clean" if this is the first time you + are running the tests. + + + This step compiles a C + program with PostgreSQL extension functions into a shared library. + Localized SQL scripts and output-comparison files are also created + for the tests that need them. The localization replaces macros in + the source files with absolute pathnames and user names. - + + + - If the postmaster is not already running, start the postmaster on an - available window by typing + If you intend to use the "sequential" test procedure, which tests + an already-installed postmaster, be sure that the postmaster + is running. If it isn't already running, + start the postmaster in an available window by typing postmaster - or start the postmaster daemon running in the background by typing cd nohup postmaster > regress.log 2>&1 & - - - - Run postmaster from your Postgres super user account (typically - account postgres). + The latter is probably preferable, since the regression test log + will be quite lengthy (60K or so, in + Postgres 7.0) and you might want to + review it for clues if things go wrong. @@ -186,56 +205,39 @@ The runtime path is /usr/local/pgsql (other paths are possible). - - - If you have previously invoked the regression test, clean up the - working directory with: - - - cd /usr/src/pgsql/src/test/regress - gmake clean - - - - You do not need to type "gmake clean" if this is the first time you - are running the tests. - - - - - Build the regression test. Type + Run the regression tests. For a sequential test, type cd /usr/src/pgsql/src/test/regress - gmake all + gmake runtest - - - - - - Run the regression tests. Type + For a parallel test, type cd /usr/src/pgsql/src/test/regress - gmake runtest + gmake runcheck + The sequential test just runs the test scripts using your + already-running postmaster. + The parallel test will perform a complete installation of + Postgres into a temporary directory, + start a private postmaster therein, and then run the test scripts. + Finally it will kill the private postmaster (but the temporary + directory isn't removed automatically). - You should get on the screen (and also written to file ./regress.out) a series of statements stating which tests passed and which tests failed. Please note that it can be normal for some of the tests to - "fail". For the failed tests, use diff to compare the files in - directories ./results and ./expected. If float8 failed, type - something like: - - cd /usr/src/pgsql/src/test/regress - diff -w expected/float8.out results - + "fail" due to platform-specific variations. See the next section + for details on determining whether a "failure" is significant. + + + Some of the tests, notably "numeric", can take a while, especially + on slower platforms. Have patience. @@ -243,11 +245,14 @@ The runtime path is /usr/local/pgsql (other paths are possible). After running the tests and examining the results, type - dropdb regression cd /usr/src/pgsql/src/test/regress gmake clean to recover the temporary disk space used by the tests. + If you ran a sequential test, also type + + dropdb regression + @@ -257,10 +262,14 @@ The runtime path is /usr/local/pgsql (other paths are possible). Regression Analysis - The results are in files in the ./results directory. These results - can be compared with results in the ./expected directory using 'diff'. - (The test script does this for you, and leaves the differences - in ./regression.diffs.) + The actual outputs of the regression tests are in files in the + ./results directory. The test script + uses diff to compare each output file + against the reference outputs stored in the + ./expected directory. Any differences are + saved for your inspection in + ./regression.diffs. (Or you can run + diff yourself, if you prefer.) @@ -354,7 +363,7 @@ The runtime path is /usr/local/pgsql (other paths are possible). When comparing the results from different platforms, differences occur in the 2nd or 3rd place to the right of the decimal point. The SQL - statements where these problems occur are the folowing: + statements where these problems occur are the following: QUERY: SELECT * from street; @@ -368,18 +377,19 @@ The runtime path is /usr/local/pgsql (other paths are possible). Random differences - There is at least one test case in random.out which is intended to produce - random results. This causes random to fail the regression testing - once in a while. + There is at least one case in the "random" test script that is + intended to produce + random results. This causes random to fail the regression test + once in a while (perhaps once in every five to ten trials). Typing diff results/random.out expected/random.out - - should produce only - one or a few lines of differences for this reason, but other floating - point differences on dissimilar architectures might cause many more - differences. See the release notes below. + should produce only one or a few lines of differences. You need + not worry unless the random test always fails in repeated attempts. + (On the other hand, if the random test is never + reported to fail even in many trials of the regress tests, you + probably should worry.) @@ -398,10 +408,10 @@ The runtime path is /usr/local/pgsql (other paths are possible). system using the postgres5-1.02a5.tar.gz source tree. It was compared with a file created on an I386 Solaris 2.4 system and the differences were only in the floating point polygons in the 3rd digit to the right - of the decimal point. (see below) + of the decimal point. The original sample.regress.out file was from the postgres-1.01 release - constructed by Jolly Chen and is included here for reference. It may + constructed by Jolly Chen. It may have been created on a DEC ALPHA machine as the Makefile.global in the postgres-1.01 release has PORTNAME=alpha. -- 2.11.0