tests/pngtest-all: detect failures add tests
This change is only verifiable in configure builds; cmake only executes
the basic test.
The previous version of tests/pngtest-all only returned the status code
of the final test. Apparently it could never fail. This adds checking
of all return status codes.
The change also adds a basic approach for regression testing with PNGs
that should fail a test; --strict ensures that PNGs which are valid do
not start to be reported as erroneous, this is the inverse.
At present the code (minimal traditional Bourne shell) only tests the
palette index checking code, a potentially important check if apps rely
on it.
The changes have been tested using the configure build both with a
regression which causes the libpng checking to cease to work and with a
corrected (reverted regression). The regression test verifies that the
intended check works as expected.
Signed-off-by: John Bowler <jbowler@acm.org>
diff --git a/tests/pngtest-all b/tests/pngtest-all
index 5e96451..ba621f3 100755
--- a/tests/pngtest-all
+++ b/tests/pngtest-all
@@ -1,16 +1,84 @@
#!/bin/sh
-# normal execution
+st=0 # exit status (set to 1 if a test fails)
+fail="**FAIL**"
+success=" SUCCESS"
+TEST(){
+ # Try to make the log file easier to read:
+ test_status="$success"
+ echo "=============== PNGTEST $* ===================="
+ ./pngtest "$@" || {
+ st=$?
+ test_status="$fail"
+ }
+ echo "===============$test_status $* ===================="
+}
-./pngtest --strict ${srcdir}/pngtest.png
+# The "standard" test
+TEST --strict "${srcdir}"/pngtest.png
-# various crashers
-# using --relaxed because some come from fuzzers that don't maintain CRC's
+# Various crashers
+# Use --relaxed because some come from fuzzers that don't maintain CRCs
+TEST --relaxed "${srcdir}"/contrib/testpngs/crashers/badcrc.png
+TEST --relaxed "${srcdir}"/contrib/testpngs/crashers/badadler.png
+TEST --xfail "${srcdir}"/contrib/testpngs/crashers/bad_iCCP.png
+TEST --xfail "${srcdir}"/contrib/testpngs/crashers/empty_ancillary_chunks.png
+for file in "${srcdir}"/contrib/testpngs/crashers/huge_*_chunk.png
+do
+ TEST --xfail "$file"
+done
+for file in "${srcdir}"/contrib/testpngs/crashers/huge_*safe_to_copy.png
+do
+ TEST --xfail "$file"
+done
+TEST --xfail "${srcdir}"/contrib/testpngs/crashers/huge_IDAT.png
-./pngtest --relaxed ${srcdir}/contrib/testpngs/crashers/badcrc.png
-./pngtest --relaxed ${srcdir}/contrib/testpngs/crashers/badadler.png
-./pngtest --xfail ${srcdir}/contrib/testpngs/crashers/bad_iCCP.png
-./pngtest --xfail ${srcdir}/contrib/testpngs/crashers/empty_ancillary_chunks.png
-./pngtest --xfail ${srcdir}/contrib/testpngs/crashers/huge_*_chunk.png \
- ${srcdir}/contrib/testpngs/crashers/huge_*safe_to_copy.png
-./pngtest --xfail ${srcdir}/contrib/testpngs/crashers/huge_IDAT.png
+# Regression tests for required warnings (or errors):
+check_stdout(){
+ # $1: the test file (a bad PNG which must produce a warning, etc)
+ # $2: a string which must occur at the end of line on stdout for success
+ # result: an error message on descriptor 3 if the string is NOT found
+ found=
+ while read line
+ do
+ case "$line" in
+ *"$2") found=1;;
+ esac
+ echo "$line" # preserve the original output verbatim
+ done
+ # output the missing warning on descriptor 3:
+ test -z "$found" && echo "$1: $2" >&3
+}
+# NOTE: traditionally the Bourne shell executed the last element in a pipe
+# sequence in the original shell so it could set variables in the original
+# shell however this is not reliable and doesn't work in bash.
+#
+# It *is* reliable to use the actual exit status of the last command in
+# the pipeline.
+exec 4>&1 # original stdout - the log file
+{
+ exec 3>&1 # stdout is the pipe at this point
+ fail=" FAIL(EXPECTED)" # runtime scope
+ success=" SUCCESS(UNEXPECTED)" # there should be a write error
+ for file in "${srcdir}"/contrib/testpngs/badpal/*.png
+ do
+ # The exit code is ignored here, the test is that the particular errors
+ # (warnings) are produced. The original output still ends up in the log
+ # file.
+ TEST "$file" |
+ check_stdout "$file" 'IDAT: Read palette index exceeding num_palette' |
+ check_stdout "$file" 'Wrote palette index exceeding num_palette' >&4
+ done
+ exec 3>&-
+} | {
+ # This may not be a sub-shell, if it is 'st' is undefined and the exit
+ # just ends up as 'exit'.
+ while read error
+ do
+ echo "MISSING REPORT: $error"
+ st=1
+ done
+ exit $st
+} || st=$?
+
+exit $st
