Fix a heap buffer overflow in `png_write_image_8bit`

The condition guarding the pre-transform path incorrectly allowed 8-bit
input data to enter `png_write_image_8bit` which expects 16-bit input.
This caused out-of-bounds reads when processing 8-bit grayscale+alpha
images (GitHub #688), or 8-bit RGB or RGB+alpha images (GitHub #746),
with the `convert_to_8bit` flag set (an invalid combination that should
bypass the pre-transform path).

The second part of the condition, i.e.

    colormap == 0 && convert_to_8bit != 0

failed to verify that input was 16-bit, i.e.

    linear != 0

contradicting the comment "This only applies when the input is 16-bit".

The fix consists in restructuring the condition to ensure both the
`alpha` path and the `convert_to_8bit` path require linear (16-bit)
input. The corrected condition, i.e.

    linear != 0 && (alpha != 0 || display->convert_to_8bit != 0)

matches the expectation of the `png_write_image_8bit` function and
prevents treating 8-bit buffers as 16-bit data.

Reported-by: Samsung-PENTEST <Samsung-PENTEST@users.noreply.github.com>
Reported-by: weijinjinnihao <weijinjinnihao@users.noreply.github.com>
Analyzed-by: degrigis <degrigis@users.noreply.github.com>
Reviewed-by: John Bowler <jbowler@acm.org>
1 file changed