[serialize] Check for overflow in allocate_size()

If size was > INT_MAX, then the out-of-room check was failing to perform
as intended.

Part of fixing https://oss-fuzz.com/testcase-detail/5362189182566400
1 file changed