Fix invalid buffer access in OOM times Hopefully fully fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=1856