diff: Fix integer overflow.

[des]

Summary

The legacy Stone algorithm uses int to represent line numbers, array
indices, and array lengths. If given inputs approaching INT_MAX lines,
it would overflow and attempt to allocate ridiculously large amounts of
memory. To avoid this without penalizing non-pathological inputs,
switch a few variables to size_t and add checks while and immediately
after reading both inputs.

MFC after: 3 days
PR: 280371
Sponsored by: Klara, Inc.

[des]

too conservative by one

[allanjude]

usr.bin/diff/diffreg.c

558

Was it intentional to drop the RH_EOF case here?

[des]

usr.bin/diff/diffreg.c

558

Yes, its only purpose was to silence a compiler warning.

[yuri]

Shouldn't

if (len[0] > INT_MAX - 2)

be

if (len[0] > SIZE_MAX - 2)

?

And shouldn't all int types representing file size be made 64-bit?
They should be uint64_t.

[des]

In D46169#1052275, @yuri wrote:

Shouldn't

if (len[0] > INT_MAX - 2)

be

if (len[0] > SIZE_MAX - 2)

?

No, because we need len[0] + 2 to not exceed INT_MAX.

And shouldn't all int types representing file size be made 64-bit?
They should be uint64_t.

The variables used to store file sizes are already size_t. The algorithm operates on line numbers, not byte offsets, and it uses negative values. I'm OK with refusing to handle files with more than 2^31 lines.