tcp rack: simplify computation of rsm start and end

While there, also fix the setting of the SYN related flag.

Reviewed by: rrs
Sponsored by: Netflix, Inc.
Differential Revision: https://reviews.freebsd.org/D35862

(cherry picked from commit 1abc27dd52e5fd151472439004f20dc798887ae1)