mlx5en: do no call if_input() while holding the rq mutex

[kib]

Summary

Instead, collect a batch of the received mbufs and push them into upper
stack with temporary unlocked rq.

PR:     281469
Reviewed by:    Ariel Ehrenberg <[e-mail]>

[jhb]

sys/dev/mlx5/mlx5_en/mlx5_en_rx.c

516

Note that you can pass a list of packets to ether_input where they are linked by m_nextpkt.

640–643

Could also use an mbufq which takes up less stack space (though this approach is probably fine).

[kib]

Use mbufq to collect the batch.

[gallatin]

I'm very afraid there will be performance implications due to new cache misses here from queing mbufs twice. On tens of thounsands of interfaces running over 8 years, we've never hit a deadlock from this lock, and I don't think fixing this is important enough to hurt performance for.

Instead of doing this, can you remove the lock entirely, change it to a sleepable lock, or reduce its scope instead? From some reading I did last week at the bug session, I gather that it exists only to guard against 2 rare conditions (doing work later because something failed to restock,the ring, and bringing the interface down).