mlx5: Zero DMA memory in mlx5_fwp_alloc()

[markj]

Summary

mlx5_alloc_cmd_msg() allocates a page, zeroes 1024 (MLX5_CMD_MBOX_SIZE)
bytes, and then calls mlx5_fwp_flush() to sync the memory. This
generates a report from KMSAN since the page is not fully initialized.
Simply zero DMA memory at allocation time.

Reported by: KMSAN
Sponsored by: Klara, Inc.
Sponsored by: Juniper Networks, Inc.

[kib]

mlx5_fwp_alloc() is used for allocation of all kind of UMA memory for the card. Zeroing it is a waste of time, and under the load card might request allocation of large enough sets. Also this dirties the caches that needs to be snooped and flushed when the card started to use the memory.

If KMSAN is so picky, why not zero only the cmd msg pages?

[markj]

In D43133#983078, @kib wrote:

mlx5_fwp_alloc() is used for allocation of all kind of UMA memory for the card. Zeroing it is a waste of time, and under the load card might request allocation of large enough sets. Also this dirties the caches that needs to be snooped and flushed when the card started to use the memory.

My reading of the driver suggested that these allocations are done only at driver initialization and queue pair initialization time. It is used to back caches used for more frequent allocations; clearly mlx5_fwp_alloc() cannot be called in a data path, as it is already quite expensive, it is serialized by driver-global locks, and it may sleep.

To give a related example, UMA always zeroes slabs upon import into a zone, even though this may be redundant. I cannot see why anyone would notice such overhead.

If KMSAN is so picky, why not zero only the cmd msg pages?

It's not "picky". It's just ensuring that the kernel has initialized all memory mapped and synced by busdma. This is the bare minimum required to ensure that we do not write uninitialized kernel memory to storage or to the network.

I could zero only the cmd message pages, indeed, but this solution seemed simpler and less intrusive. If you insist, I will change it.

[kib]

It is not called on data path, but it can be called during the operations. The flow is interrupt->mlx5_eq_int with EVENT_TYPE_PAGE_REQUEST->mlx5_core_req_pages_handler->(some time later in sleepable context)give_pages()->alloc_4k()->mlx5_fwp_alloc(), then a cmd is sent back to the device with information about added page. All this time some execution inside the card is held waiting for the memory.

Live example could be a fresh KTLS context allocation.

[markj]

Push zeroing into consumers which require it.