qat: Add Intel® 4xxx Series platform support

[MichalX.Gulbicki_intel.com]

Summary

The driver exposes complete cryptography and data compression
API in the kernel and integrates with Open Crypto Framework.
Details of supported operations, devices and usage can be found
in man and on 01.org.

OCF backend changes:

• changed GCM/CCM MAC validation policy to generate MAC by HW and validate by SW due to the QAT HW limitations.

Patch co-authored by: Krzysztof Zdziarski <[e-mail]>
Patch co-authored by: Michal Jaraczewski <[e-mail]>
Patch co-authored by: Michal Gulbicki <[e-mail]>
Patch co-authored by: Julian Grajkowski <[e-mail]>
Patch co-authored by: Piotr Kasierski <[e-mail]>
Patch co-authored by: Adam Czupryna <[e-mail]>
Patch co-authored by: Konrad Zelazny <[e-mail]>
Patch co-authored by: Katarzyna Rucinska <[e-mail]>
Patch co-authored by: Lukasz Kolodzinski <[e-mail]>
Patch co-authored by: Zbigniew Jedlinski <[e-mail]>

Sponsored by: Intel Corporation

[MichalX.Gulbicki_intel.com]

Fixed invalid executable rights

[pauamma_gundo.com]

I'm pretty sure you can mark this a a draft review.

[MichalX.Gulbicki_intel.com]

Addressed issue raised in already merged review:
https://reviews.freebsd.org/D34632

[pauamma_gundo.com]

No manual page changes, so approval still holds. Remember to bump .Dd on commit.

[jhb]

I did not look at any of the common bits of the driver, just at the FreeBSD-specific OCF bits.

sys/dev/qat/qat/qat_ocf.c

1074

I'm curious why you need this? OCF in 13.0 and later mandates HW MAC verification for both EtA and AEAD modes. Or rather, clients must request MAC verification (CRYPTO_OP_VERIFY_DIGEST) in any decryption requests. Trying to set CRYPTO_OP_DECRYPT with CRYPTO_OP_COMPUTE_DIGEST is rejected in crypto.c before it even gets to the driver by crp_sanity() if INVARIANTS is enabled (it causes a panic even) and this requirement is documented in crypto_request(9).

[MichalX.Gulbicki_intel.com]

sys/dev/qat/qat/qat_ocf.c

1074

This change has been introduced as result of following discussion:

• GMAC MAC validation discussion

due the HW limitations when it comes to MAC validation (such like mlen and crp_digest_start)
we have decided to this partially in SW (the same approach as initial QAT OCF implementation).
HW is responsible for generating MAC and SW checks if it is consistent with provided one - taking into
account mlen as well (see symDpCallback). It has been tested and accepted by Mark.

In case of GCM and CCM, QAT driver forces HW MAC validation regardles of what the caller sets in
sessionSetupData.verifyDigest. To overcome this limitation and not break backward compatibility
we have introduced:

• icp_sal_setForceAEADMACVerify(CpaInstanceHandle instanceHandle, CpaBoolean forceAEADMacVerify)

By default force flag is set to true - OCF disables this to generate MAC by HW and perform additional checks. Please let me know if this answers your questions.

[jhb]

sys/dev/qat/qat/qat_ocf.c

1074

Ok. This is fine, but some other options you might consider:

1. You could reject GCM requests whose csp_auth_mlen specifies a value that QAT doesn't support. I think there are some NIST KAT tests that use tag lengths that aren't 16, but I think all of the other use cases in the kernel (IPsec, TLS, disk encryption) all use 16 byte tags.

2. You could work around the issue of needing the digest adjacent to the payload by just rewriting your scatter/gather lists you submit to the firmware to make the fields adjacent. This is the approach I used in ccr(4) which had some similar constraints. I get the "main" scatter/gather list from the buffer, but I then use the offset in struct cryptop to construct the scatter/gather list I send to the hardware, and am able to construct a scatter/gather list that always has AAD || IV || plaintext/ciphertext || MAC.

[MichalX.Gulbicki_intel.com]

sys/dev/qat/qat/qat_ocf.c

1074

Yes, it makes sense. We can change implementation and upstream together with one of the next drops (we plan a few next year).

Some concerns:

1. Further performance measurements are required to ensure that adding another flat buffer to SGL performs at least as well as writing calculated digest back to DRAM.
2. Another concern is that it breaks the backward compatibility with the original QAT OCF driver from FBSD 13 - not sure if we can do this.

Let us know if it is acceptable to you to land this patch and address this comment in the next drops?

[jhb]

Oh, I was just making some suggestions on alternate options you might want to evaluate, I think your existing approach is fine. I think it is also fine for you to decide in the future if you want to adopt a different approach.

[MichalX.Gulbicki_intel.com]

QAT Gen4 firmware placeholders replaced by official firmware binaries

[markj]

Please include some comment about the GCM digest verification in the commit log message. Ideally it would be committed as a separate patch, but I'm not sure if that's a lot of work to split out.

[MichalX.Gulbicki_intel.com]

Commit message update

[pauamma_gundo.com]

Manual page didn't change

[markj]

Committed, I'm not sure why the revision didn't close automatically: https://cgit.freebsd.org/src/commit/?id=a977168c48d45085cdf0c40f9b9bde3850b1f3ea