fwget: Introduce new utility

[manu]

Summary

This script's goal is to check the system for peripherals that needs
firmware and install the needed packages for them.
For now it only support pci subsystem and only video classes for AMD
and Intel GPUs.

Sponsored by: Beckhoff Automation GmbH & Co. KG

[lwhsu]

Does it make sense if we move the mapping to an external place like download.freebsd.org or somewhere else? (We can even create firmware.freebsd.org if needed.)
My concern is that once it's in the base it's hard to modify the list and also means that it's hard to do changes in the ports.

[manu]

In D39825#906522, @lwhsu wrote:

Does it make sense if we move the mapping to an external place like download.freebsd.org or somewhere else? (We can even create firmware.freebsd.org if needed.)

I don't think so, one of my wish is that at some point we include the firmware packages (for gpu, wifi, whatever) in the release images and that this script is runned at the end so users will have a full functional install with everything needed.

My concern is that once it's in the base it's hard to modify the list and also means that it's hard to do changes in the ports.

It's not hard to modify and if the ports are done right (like the gpu-firmware ones) there is no need to change them.

[bapt]

usr.sbin/fwget/fwget.sh

111

why not directly a pkg install -q ${packages} is the loop actually needed?

[manu]

Do not loop to install packages.

[bz]

I understand the idea. There's several drawback here for wifi:

(1) class information is (at times) useless for wifi cards - on one machine I see (for various vendors):

1 class=0x000280
3 class=0x028000
1 class=0xff0000

(2) Certain cards have "oddities" in that they probe further; for Intel I added a sysctl to export that in order to create iwlwififw.4 which I'd probably use (to script) building the mapping table.

(3) Certain chipsets are used (re-sold) by a lot of vendors so for a single card we may end up with 20 vendor files which is ugly.

(4) Maintaining these mapping tables by hand is a pain and will be inconsistent in the long term.

A lot of that, I think, could be overcome if we'd keep the per-bus information but matching more specifics and maintain a per-driver file.

[manu]

In D39825#907512, @bz wrote:

I understand the idea. There's several drawback here for wifi:

(1) class information is (at times) useless for wifi cards - on one machine I see (for various vendors):

1 class=0x000280
3 class=0x028000
1 class=0xff0000

I don't see any problems here with intel cards.
We can always expand the script later to check the pciids with another table if class matching wasn't a success.
The advantage of splitting by class is that we have smaller files to read/modify to add new devices.

(2) Certain cards have "oddities" in that they probe further; for Intel I added a sysctl to export that in order to create iwlwififw.4 which I'd probably use (to script) building the mapping table.

I'm not following.

(3) Certain chipsets are used (re-sold) by a lot of vendors so for a single card we may end up with 20 vendor files which is ugly.

Why ? Don't they use the same firmware ? and if not they don't have the same pci id no ?

(4) Maintaining these mapping tables by hand is a pain and will be inconsistent in the long term.

Not much we can do about that.

A lot of that, I think, could be overcome if we'd keep the per-bus information but matching more specifics and maintain a per-driver file.

Patches are welcome, but again class based matching should work for most devices so it should be the base.

[bz]

In D39825#907514, @manu wrote:

In D39825#907512, @bz wrote:

I understand the idea. There's several drawback here for wifi:

(1) class information is (at times) useless for wifi cards - on one machine I see (for various vendors):

1 class=0x000280
3 class=0x028000
1 class=0xff0000

I don't see any problems here with intel cards.

This example is from Atheros (QCA) and Mediatek. I don't even know what 0xff0000 is for a class. 0x000280 pciconf calls "old" (or maybe just done wrongly?)

We can always expand the script later to check the pciids with another table if class matching wasn't a success.
The advantage of splitting by class is that we have smaller files to read/modify to add new devices.

(2) Certain cards have "oddities" in that they probe further; for Intel I added a sysctl to export that in order to create iwlwififw.4 which I'd probably use (to script) building the mapping table.

I'm not following.

There are cards which beyond PCI ID checks do further checks and depending on other parts switch firmware. So you can have same PCI ID checks and different firmware.

(3) Certain chipsets are used (re-sold) by a lot of vendors so for a single card we may end up with 20 vendor files which is ugly.

Why ? Don't they use the same firmware ? and if not they don't have the same pci id no ?

No, not always; ath10k has a ubiquity PCI ID for example in addition to QCA. It's way worse on USB than on PCI though. I think rtw8822bu had 5 different vendor_ids in the early snapshot and that was when the first dongles came up. In December it was 13 different vendors already... Same chipset. It's all rebranded and IDs are just added.

(4) Maintaining these mapping tables by hand is a pain and will be inconsistent in the long term.

Not much we can do about that.

A lot of that, I think, could be overcome if we'd keep the per-bus information but matching more specifics and maintain a per-driver file.

Patches are welcome, but again class based matching should work for most devices so it should be the base.

Can we do "bus"_"class"_"drivername" at least then -- though that doesn't work on the search parts so easily as you have to iterate over all the matching bus_class_* files.

How much in a hurry are you with this? Can we think about it 5 days?

[manu]

In D39825#907702, @bz wrote:

In D39825#907514, @manu wrote:

In D39825#907512, @bz wrote:

I understand the idea. There's several drawback here for wifi:

(1) class information is (at times) useless for wifi cards - on one machine I see (for various vendors):

1 class=0x000280
3 class=0x028000
1 class=0xff0000

I don't see any problems here with intel cards.

This example is from Atheros (QCA) and Mediatek. I don't even know what 0xff0000 is for a class. 0x000280 pciconf calls "old" (or maybe just done wrongly?)

We can always expand the script later to check the pciids with another table if class matching wasn't a success.
The advantage of splitting by class is that we have smaller files to read/modify to add new devices.

(2) Certain cards have "oddities" in that they probe further; for Intel I added a sysctl to export that in order to create iwlwififw.4 which I'd probably use (to script) building the mapping table.

I'm not following.

There are cards which beyond PCI ID checks do further checks and depending on other parts switch firmware. So you can have same PCI ID checks and different firmware.

Then this script should install all possible firmware if it's not possible to know from userland which one will be used.
I don't see another way.

(3) Certain chipsets are used (re-sold) by a lot of vendors so for a single card we may end up with 20 vendor files which is ugly.

Why ? Don't they use the same firmware ? and if not they don't have the same pci id no ?

No, not always; ath10k has a ubiquity PCI ID for example in addition to QCA. It's way worse on USB than on PCI though. I think rtw8822bu had 5 different vendor_ids in the early snapshot and that was when the first dongles came up. In December it was 13 different vendors already... Same chipset. It's all rebranded and IDs are just added.

(4) Maintaining these mapping tables by hand is a pain and will be inconsistent in the long term.

Not much we can do about that.

A lot of that, I think, could be overcome if we'd keep the per-bus information but matching more specifics and maintain a per-driver file.

Patches are welcome, but again class based matching should work for most devices so it should be the base.

Can we do "bus"_"class"_"drivername" at least then -- though that doesn't work on the search parts so easily as you have to iterate over all the matching bus_class_* files.

That's what's done here no ?

How much in a hurry are you with this? Can we think about it 5 days?

I want it before 14.0 for sure but I also want it soon so people can report if I messed up some firmware/id matches.

[bz]

Can we do "bus"_"class"_"drivername" at least then -- though that doesn't work on the search parts so easily as you have to iterate over all the matching bus_class_* files.

That's what's done here no ?

I was under the assumption that pci_get_vendor() returns a vendor name and not a driver name?

How much in a hurry are you with this? Can we think about it 5 days?

I want it before 14.0 for sure but I also want it soon so people can report if I messed up some firmware/id matches.

ACK.

[imp]

is there any way to leverage the firmware system to do this? Eg, when a firmware can't be found, then a script likes this asking for it by name. We can then use that to lookup what packageg to download, no?

It would be relatively simple to add this to devd, but when the load happens is early in boot, so we may need to defer it to later in boot... That way, we don't need a hardware to driver mapping... It might be a little ugly to have the driver fail when there's no firmware, but I think it would mean not having to have a complete database and leverage what's already in the tree. We should package the firmware such that if we install one package, the any device that installs multiple firmwares would have it... if at all possible...

[manu]

In D39825#907729, @bz wrote:

Can we do "bus"_"class"_"drivername" at least then -- though that doesn't work on the search parts so easily as you have to iterate over all the matching bus_class_* files.

That's what's done here no ?

I was under the assumption that pci_get_vendor() returns a vendor name and not a driver name?

Yes because in my case there is no driver (it's vgapci), and likely for your case too, as the driver will not attach if there is no firmware available to be loaded. So yes I could add the 'vgapci' case for me but for you (and some network adapter too I guess) this would be the 'none' part of pciconf which doesn't help.

How much in a hurry are you with this? Can we think about it 5 days?

I want it before 14.0 for sure but I also want it soon so people can report if I messed up some firmware/id matches.

ACK.

[manu]

In D39825#907763, @imp wrote:

is there any way to leverage the firmware system to do this? Eg, when a firmware can't be found, then a script likes this asking for it by name. We can then use that to lookup what packageg to download, no?

I don't think that's possible as the firmware subsystem is all kernel and we can't call this script from it (well it's not good to to that :P)

It would be relatively simple to add this to devd, but when the load happens is early in boot, so we may need to defer it to later in boot... That way, we don't need a hardware to driver mapping... It might be a little ugly to have the driver fail when there's no firmware, but I think it would mean not having to have a complete database and leverage what's already in the tree.

In case of GPU drivers it always fails for AMD as the firmware is always needed, for intel it will only warn as the firmware is used only for suspend/resume, driver works fine without it.

We should package the firmware such that if we install one package, the any device that installs multiple firmwares would have it... if at all possible...

[imp]

In D39825#907766, @manu wrote:

In D39825#907763, @imp wrote:

is there any way to leverage the firmware system to do this? Eg, when a firmware can't be found, then a script likes this asking for it by name. We can then use that to lookup what packageg to download, no?

I don't think that's possible as the firmware subsystem is all kernel and we can't call this script from it (well it's not good to to that :P)

They said that about device matching too :)

You can create a devd event to indicate firmware loading failure, and later use that to work things out. Short term the catalog may work (and that may be fine for 14.0), but longer term I think we'll need to cope with things like this.

It would be relatively simple to add this to devd, but when the load happens is early in boot, so we may need to defer it to later in boot... That way, we don't need a hardware to driver mapping... It might be a little ugly to have the driver fail when there's no firmware, but I think it would mean not having to have a complete database and leverage what's already in the tree.

In case of GPU drivers it always fails for AMD as the firmware is always needed, for intel it will only warn as the firmware is used only for suspend/resume, driver works fine without it.

For AMD you still have video without the drivers... But I'm not sure the failure mode to know if you can try loading once, then snag thee firmware and then try again or not :).

The only other way around that would be to add a lot of code to the firmware loading that treats failure as "devd event plus wait". And then later, something wakes up the firmware loader to try again... But this level of interlock would be a lot harder to do and chock-a-block full of races and edge cases (which is why devd's NOMATCH events are fire and forget).

What's here is cool and all, and it's no doubt useful for 14 since it's the bird in the hand. So don't let my notions for a better way to build the mousetrap derail that...

As for the ambiguity, I'd suggest we take a page from devmatch's approach: load them all. While it's nice to only install one firmware, installing 3 or 5 instead of 50 is still a big win.

[manu]

In D39825#907787, @imp wrote:

In D39825#907766, @manu wrote:

In D39825#907763, @imp wrote:

is there any way to leverage the firmware system to do this? Eg, when a firmware can't be found, then a script likes this asking for it by name. We can then use that to lookup what packageg to download, no?

I don't think that's possible as the firmware subsystem is all kernel and we can't call this script from it (well it's not good to to that :P)

They said that about device matching too :)

You can create a devd event to indicate firmware loading failure, and later use that to work things out. Short term the catalog may work (and that may be fine for 14.0), but longer term I think we'll need to cope with things like this.

Yeah we can probably extend the firmware subsystem to do something like that in the near futur.

It would be relatively simple to add this to devd, but when the load happens is early in boot, so we may need to defer it to later in boot... That way, we don't need a hardware to driver mapping... It might be a little ugly to have the driver fail when there's no firmware, but I think it would mean not having to have a complete database and leverage what's already in the tree.

In case of GPU drivers it always fails for AMD as the firmware is always needed, for intel it will only warn as the firmware is used only for suspend/resume, driver works fine without it.

For AMD you still have video without the drivers... But I'm not sure the failure mode to know if you can try loading once, then snag thee firmware and then try again or not :).

You don't, without firmware the driver fails but we already switched the framebuffer and don't have efifb anymore. IIRC it was fixed in Linux around 6.0-6.1 . At least that's what I see on my machine, maybe it depends on the generation.

The only other way around that would be to add a lot of code to the firmware loading that treats failure as "devd event plus wait". And then later, something wakes up the firmware loader to try again... But this level of interlock would be a lot harder to do and chock-a-block full of races and edge cases (which is why devd's NOMATCH events are fire and forget).

What's here is cool and all, and it's no doubt useful for 14 since it's the bird in the hand. So don't let my notions for a better way to build the mousetrap derail that...

As for the ambiguity, I'd suggest we take a page from devmatch's approach: load them all. While it's nice to only install one firmware, installing 3 or 5 instead of 50 is still a big win.