Newly added features and bug fixes in latest Microchip SmartPQI driver.

[imp]

Summary

It includes:

1)Newly added TMF feature.
2)Added newly Huawei & Inspur PCI ID's
3)Fixed smartpqi driver hangs in Z-Pool while running on FreeBSD12.1
4)Fixed flooding dmesg in kernel while the controller is offline during in ioctls.
5)Avoided unnecessary host memory allocation for rcb sg buffers.
6)Fixed race conditions while accessing internal rcb structure.
7)Fixed where Logical volumes exposing two different names to the OS it's due to the system memory is overwritten with DMA stale data.
8)Fixed dynamically unloading a smartpqi driver.
9)Added device_shutdown callback instead of deprecated shutdown_final kernel event in smartpqi driver.
10)Fixed where Os is crashed during physical drive hot removal during heavy IO.
11)Fixed OS crash during controller lockup/offline during heavy IO.
12)Fixed coverity issues in smartpqi driver
13)Fixed system crash while creating and deleting logical volume in a continuous loop.
14)Fixed where the volume size is not exposing to OS when it expands.
15)Added HC3 pci id's.

[papani.srikanth_microchip.com]

Can someone approve/review the code ?

[yuripv]

With this patch applied, it fails much faster for me completely locking up ZFS, usually after some minutes of light load (e.g. I was doing git gc when this happened):

[ERROR]::[4:655.0][CPU 0][pqisrc_heartbeat_timer_handler][178]:controller is offline
(da1:smartpqi0:0:65:0): WRITE(10). CDB: 2a 00 05 82 4f e0 00 07 e8 00
(da2:smartpqi0:0:66:0): WRITE(10). CDB: 2a 00 05 82 7d 98 00 00 58 00
(da1:smartpqi0:0:65:0): CAM status: Unable to abort CCB request
(da1:smartpqi0:0:65:0): Error 5, Unretryable error
(da1:smartpqi0:0:65:0): WRITE(10). CDB: 2a 00 05 82 67 98 00 07 e8 00
(da1:smartpqi0:0:65:0): CAM status: Unable to abort CCB request
(da1:smartpqi0:0:65:0): Error 5, Unretryable error
...
da0 at smartpqi0 bus 0 scbus0 target 64 lun 0
da0: <ATA WDC WD40PURZ-85A 0A80>  s/n WD-WX32D7088CCV detached
(da1:smartpqi0:0:65:0): Error 5, Unretryable error
(da1:smartpqi0:0:65:0): WRITE(10). CDB: 2a 00 05 82 38 28 00 07 e8 00
(da1:smartpqi0:0:65:0): CAM status: Unable to abort CCB request
(da1:smartpqi0:0:65:0): Error 5, Unretryable error
da1 at smartpqi0 bus 0 scbus0 target 65 lun 0
da1: <ATA WDC WD40PURZ-85A 0A80>  s/n WD-WX42D70CHZS7 detached
(da2:smartpqi0:0:66:0): CAM status: Unable to abort CCB request
(da2:smartpqi0:0:66:0): Error 5, Unretryable error
(da2:smartpqi0:0:66:0): WRITE(10). CDB: 2a 00 05 82 75 b0 00 07 e8 00
(da2:smartpqi0:0:66:0): CAM status: Unable to abort CCB request
(da2:smartpqi0:0:66:0): Error 5, Unretryable error
da2 at smartpqi0 bus 0 scbus0 target 66 lun 0
da2: <ATA WDC WD40PURZ-85A 0A80>  s/n WD-WXC2D90D7YAX detached
da3 at smartpqi0 bus 0 scbus0 target 67 lun 0
da3: <ATA WDC WD40PURZ-85A 0A80>  s/n WD-WX12DB0N8F4X detached
ses0 at smartpqi0 bus 0 scbus0 target 68 lun 0
ses0: <Adaptec Smart Adapter 3.53>  s/n 7A4263EAB3E     detached
pass5 at smartpqi0 bus 0 scbus0 target 1088 lun 1
pass5: <Adaptec 1100-8i 3.53>  s/n 7A4263EAB3E     detached
(ses0:smartpqi0:0:68:0): Periph destroyed
(pass5:smartpqi0:0:1088:1): Periph destroyed
Solaris: WARNING: Pool 'data' has encountered an uncorrectable I/O failure and has been suspended.

Apr 16 11:36:45 sun ZFS[45956]: catastrophic pool I/O failure, zpool=data

Now every FS access is stuck. I was able to save the kernel dump using NMI (in case we can get something interesting from it).

This same system works just fine using the same HBA, same disks, same cabling under illumos (using our internal smartpqi driver) transferring TBs worth of data, so I don't expect this to be hardware issue.

HBA is 1100-8i, disks are 4x WDC WD40PURZ SATA3 HDDs, connected using breakout cable.

[mav]

It's a pretty big patch with many changes to easily review. I barely looked through 1/8th of it. It would be good if you at least separated formatting from substantial changes.

sys/dev/smartpqi/smartpqi_cam.c

269

Why have you decided to terminate those strings? It is allowed, but not required by the standard.

605–606

style(9) tells: "Values in return statements should be enclosed in parentheses." Plus you are simply inconsistent with other code parts.

[imp]

In D29584#668490, @mav wrote:

It's a pretty big patch with many changes to easily review. I barely looked through 1/8th of it. It would be good if you at least separated formatting from substantial changes.

Thats' why I've not looked more deeply at it. OTOH, since this is basically vendor code, I'm inclined to not worry so much about the formatting and similar issues, but they do make it hard to review whether or not new bugs are sneaking in with all the bug fixes and formatting changes. It's tough to know how to balance these issues. Though reports of lockups here are concerning enough that I think some response to them is needed before proceeding.

[scott.benesh_microsemi.com]

In D29584#668508, @imp wrote:

In D29584#668490, @mav wrote:

It's a pretty big patch with many changes to easily review. I barely looked through 1/8th of it. It would be good if you at least separated formatting from substantial changes.

Thats' why I've not looked more deeply at it. OTOH, since this is basically vendor code, I'm inclined to not worry so much about the formatting and similar issues, but they do make it hard to review whether or not new bugs are sneaking in with all the bug fixes and formatting changes. It's tough to know how to balance these issues. Though reports of lockups here are concerning enough that I think some response to them is needed before proceeding.

Yeah, I saw the lockup report came in earlier today. Unfortunately Srikanth is in a different time zone so he will take a look at it as soon as he can. I do understand the code changes got a bit out of hand as we are trying to sync up to our latest code base. We'll see if we can make the patches more palatable. Any suggestions are always appreciated.

[papani.srikanth_microchip.com]

In D29584#668301, @yuripv wrote:

With this patch applied, it fails much faster for me completely locking up ZFS, usually after some minutes of light load (e.g. I was doing git gc when this happened):

[ERROR]::[4:655.0][CPU 0][pqisrc_heartbeat_timer_handler][178]:controller is offline
(da1:smartpqi0:0:65:0): WRITE(10). CDB: 2a 00 05 82 4f e0 00 07 e8 00
(da2:smartpqi0:0:66:0): WRITE(10). CDB: 2a 00 05 82 7d 98 00 00 58 00
(da1:smartpqi0:0:65:0): CAM status: Unable to abort CCB request
(da1:smartpqi0:0:65:0): Error 5, Unretryable error
(da1:smartpqi0:0:65:0): WRITE(10). CDB: 2a 00 05 82 67 98 00 07 e8 00
(da1:smartpqi0:0:65:0): CAM status: Unable to abort CCB request
(da1:smartpqi0:0:65:0): Error 5, Unretryable error
...
da0 at smartpqi0 bus 0 scbus0 target 64 lun 0
da0: <ATA WDC WD40PURZ-85A 0A80>  s/n WD-WX32D7088CCV detached
(da1:smartpqi0:0:65:0): Error 5, Unretryable error
(da1:smartpqi0:0:65:0): WRITE(10). CDB: 2a 00 05 82 38 28 00 07 e8 00
(da1:smartpqi0:0:65:0): CAM status: Unable to abort CCB request
(da1:smartpqi0:0:65:0): Error 5, Unretryable error
da1 at smartpqi0 bus 0 scbus0 target 65 lun 0
da1: <ATA WDC WD40PURZ-85A 0A80>  s/n WD-WX42D70CHZS7 detached
(da2:smartpqi0:0:66:0): CAM status: Unable to abort CCB request
(da2:smartpqi0:0:66:0): Error 5, Unretryable error
(da2:smartpqi0:0:66:0): WRITE(10). CDB: 2a 00 05 82 75 b0 00 07 e8 00
(da2:smartpqi0:0:66:0): CAM status: Unable to abort CCB request
(da2:smartpqi0:0:66:0): Error 5, Unretryable error
da2 at smartpqi0 bus 0 scbus0 target 66 lun 0
da2: <ATA WDC WD40PURZ-85A 0A80>  s/n WD-WXC2D90D7YAX detached
da3 at smartpqi0 bus 0 scbus0 target 67 lun 0
da3: <ATA WDC WD40PURZ-85A 0A80>  s/n WD-WX12DB0N8F4X detached
ses0 at smartpqi0 bus 0 scbus0 target 68 lun 0
ses0: <Adaptec Smart Adapter 3.53>  s/n 7A4263EAB3E     detached
pass5 at smartpqi0 bus 0 scbus0 target 1088 lun 1
pass5: <Adaptec 1100-8i 3.53>  s/n 7A4263EAB3E     detached
(ses0:smartpqi0:0:68:0): Periph destroyed
(pass5:smartpqi0:0:1088:1): Periph destroyed
Solaris: WARNING: Pool 'data' has encountered an uncorrectable I/O failure and has been suspended.

Apr 16 11:36:45 sun ZFS[45956]: catastrophic pool I/O failure, zpool=data

Now every FS access is stuck. I was able to save the kernel dump using NMI (in case we can get something interesting from it).

This same system works just fine using the same HBA, same disks, same cabling under illumos (using our internal smartpqi driver) transferring TBs worth of data, so I don't expect this to be hardware issue.

HBA is 1100-8i, disks are 4x WDC WD40PURZ SATA3 HDDs, connected using breakout cable.

Thanks for testing the changes, The lockup issue is new to us. Could you please provide the Z-Pool reproduction steps so we can reflect the setup in our lab too for testing.

[yuripv]

In D29584#669299, @papani.srikanth_microchip.com wrote:

In D29584#668301, @yuripv wrote:

With this patch applied, it fails much faster for me completely locking up ZFS, usually after some minutes of light load (e.g. I was doing git gc when this happened):

[ERROR]::[4:655.0][CPU 0][pqisrc_heartbeat_timer_handler][178]:controller is offline
(da1:smartpqi0:0:65:0): WRITE(10). CDB: 2a 00 05 82 4f e0 00 07 e8 00
(da2:smartpqi0:0:66:0): WRITE(10). CDB: 2a 00 05 82 7d 98 00 00 58 00
(da1:smartpqi0:0:65:0): CAM status: Unable to abort CCB request
(da1:smartpqi0:0:65:0): Error 5, Unretryable error
(da1:smartpqi0:0:65:0): WRITE(10). CDB: 2a 00 05 82 67 98 00 07 e8 00
(da1:smartpqi0:0:65:0): CAM status: Unable to abort CCB request
(da1:smartpqi0:0:65:0): Error 5, Unretryable error
...
da0 at smartpqi0 bus 0 scbus0 target 64 lun 0
da0: <ATA WDC WD40PURZ-85A 0A80>  s/n WD-WX32D7088CCV detached
(da1:smartpqi0:0:65:0): Error 5, Unretryable error
(da1:smartpqi0:0:65:0): WRITE(10). CDB: 2a 00 05 82 38 28 00 07 e8 00
(da1:smartpqi0:0:65:0): CAM status: Unable to abort CCB request
(da1:smartpqi0:0:65:0): Error 5, Unretryable error
da1 at smartpqi0 bus 0 scbus0 target 65 lun 0
da1: <ATA WDC WD40PURZ-85A 0A80>  s/n WD-WX42D70CHZS7 detached
(da2:smartpqi0:0:66:0): CAM status: Unable to abort CCB request
(da2:smartpqi0:0:66:0): Error 5, Unretryable error
(da2:smartpqi0:0:66:0): WRITE(10). CDB: 2a 00 05 82 75 b0 00 07 e8 00
(da2:smartpqi0:0:66:0): CAM status: Unable to abort CCB request
(da2:smartpqi0:0:66:0): Error 5, Unretryable error
da2 at smartpqi0 bus 0 scbus0 target 66 lun 0
da2: <ATA WDC WD40PURZ-85A 0A80>  s/n WD-WXC2D90D7YAX detached
da3 at smartpqi0 bus 0 scbus0 target 67 lun 0
da3: <ATA WDC WD40PURZ-85A 0A80>  s/n WD-WX12DB0N8F4X detached
ses0 at smartpqi0 bus 0 scbus0 target 68 lun 0
ses0: <Adaptec Smart Adapter 3.53>  s/n 7A4263EAB3E     detached
pass5 at smartpqi0 bus 0 scbus0 target 1088 lun 1
pass5: <Adaptec 1100-8i 3.53>  s/n 7A4263EAB3E     detached
(ses0:smartpqi0:0:68:0): Periph destroyed
(pass5:smartpqi0:0:1088:1): Periph destroyed
Solaris: WARNING: Pool 'data' has encountered an uncorrectable I/O failure and has been suspended.

Apr 16 11:36:45 sun ZFS[45956]: catastrophic pool I/O failure, zpool=data

Now every FS access is stuck. I was able to save the kernel dump using NMI (in case we can get something interesting from it).

This same system works just fine using the same HBA, same disks, same cabling under illumos (using our internal smartpqi driver) transferring TBs worth of data, so I don't expect this to be hardware issue.

HBA is 1100-8i, disks are 4x WDC WD40PURZ SATA3 HDDs, connected using breakout cable.

Thanks for testing the changes, The lockup issue is new to us. Could you please provide the Z-Pool reproduction steps so we can reflect the setup in our lab too for testing.

Pool configuration is simple -- raidz of 4 sata disks connected using breakout cable, i.e.:

# for i in $(seq 0 3); do gpart create -s gpt da$i; gpart add -t freebsd-zfs da$i; done
da0 created
da0p1 added
da1 created
da1p1 added
da2 created
da2p1 added
da3 created
da3p1 added
# zpool create -O atime=off -O compression=zstd data raidz da0p1 da1p1 da2p1 da3p1
# zpool status data
  pool: data
 state: ONLINE
config:

        NAME        STATE     READ WRITE CKSUM
        data        ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            da0p1   ONLINE       0     0     0
            da1p1   ONLINE       0     0     0
            da2p1   ONLINE       0     0     0
            da3p1   ONLINE       0     0     0

errors: No known data errors

I don't know what exactly triggers the issue, usually it's some simple operation like git clone, git pull, git gc and so on.

[chuck]

FWIW, applying this patch via git apply D29584.diff results in a number of trailing whitespace complaints:

/home/tuffli/D29584.diff:70: trailing whitespace.
/home/tuffli/D29584.diff:100: trailing whitespace.
/home/tuffli/D29584.diff:1126: trailing whitespace.
/home/tuffli/D29584.diff:1850: trailing whitespace.
/home/tuffli/D29584.diff:1905: trailing whitespace.
/home/tuffli/D29584.diff:2113: trailing whitespace.
/home/tuffli/D29584.diff:3307: trailing whitespace.
/home/tuffli/D29584.diff:4254: trailing whitespace.
/home/tuffli/D29584.diff:4326: trailing whitespace.
/home/tuffli/D29584.diff:5418: trailing whitespace.
/home/tuffli/D29584.diff:6860: trailing whitespace.

[papani.srikanth_microchip.com]

• Added lockup code info in driver,

Re-try the IO's if there is a lack of DMA resources instead of deferring.

[papani.srikanth_microchip.com]

In D29584#670456, @chuck wrote:

FWIW, applying this patch via git apply D29584.diff results in a number of trailing whitespace complaints:

/home/tuffli/D29584.diff:70: trailing whitespace.
/home/tuffli/D29584.diff:100: trailing whitespace.
/home/tuffli/D29584.diff:1126: trailing whitespace.
/home/tuffli/D29584.diff:1850: trailing whitespace.
/home/tuffli/D29584.diff:1905: trailing whitespace.
/home/tuffli/D29584.diff:2113: trailing whitespace.
/home/tuffli/D29584.diff:3307: trailing whitespace.
/home/tuffli/D29584.diff:4254: trailing whitespace.
/home/tuffli/D29584.diff:4326: trailing whitespace.
/home/tuffli/D29584.diff:5418: trailing whitespace.
/home/tuffli/D29584.diff:6860: trailing whitespace.

Hi Chuck,
This code is for FreeBSD 14.0 main branch, You can take it out for FreeBSD 13.0 testing

[papani.srikanth_microchip.com]

In D29584#669304, @yuripv wrote:

In D29584#669299, @papani.srikanth_microchip.com wrote:

In D29584#668301, @yuripv wrote:

With this patch applied, it fails much faster for me completely locking up ZFS, usually after some minutes of light load (e.g. I was doing git gc when this happened):

[ERROR]::[4:655.0][CPU 0][pqisrc_heartbeat_timer_handler][178]:controller is offline
(da1:smartpqi0:0:65:0): WRITE(10). CDB: 2a 00 05 82 4f e0 00 07 e8 00
(da2:smartpqi0:0:66:0): WRITE(10). CDB: 2a 00 05 82 7d 98 00 00 58 00
(da1:smartpqi0:0:65:0): CAM status: Unable to abort CCB request
(da1:smartpqi0:0:65:0): Error 5, Unretryable error
(da1:smartpqi0:0:65:0): WRITE(10). CDB: 2a 00 05 82 67 98 00 07 e8 00
(da1:smartpqi0:0:65:0): CAM status: Unable to abort CCB request
(da1:smartpqi0:0:65:0): Error 5, Unretryable error
...
da0 at smartpqi0 bus 0 scbus0 target 64 lun 0
da0: <ATA WDC WD40PURZ-85A 0A80>  s/n WD-WX32D7088CCV detached
(da1:smartpqi0:0:65:0): Error 5, Unretryable error
(da1:smartpqi0:0:65:0): WRITE(10). CDB: 2a 00 05 82 38 28 00 07 e8 00
(da1:smartpqi0:0:65:0): CAM status: Unable to abort CCB request
(da1:smartpqi0:0:65:0): Error 5, Unretryable error
da1 at smartpqi0 bus 0 scbus0 target 65 lun 0
da1: <ATA WDC WD40PURZ-85A 0A80>  s/n WD-WX42D70CHZS7 detached
(da2:smartpqi0:0:66:0): CAM status: Unable to abort CCB request
(da2:smartpqi0:0:66:0): Error 5, Unretryable error
(da2:smartpqi0:0:66:0): WRITE(10). CDB: 2a 00 05 82 75 b0 00 07 e8 00
(da2:smartpqi0:0:66:0): CAM status: Unable to abort CCB request
(da2:smartpqi0:0:66:0): Error 5, Unretryable error
da2 at smartpqi0 bus 0 scbus0 target 66 lun 0
da2: <ATA WDC WD40PURZ-85A 0A80>  s/n WD-WXC2D90D7YAX detached
da3 at smartpqi0 bus 0 scbus0 target 67 lun 0
da3: <ATA WDC WD40PURZ-85A 0A80>  s/n WD-WX12DB0N8F4X detached
ses0 at smartpqi0 bus 0 scbus0 target 68 lun 0
ses0: <Adaptec Smart Adapter 3.53>  s/n 7A4263EAB3E     detached
pass5 at smartpqi0 bus 0 scbus0 target 1088 lun 1
pass5: <Adaptec 1100-8i 3.53>  s/n 7A4263EAB3E     detached
(ses0:smartpqi0:0:68:0): Periph destroyed
(pass5:smartpqi0:0:1088:1): Periph destroyed
Solaris: WARNING: Pool 'data' has encountered an uncorrectable I/O failure and has been suspended.

Apr 16 11:36:45 sun ZFS[45956]: catastrophic pool I/O failure, zpool=data

Now every FS access is stuck. I was able to save the kernel dump using NMI (in case we can get something interesting from it).

This same system works just fine using the same HBA, same disks, same cabling under illumos (using our internal smartpqi driver) transferring TBs worth of data, so I don't expect this to be hardware issue.

HBA is 1100-8i, disks are 4x WDC WD40PURZ SATA3 HDDs, connected using breakout cable.

Thanks for testing the changes, The lockup issue is new to us. Could you please provide the Z-Pool reproduction steps so we can reflect the setup in our lab too for testing.

Pool configuration is simple -- raidz of 4 sata disks connected using breakout cable, i.e.:

# for i in $(seq 0 3); do gpart create -s gpt da$i; gpart add -t freebsd-zfs da$i; done
da0 created
da0p1 added
da1 created
da1p1 added
da2 created
da2p1 added
da3 created
da3p1 added
# zpool create -O atime=off -O compression=zstd data raidz da0p1 da1p1 da2p1 da3p1
# zpool status data
  pool: data
 state: ONLINE
config:

        NAME        STATE     READ WRITE CKSUM
        data        ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            da0p1   ONLINE       0     0     0
            da1p1   ONLINE       0     0     0
            da2p1   ONLINE       0     0     0
            da3p1   ONLINE       0     0     0

errors: No known data errors

I don't know what exactly triggers the issue, usually it's some simple operation like git clone, git pull, git gc and so on.

Can you please take this patch for testing

[chuck]

I have several systems with zpools on top of SAS drives behind an Adaptec Smart Storage PQI SAS (rev 01) controller. These zpools host Zvols used by VMs running under bhyve. On 13.0-RELEASE, the Linux guests have a workload which consistently hangs. On the FreeBSD host, dmesg contains errors like:

May 17 09:20:17 xxxxxxxx kernel: [ERROR]::[92:655.0][0,70,0][CPU 58][pqi_map_request][551]:bus_dmamap_load_ccb failed = 36 count = 1036288
May 17 09:20:17 xxxxxxxx kernel: [WARN]:[92:655.0][CPU 58][pqisrc_io_start][802]:In Progress on 70

Applying this patch to releng/13.0 fixes the hangs seen in the Linux guests.

Tested by: chuck

[papani.srikanth_microchip.com]

Quoted Text

In D29584#680839, @chuck wrote:

I have several systems with zpools on top of SAS drives behind an Adaptec Smart Storage PQI SAS (rev 01) controller. These zpools host Zvols used by VMs running under bhyve. On 13.0-RELEASE, the Linux guests have a workload which consistently hangs. On the FreeBSD host, dmesg contains errors like:

May 17 09:20:17 xxxxxxxx kernel: [ERROR]::[92:655.0][0,70,0][CPU 58][pqi_map_request][551]:bus_dmamap_load_ccb failed = 36 count = 1036288
May 17 09:20:17 xxxxxxxx kernel: [WARN]:[92:655.0][CPU 58][pqisrc_io_start][802]:In Progress on 70

Applying this patch to releng/13.0 fixes the hangs seen in the Linux guests.

Tested by: chuck

Thank you chuck for spending your valuable time on SmartPQI driver.

Regards,
Srikanth

[papani.srikanth_microchip.com]

Fixed lockup issue

[papani.srikanth_microchip.com]

Fixed lockup issues and trailing white space errors.
FreeBSD 12.2 and 13.0 code reviews patches doesn't have lockup code fix, it
will be pulled once it's verified in main branch.

[papani.srikanth_microchip.com]

Hi Yuripv,

Can you please take the patch, We fixed the lockup issue in latest patch set. I hope it solves the issue.

Regards,
Srikanth

In D29584#668301, @yuripv wrote:

With this patch applied, it fails much faster for me completely locking up ZFS, usually after some minutes of light load (e.g. I was doing git gc when this happened):

[ERROR]::[4:655.0][CPU 0][pqisrc_heartbeat_timer_handler][178]:controller is offline
(da1:smartpqi0:0:65:0): WRITE(10). CDB: 2a 00 05 82 4f e0 00 07 e8 00
(da2:smartpqi0:0:66:0): WRITE(10). CDB: 2a 00 05 82 7d 98 00 00 58 00
(da1:smartpqi0:0:65:0): CAM status: Unable to abort CCB request
(da1:smartpqi0:0:65:0): Error 5, Unretryable error
(da1:smartpqi0:0:65:0): WRITE(10). CDB: 2a 00 05 82 67 98 00 07 e8 00
(da1:smartpqi0:0:65:0): CAM status: Unable to abort CCB request
(da1:smartpqi0:0:65:0): Error 5, Unretryable error
...
da0 at smartpqi0 bus 0 scbus0 target 64 lun 0
da0: <ATA WDC WD40PURZ-85A 0A80>  s/n WD-WX32D7088CCV detached
(da1:smartpqi0:0:65:0): Error 5, Unretryable error
(da1:smartpqi0:0:65:0): WRITE(10). CDB: 2a 00 05 82 38 28 00 07 e8 00
(da1:smartpqi0:0:65:0): CAM status: Unable to abort CCB request
(da1:smartpqi0:0:65:0): Error 5, Unretryable error
da1 at smartpqi0 bus 0 scbus0 target 65 lun 0
da1: <ATA WDC WD40PURZ-85A 0A80>  s/n WD-WX42D70CHZS7 detached
(da2:smartpqi0:0:66:0): CAM status: Unable to abort CCB request
(da2:smartpqi0:0:66:0): Error 5, Unretryable error
(da2:smartpqi0:0:66:0): WRITE(10). CDB: 2a 00 05 82 75 b0 00 07 e8 00
(da2:smartpqi0:0:66:0): CAM status: Unable to abort CCB request
(da2:smartpqi0:0:66:0): Error 5, Unretryable error
da2 at smartpqi0 bus 0 scbus0 target 66 lun 0
da2: <ATA WDC WD40PURZ-85A 0A80>  s/n WD-WXC2D90D7YAX detached
da3 at smartpqi0 bus 0 scbus0 target 67 lun 0
da3: <ATA WDC WD40PURZ-85A 0A80>  s/n WD-WX12DB0N8F4X detached
ses0 at smartpqi0 bus 0 scbus0 target 68 lun 0
ses0: <Adaptec Smart Adapter 3.53>  s/n 7A4263EAB3E     detached
pass5 at smartpqi0 bus 0 scbus0 target 1088 lun 1
pass5: <Adaptec 1100-8i 3.53>  s/n 7A4263EAB3E     detached
(ses0:smartpqi0:0:68:0): Periph destroyed
(pass5:smartpqi0:0:1088:1): Periph destroyed
Solaris: WARNING: Pool 'data' has encountered an uncorrectable I/O failure and has been suspended.

Apr 16 11:36:45 sun ZFS[45956]: catastrophic pool I/O failure, zpool=data

Now every FS access is stuck. I was able to save the kernel dump using NMI (in case we can get something interesting from it).

This same system works just fine using the same HBA, same disks, same cabling under illumos (using our internal smartpqi driver) transferring TBs worth of data, so I don't expect this to be hardware issue.

HBA is 1100-8i, disks are 4x WDC WD40PURZ SATA3 HDDs, connected using breakout cable.

[papani.srikanth_microchip.com]

Hi Yuripv,

Could you please spend some time for testing the lockup issue, if the testing is already started then please discard this message.

---

Regards,
Srikanth

In D29584#669304, @yuripv wrote:

In D29584#669299, @papani.srikanth_microchip.com wrote:

In D29584#668301, @yuripv wrote:

With this patch applied, it fails much faster for me completely locking up ZFS, usually after some minutes of light load (e.g. I was doing git gc when this happened):

[ERROR]::[4:655.0][CPU 0][pqisrc_heartbeat_timer_handler][178]:controller is offline
(da1:smartpqi0:0:65:0): WRITE(10). CDB: 2a 00 05 82 4f e0 00 07 e8 00
(da2:smartpqi0:0:66:0): WRITE(10). CDB: 2a 00 05 82 7d 98 00 00 58 00
(da1:smartpqi0:0:65:0): CAM status: Unable to abort CCB request
(da1:smartpqi0:0:65:0): Error 5, Unretryable error
(da1:smartpqi0:0:65:0): WRITE(10). CDB: 2a 00 05 82 67 98 00 07 e8 00
(da1:smartpqi0:0:65:0): CAM status: Unable to abort CCB request
(da1:smartpqi0:0:65:0): Error 5, Unretryable error
...
da0 at smartpqi0 bus 0 scbus0 target 64 lun 0
da0: <ATA WDC WD40PURZ-85A 0A80>  s/n WD-WX32D7088CCV detached
(da1:smartpqi0:0:65:0): Error 5, Unretryable error
(da1:smartpqi0:0:65:0): WRITE(10). CDB: 2a 00 05 82 38 28 00 07 e8 00
(da1:smartpqi0:0:65:0): CAM status: Unable to abort CCB request
(da1:smartpqi0:0:65:0): Error 5, Unretryable error
da1 at smartpqi0 bus 0 scbus0 target 65 lun 0
da1: <ATA WDC WD40PURZ-85A 0A80>  s/n WD-WX42D70CHZS7 detached
(da2:smartpqi0:0:66:0): CAM status: Unable to abort CCB request
(da2:smartpqi0:0:66:0): Error 5, Unretryable error
(da2:smartpqi0:0:66:0): WRITE(10). CDB: 2a 00 05 82 75 b0 00 07 e8 00
(da2:smartpqi0:0:66:0): CAM status: Unable to abort CCB request
(da2:smartpqi0:0:66:0): Error 5, Unretryable error
da2 at smartpqi0 bus 0 scbus0 target 66 lun 0
da2: <ATA WDC WD40PURZ-85A 0A80>  s/n WD-WXC2D90D7YAX detached
da3 at smartpqi0 bus 0 scbus0 target 67 lun 0
da3: <ATA WDC WD40PURZ-85A 0A80>  s/n WD-WX12DB0N8F4X detached
ses0 at smartpqi0 bus 0 scbus0 target 68 lun 0
ses0: <Adaptec Smart Adapter 3.53>  s/n 7A4263EAB3E     detached
pass5 at smartpqi0 bus 0 scbus0 target 1088 lun 1
pass5: <Adaptec 1100-8i 3.53>  s/n 7A4263EAB3E     detached
(ses0:smartpqi0:0:68:0): Periph destroyed
(pass5:smartpqi0:0:1088:1): Periph destroyed
Solaris: WARNING: Pool 'data' has encountered an uncorrectable I/O failure and has been suspended.

Apr 16 11:36:45 sun ZFS[45956]: catastrophic pool I/O failure, zpool=data

Now every FS access is stuck. I was able to save the kernel dump using NMI (in case we can get something interesting from it).

This same system works just fine using the same HBA, same disks, same cabling under illumos (using our internal smartpqi driver) transferring TBs worth of data, so I don't expect this to be hardware issue.

HBA is 1100-8i, disks are 4x WDC WD40PURZ SATA3 HDDs, connected using breakout cable.

Thanks for testing the changes, The lockup issue is new to us. Could you please provide the Z-Pool reproduction steps so we can reflect the setup in our lab too for testing.

Pool configuration is simple -- raidz of 4 sata disks connected using breakout cable, i.e.:

# for i in $(seq 0 3); do gpart create -s gpt da$i; gpart add -t freebsd-zfs da$i; done
da0 created
da0p1 added
da1 created
da1p1 added
da2 created
da2p1 added
da3 created
da3p1 added
# zpool create -O atime=off -O compression=zstd data raidz da0p1 da1p1 da2p1 da3p1
# zpool status data
  pool: data
 state: ONLINE
config:

        NAME        STATE     READ WRITE CKSUM
        data        ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            da0p1   ONLINE       0     0     0
            da1p1   ONLINE       0     0     0
            da2p1   ONLINE       0     0     0
            da3p1   ONLINE       0     0     0

errors: No known data errors

I don't know what exactly triggers the issue, usually it's some simple operation like git clone, git pull, git gc and so on.

[papani.srikanth_microchip.com]

Hi @yuripv

Have you tested the current patch set? if not can you please complete it ASAP, Since it has been idle from one month.

Regards,
Srikanth

[hermes.tk_microchip.com]

In D29584#692127, @papani.srikanth_microchip.com wrote:

Hi @yuripv

Have you tested the current patch set? if not can you please complete it ASAP, Since it has been idle from one month.

Regards,
Srikanth

HI @yuripv,

Have you tested the current patch ? if not can you please complete the testing ASAP ?

Thanks & Regards
Hermes T K

[hermes.tk_microchip.com]

HI @yuripv ,

The lockup issue is fixed with last patch set. Did you take chance to test? Could you please take a look/test and help us to commit.

Thanks & Regards
Hermes T K

[imp]

I've committed the 12.2 and 13.0 code reviews. If there's additional fixes to what's there, please open a new code review with those fixes. I think it's all in, but I'm happy to push additional fixes (the lockup fixes) if need be, but that will be easiest/fastest for me with a new review.

[hermes.tk_microchip.com]

We have tested this lockup fix in our local machine & it is working fine. But we need it to be tested & verified from your side also.
Once it is verified , we will push the lockup fix as next individual commit.

Regards
Hermes T K

[rainer_ultra-secure.de]

Hi,

does/should this apply against 12.2?

I now had this lockup issue, too.

Rainer

[hermes.tk_microchip.com]

Hi @rainer_ultra-secure.de ,

In freeBSD 13.0, when we ran IO with blocksize 1MB we are observing a corruption in SGL.
We were running IO with FIO (version 3.28)IO tool.
Command used :

fio --filename=/dev/da0: -direct=1 -iodepth=32 -ioengine=posixaio -rw=randrw -bs=1024k -numjobs=8 -runtime=30 -group_reporting -name=stress

Transfer size used in smartpqi driver:

In update_sim_properties FUNCTION
cpi->maxio = (softs->pqi_cap.max_sg_elem - 1) * PAGE_SIZE;
ie: Max transfer size: 0x21f000 (2.24 MB)

While running IO with 1mb block size , we are getting EINPROGRESS status for bus_dmamap_load_ccb in pqi_map_request function.

function stack => smartpqi_cam_action() -> pqisrc_io_start() -> pqi_map_request() -> bus_dmamap_load_ccb() -> pqi_request_map_helper

After receiving the EINPROGRESS status for bus_dmamap_load_ccb(), pqi_request_map_helper callback function is called .
Then we could see a corruption in the SGL received.

In pqi_map_request function,
bsd_status = bus_dmamap_load_ccb(softs->os_specific.pqi_buffer_dmat,

rcb->cm_datamap, ccb, pqi_request_map_helper, rcb, 0);

SGL dump(Added debug prints to dump sgl):
Eg:

05:30:29 freebsd13 kernel: [NOTE]:[ pqi_map_request ] [ 615 ]Total transfer size : 100000 (1.04MB)for tag value 37, Max transfer size: 21f000 (2.24 MB)
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqi_request_map_helper ] [ 577 ]Submitting IO With tag 30 after EINPROGRESS status[NOTE]:[ pqisrc_build_sgl ] [ 108 ]SGL_Count :256 for tag 30
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 131 ]SGL Chaining
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 140 ]TAG: 30 sgt->addr[0]: 5930000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 142 ]TAG: 30 sgt->len[0]: 1000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 140 ]TAG: 30 sgt->addr[1]: 592f000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 142 ]TAG: 30 sgt->len[1]: 1000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 140 ]TAG: 30 sgt->addr[2]: 592e000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 142 ]TAG: 30 sgt->len[2]: 1000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 140 ]TAG: 30 sgt->addr[3]: 592d000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 142 ]TAG: 30 sgt->len[3]: 1000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 140 ]TAG: 30 sgt->addr[4]: 592c000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 142 ]TAG: 30 sgt->len[4]: 1000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 140 ]TAG: 30 sgt->addr[5]: 592b000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 142 ]TAG: 30 sgt->len[5]: 1000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 140 ]TAG: 30 sgt->addr[6]: 592a000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 142 ]TAG: 30 sgt->len[6]: 1000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 140 ]TAG: 30 sgt->addr[7]: 5929000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 142 ]TAG: 30 sgt->len[7]: 1000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 140 ]TAG: 30 sgt->addr[8]: 5928000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 142 ]TAG: 30 sgt->len[8]: 1000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 140 ]TAG: 30 sgt->addr[9]: 5927000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 142 ]TAG: 30 sgt->len[9]: 1000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 140 ]TAG: 30 sgt->addr[10]: 5926000

• -

05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 142 ]TAG: 30 sgt->len[240]: 1000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 140 ]TAG: 30 sgt->addr[241]: 583f000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 142 ]TAG: 30 sgt->len[241]: 1000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 140 ]TAG: 30 sgt->addr[242]: 583e000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 142 ]TAG: 30 sgt->len[242]: 1000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 140 ]TAG: 30 sgt->addr[243]: 583d000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 142 ]TAG: 30 sgt->len[243]: 1000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 140 ]TAG: 30 sgt->addr[244]: 583c000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 142 ]TAG: 30 sgt->len[244]: 1000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 140 ]TAG: 30 sgt->addr[245]: 583b000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 142 ]TAG: 30 sgt->len[245]: 1000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 140 ]TAG: 30 sgt->addr[246]: 583a000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 142 ]TAG: 30 sgt->len[246]: 1000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 140 ]TAG: 30 sgt->addr[247]: 5839000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 142 ]TAG: 30 sgt->len[247]: 1000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 140 ]TAG: 30 sgt->addr[248]: 5838000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 142 ]TAG: 30 sgt->len[248]: 1000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 140 ]TAG: 30 sgt->addr[249]: 5837000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 142 ]TAG: 30 sgt->len[249]: 1000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 140 ]TAG: 30 sgt->addr[250]: 5836000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 142 ]TAG: 30 sgt->len[250]: 1000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 140 ]TAG: 30 sgt->addr[251]: 5835000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 142 ]TAG: 30 sgt->len[251]: 1000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 140 ]TAG: 30 sgt->addr[252]: 5834000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 142 ]TAG: 30 sgt->len[252]: 1000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 140 ]TAG: 30 sgt->addr[253]: 5833000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 142 ]TAG: 30 sgt->len[253]: 1000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 140 ]TAG: 30 sgt->addr[254]: 5832000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 142 ]TAG: 30 sgt->len[254]: 1000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 140 ]TAG: 30 sgt->addr[255]: 5831000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 142 ]TAG: 30 sgt->len[255]: 1000
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_build_sgl ] [ 147 ]Total sg length = 3002c0(3.14MB) for tag value 30
Oct 6 05:30:29 freebsd13 kernel: [NOTE]:[ pqisrc_show_cdb ] [ 1615 ]TAG: 30
Oct 6 05:30:29 freebsd13 kernel: [INFO]:[ pqisrc_print_buffer ] [ 1590 ] header:[ 00 01 02 03 04 05 06 07 08 09 ]
Oct 6 05:30:29 freebsd13 kernel: [INFO]:[ pqisrc_print_buffer ] [ 1607 ] AIOC:[ 28 00 37 19 88 00 00 08 00 00 ]

Here the total mapped SGL length is 0x3002c0 (3.14MB) & that is not equal to 0x100000 (1.04MB) .

When we tried in FreeBSD 12.2, maximum block size allowed to run in fio is 128k.
If we increase the block size we are getting the below error.

root@freebsd12:~ # fio --filename=/dev/da0: -direct=1 -iodepth=32 -ioengine=posixaio -rw=randrw -bs=129k -numjobs=8 -runtime=30 -group_reporting -name=stress
stress: (g=0): rw=randrw, bs=(R) 129KiB-129KiB, (W) 129KiB-129KiB, (T) 129KiB-129KiB, ioengine=posixaio, iodepth=32
...
fio-3.28
Starting 8 processes
fio: pid=1991, err=45/file:engines/posixaio.c:180, func=xfer, error=Operation not supported
fio: pid=1985, err=45/file:engines/posixaio.c:180, func=xfer, error=Operation not supported
fio: pid=1986, err=45/file:engines/posixaio.c:180, func=xfer, error=Operation not supported
fio: pid=1987, err=45/file:engines/posixaio.c:180, func=xfer, error=Operation not supported
fio: pid=1990, err=45/file:engines/posixaio.c:180, func=xfer, error=Operation not supported
fio: pid=1988, err=45/file:engines/posixaio.c:180, func=xfer, error=Operation not supported
fio: pid=1989, err=45/file:engines/posixaio.c:180, func=xfer, error=Operation not supported
fio: pid=1984, err=45/file:engines/posixaio.c:180, func=xfer, error=Operation not supported

stress: (groupid=0, jobs=8): err=45 (file:engines/posixaio.c:180, func=xfer, error=Operation not supported): pid=1984: Tue Oct 5 14:11:05 2021

cpu          : usr=16.13%, sys=22.58%, ctx=44, majf=0, minf=48
IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
   submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
   complete  : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
   issued rwts: total=2,6,0,0 short=0,0,0,0 dropped=0,0,0,0
   latency   : target=0, window=0, percentile=100.00%, depth=32

We are suspecting some issue in SGL handling with FreeBSD 13.0.
Can you please check & let us know your views.

Thanks & Regards
Hermes T K

[rainer_ultra-secure.de]

Hi,

sorry, I don't have 13.0 on any system with this HBA. Only 12.2

And I only got this single lockup over several months.

The other system with 6x8 disks has also been stable for over 60 days.

[imp]

As far as I know, the latest are in the relevant branches. If not, please let me know.

[imp]

Already committed as 4f77349d5777c8ac504387e7e6d37c13da222294