MP 7,1 Mac Pro 2019 NVME Raid drives failing to mount

[aventham]

Has it carried on in the same trouble-free vein for you @Iomahs?
I have the same PCIE drive setup as you and have just upgraded to the release version of 14.4.
Initially things looked good, but then I had a kernel panic on restart, and two more shortly after the desktop reappeared. After the first one, my drives went missing, but were present last time and are present again this time.
The info at the start of the "send a report" window referred to the keyboard driver the first time, but the last couple of times it has referred to the NVME driver:

Backtrace (CPU 0), panicked thread:
...
Kernel Extensions in backtrace:
com.apple.iokit.IONVMeFamily(2.1)

Mercifully, these kernel panics, which I'd also been getting with 14.3.1, still only seem to strike on shutdown or shortly (a minute or two) after startup; in between, everything seems to be stable.
I suppose it counts as an improvement with 14.4 (so far) that the only time the drives haven't shown up is after a mid-shutdown kernel panic!

OK, so I'm going to add a number 5 to my "menu" of mitigations -

5) unmount PCIE drives before shutting down or restarting

- and see if that (alone) is enough to avoid any kernel panics or failed auto-mounts, on 14.4.

 

[rhoulian]

@H2SO4

So once I ran that terminal command my statistics are thus;
Log - Default: 8,690, Info: 0, Debug: 919, Error: 2,460, Fault: 139
Activity - Create: 0, Transition: 0, Actions: 0

I typed the command (no verbose mode, all devices connected ) and I get a very long list. And I am not very familiar with what I can read on it… what kind of message should I look for?

Otherwise, still no issues with MP2019 since the NVRAM reset. I have switched of the mac every night since 3 days and no problem after rebooting …. So far….

 

[aventham]

@H2SO4

I typed the command (no verbose mode, all devices connected ) and I get a very long list. And I am not very familiar with what I can read on it… what kind of message should I look for?

Otherwise, still no issues with MP2019 since the NVRAM reset. I have switched of the mac every night since 3 days and no problem after rebooting …. So far….

Just to confirm - was that a single NVRAM reset, without an SMC reset, and then just regular booting after that? On 14.4?

 

[rhoulian]

Just to confirm - was that a single NVRAM reset, without an SMC reset, and then just regular booting after that? On 14.4?

Yes.
Prior to that I had tried smc but no success
The version is 14.4 (23E214)

 

[aventham]

Yes.
Prior to that I had tried smc but no success
The version is 14.4 (23E214)

OK - I guess it'd be rude of me not to try that next then. Thanks!

 

[rhoulian]

Yeah!
And as this was supposed to be a fix, I erased the apple assistant made Raid, and built the new one using soft raid.
Still no problem on reboot today.

EDIT: the user mentionning softraid as a fix has disappeared from the thread. Actually it has fixed nothing since I also had to install 14.4 and try NVRAM resets to (apparently) solve the problem. I hope he/she was not a troll working for OWC trying to sell some softraid licences 😊 No clue if this would be the same using Apple raid assistant

 

[H2SO4]

@H2SO4

I typed the command (no verbose mode, all devices connected ) and I get a very long list. And I am not very familiar with what I can read on it… what kind of message should I look for?

Otherwise, still no issues with MP2019 since the NVRAM reset. I have switched of the mac every night since 3 days and no problem after rebooting …. So far….

Export the data as a text file and import it into Excel, (or similar). Then filter and concentrate on the errors and faults is what I did.

 

[aventham]

Yeah!
And as this was supposed to be a fix, I erased the apple assistant made Raid, and built the new one using soft raid.
Still no problem on reboot today.

EDIT: the user mentionning softraid as a fix has disappeared from the thread. Actually it has fixed nothing since I also had to install 14.4 and try NVRAM resets to (apparently) solve the problem. I hope he/she was not a troll working for OWC trying to sell some softraid licences 😊 No clue if this would be the same using Apple raid assistant

The NVRAM reset didn't fix it for me, sadly. No RAID - just two SSDs on 1M2 cards.
The NVRAM-resetting boot itself and a subsequent restart were fine, but after a shutdown last night and another boot this morning my drives disappeared again.
And I've just tried the SMC+NVRAM reset - no joy with that either. It was "worse" in that the drives didn't appear straight after the resetting boot itself, but it's probably really just luck of the draw every time.
Verbose boot also doesn't do the trick, for me.
Unmounting the drives before shutdown still seems to be the only reliable way I have of avoiding kernel panics; it doesn't stop my drives going missing on startup, but so far on 14.4, whenever they do go missing, they reappear after one additional restart.

 

[aventham]

The NVRAM reset didn't fix it for me, sadly. No RAID - just two SSDs on 1M2 cards.
The NVRAM-resetting boot itself and a subsequent restart were fine, but after a shutdown last night and another boot this morning my drives disappeared again.
And I've just tried the SMC+NVRAM reset - no joy with that either. It was "worse" in that the drives didn't appear straight after the resetting boot itself, but it's probably really just luck of the draw every time.
Verbose boot also doesn't do the trick, for me.
Unmounting the drives before shutdown still seems to be the only reliable way I have of avoiding kernel panics; it doesn't stop my drives going missing on startup, but so far on 14.4, whenever they do go missing, they reappear after one additional restart.

@rhoulian would you be willing to share details of which cards you have installed into which slots?
Here's my list for comparison:
Two W5700X MPX modules in first four slots
"Farboko" 7-port USB 3.1 card in slot 5
Two OWC 1M2 cards in slots 6 and 7, with Crucial P5 2TB drives fitted
Standard IO card in slot 8 (with a Sabrent USB 2 hub plugged into one of the USB 3 ports)
All the third party stuff (with the possible exception of the SSDs!?) is advertised as Mac-compatible and seems to be working.
Also, do you have the same secure boot settings as me (full security but with booting from external disks allowed)?

 

[mattspace]

@rhoulian would you be willing to share details of which cards you have installed into which slots?
Here's my list for comparison:
Two W5700X MPX modules in first four slots
"Farboko" 7-port USB 3.1 card in slot 5
Two OWC 1M2 cards in slots 6 and 7, with Crucial P5 2TB drives fitted
Standard IO card in slot 8 (with a Sabrent USB 2 hub plugged into one of the USB 3 ports)
All the third party stuff (with the possible exception of the SSDs!?) is advertised as Mac-compatible and seems to be working.
Also, do you have the same secure boot settings as me (full security but with booting from external disks allowed)?

Curious, as a fellow dual W5700X user, what's your PCI lane budget looking like in expansion slot utility?

 

[aventham]

Curious, as a fellow dual W5700X user, what's your PCI lane budget looking like in expansion slot utility?

I'm not sure why it says "x0" against slot 2!?

 

[aventham]

My latest brainwave is to run the following Terminal command:

sudo trimforce disable

Since doing so I've rebooted several times without any kernel panics or missing drives.
I can also copy between the two PCIE drives significantly faster than before (~1.25 GB per second instead of ~1).
Of course, it's early days!
But it does seem plausible that the problem(s) could relate to TRIM operations, which I can well imagine might be set up to run at the sort of frequency at which I've been running into trouble (every second or third shutdown or startup, or after so many hours between boots).
I hadn't explicitly enabled or disabled trimforce previously though, so am not sure what state it was in before running the above. I thought it was supposed to be disabled by default, but I bought this machine second-hand - perhaps the setting survives the resetting/reinstallation of the OS? Or could there be a bug that causes it to misbehave until it's explicitly set one way or the other? Or could Apple be messing with the default setting in different OS versions?!

Then there's this OWC article, which although it relates to a problem they think is specific to the Mac Pro 2023, and which was supposedly fixed in MacOS 13.5, does show that third party TRIM support in MacOS has a bit of a chequered recent past:
NVMe Storage Issue with the new Mac Studio and Mac Pro with M2 Chips (owc.com)

 

[profdraper]

FWIW, I'd thought the problem resolved after 14.3.1 and then 14.4: many cold boots since then, no issues & with Sonnet 4xM2 mounting as it should ... until I installed a sw update for UAD plugins (and which requires Privacy & Permissions acknowledgement & restart). Then the problem comes back - seems to be temporarily fixed by SMC+NVRAM reset. And all of that does appear to be related to Apple's 'security' thing.

Speaking of UAD, new authorisation problems just appeared courtesy of 14.4:

https://help.uaudio.com/hc/en-us/articles/24492336117652-PACE-iLok-Issue-with-macOS-Sonoma-14-4-Update?__s=5j54cfmw5p0u40mx6oja

 

[aventham]

My latest brainwave is to run the following Terminal command:

sudo trimforce disable

Since doing so I've rebooted several times without any kernel panics or missing drives.
I can also copy between the two PCIE drives significantly faster than before (~1.25 GB per second instead of ~1).
Of course, it's early days!
But it does seem plausible that the problem(s) could relate to TRIM operations, which I can well imagine might be set up to run at the sort of frequency at which I've been running into trouble (every second or third shutdown or startup, or after so many hours between boots).
I hadn't explicitly enabled or disabled trimforce previously though, so am not sure what state it was in before running the above. I thought it was supposed to be disabled by default, but I bought this machine second-hand - perhaps the setting survives the resetting/reinstallation of the OS? Or could there be a bug that causes it to misbehave until it's explicitly set one way or the other? Or could Apple be messing with the default setting in different OS versions?!

Then there's this OWC article, which although it relates to a problem they think is specific to the Mac Pro 2023, and which was supposedly fixed in MacOS 13.5, does show that third party TRIM support in MacOS has a bit of a chequered recent past:
NVMe Storage Issue with the new Mac Studio and Mac Pro with M2 Chips (owc.com)

Update: I just had another kernel panic shortly after restarting, so I guess I'll have to go back to unmounting my drives before shutdown again; but I still haven't had them go missing since I disabled trimforce.

 

[rhoulian]

Hi guys,

My system is MacPro 2019
3,3 GHz Intel
AMD Radeon Pro W5500X 8 Go
128 Go 2933 MHz DDR4
Sonoma 14.4 (23E214)

I use only one PCI card (OWC Excelsior 4M2 / 4 SSD drives 2To each / Softraid RAID 0)

I also have a USB direct attached storage, never experienced the disconnection issue.

Still no problem this morning (I switch off the mac at each end of the day work)

Maybe the chances of getting this issue increase if you multiply the PCI cards ?

Thanks aventham for mentionning the possible the UAD issues. I use the 11.1.0 01-30-2024 (built 171666) witch seems to be the latest version with no problem (Logic X). The only weird thing is that when I hit space on an audio file yesterday, the preview was like a half tone up the original (seems like a sample frequency issue). A simple reboot of the UAD solved the problem... It may be a common compatibility problem between sonoma and UAD, but let's not develop about this in this thread

EDIT : for the booting options, I never touched anything so I guess it's default settings... I just don't know where I can see that information

 

[aventham]

Hi guys,

My system is MacPro 2019
3,3 GHz Intel
AMD Radeon Pro W5500X 8 Go
128 Go 2933 MHz DDR4
Sonoma 14.4 (23E214)

I use only one PCI card (OWC Excelsior 4M2 / 4 SSD drives 2To each / Softraid RAID 0)

I also have a USB direct attached storage, never experienced the disconnection issue.

Still no problem this morning (I switch off the mac at each end of the day work)

Maybe the chances of getting this issue increase if you multiply the PCI cards ?

Thanks aventham for mentionning the possible the UAD issues. I use the 11.1.0 01-30-2024 (built 171666) witch seems to be the latest version with no problem (Logic X). The only weird thing is that when I hit space on an audio file yesterday, the preview was like a half tone up the original (seems like a sample frequency issue). A simple reboot of the UAD solved the problem... It may be a common compatibility problem between sonoma and UAD, but let's not develop about this in this thread

EDIT : for the booting options, I never touched anything so I guess it's default settings... I just don't know where I can see that information

Thanks @rhoulian
Are you using OWC's own SSDs in your 4M2?
I think you have to boot into recovery mode to get to the secure boot settings, so don't worry on my account!

About Startup Security Utility on a Mac with the Apple T2 Security Chip – Apple Support (UK)

Use Startup Security Utility to make sure your Mac always starts up from your designated startup disk and always from a legitimate, trusted operating system.

support.apple.com

I don't think allowing booting from external drives made any difference for me - the improvement came from disabling trimforce, or so it seems.
But my system evidently still isn't 100% stable, and I think that's probably due to the Crucial SSDs not having the best drivers under MacOS.

 

[rhoulian]

Are you using OWC's own SSDs in your 4M2?

SSD's are samsung EVO plus

Still no problem this morning... starts to be quite satisfying !

 

[mattspace]

Update: I just had another kernel panic shortly after restarting, so I guess I'll have to go back to unmounting my drives before shutdown again; but I still haven't had them go missing since I disabled trimforce.

It may not make a difference, but I am curious if you could confirm whether you've set your machine to boot verbosely?

I've always run verbose boot when I could (for style reasons), since it does provide a practical look at what is being done when the system starts, and when it shuts down.

 

[bax2023]

Thanks @rhoulian
But my system evidently still isn't 100% stable, and I think that's probably due to the Crucial SSDs not having the best drivers under MacOS.

macOS does not need or have any drivers for HDD/SATA/SSD/NVMe drives. PCIe bus or Disk controller are taking care od recognizing and communicating with drives. You would need driver for 3rd party controllers such are HighPoint or Areca for example, but not for drives themselves.

 

[ZombiePhysicist]

macOS does not need or have any drivers for HDD/SATA/SSD/NVMe drives. PCIe bus or Disk controller are taking care od recognizing and communicating with drives. You would need driver for 3rd party controllers such are HighPoint or Areca for example, but not for drives themselves.

You don’t need drivers for high point cards to work. They offer them, I believe, to allow you to do some raid configurations etc, but you can use them with no drivers (which I do because no companies do a good job of updating and main ting drivers, and Apple seems to love breaking all drivers with every update).

 

[bax2023]

You don’t need drivers for high point cards to work. They offer them, I believe, to allow you to do some raid configurations etc, but you can use them with no drivers (which I do because no companies do a good job of updating and main ting drivers, and Apple seems to love breaking all drivers with every update).

You don't need drivers for some High Point models because there is controller chip on board for which macOS already has integrated drivers. If you would delete them, your card would not work. Also, manufacturers such is HighPoint may have updated drivers which macOS still does not have in current version.

Drivers....

Point here was - drives themselves do not need any drivers.

 

[aventham]

SSD's are samsung EVO plus

Still no problem this morning... starts to be quite satisfying !

Thanks - the 970 EVO Plus?

 

[aventham]

It may not make a difference, but I am curious if you could confirm whether you've set your machine to boot verbosely?

I've always run verbose boot when I could (for style reasons), since it does provide a practical look at what is being done when the system starts, and when it shuts down.

I tried it before disabling trimforce and it didn't stop the drives disappearing, but good thought - worth trying again to see whether it helps avoid the kernel panics, and if it doesn't there might be useful diagnostic info to glean from it at least.
However I'm pretty much set on swapping out my Crucial SSDs for Samsungs.
Sorry I phrased my "drivers" comment poorly, and further apologies if I'm doing so again! But my theory is that the Crucial SSDs just aren't fully compatible with the MacOS NVME driver. Crucial themselves apologise on their website for not offering any compatible RAM or storage upgrades for the Mac Pro 2019.

Memory RAM & SSD Upgrades | Crucial.com

Crucial Memory and SSD upgrades - 100% Compatibility Guaranteed for - FREE US Delivery.

www.crucial.com

 

[mattspace]

But my theory is that the Crucial SSDs just aren't fully compatible with the MacOS NVME driver. Crucial themselves apologise on their website for not offering any compatible RAM or storage upgrades for the Mac Pro 2019.

Memory RAM & SSD Upgrades | Crucial.com

Crucial Memory and SSD upgrades - 100% Compatibility Guaranteed for - FREE US Delivery.

www.crucial.com

There are warnings about Sabrient drives dying in various places online, and yet they're one of the brands recommended here in the guide, so it's possible the Crucials have problems. 🤷‍♂️

 

[aventham]

There are warnings about Sabrient drives dying in various places online, and yet they're one of the brands recommended here in the guide, so it's possible the Crucials have problems. 🤷‍♂️

I had a similar problem with the SSD upgrade I put in my Surface Pro 7+, where the system would be running fine, but then suddenly freeze and reboot. A clever fellow on a forum somewhere worked out a very specific set of power settings that essentially disabled enough energy efficency measures to allow the system to "keep up" with the drive! That may be all the more tricky on MacOS I guess and is certainly not something I would have much of a clue how to do off my own bat, so I think I'd rather just sidestep the problem, and "crossgrade" to some SSDs that the Mac is seemingly happier with.