MP 7,1 Mac Pro 2019 NVME Raid drives failing to mount

[Radium]

Sorry trying to understand.

Is the TLDR you need to wipe the boot drive clean and do a clean install of 14.4 and then you wont get mystery disappearing drives?

And is the other TLDR, even that nuke and pave 'solution' above, will only work for the 2023 and not the 2019?

What I'm trying to tell you is what I'm learning as I'm going the 2023 Mac Pro that came with 14.3.1 which was a refurbished never exhibited losing the drives it loads them right after login screen each drive would appear one at a time never disconnect or fail to load. The other 2023 Mac Pro which I purchased new had 13 something on it I can't remember and it was exhibiting from day one drive dropping they would always Mount but you get the warning that they were disconnected sometimes you go for weeks and then other times you couldn't go a day without shutting the machine off and rebooting. The 2023 new Mac Pro now has 14. 3.1 and it's doing the same thing the other 2023 Mac is now and not dropping the drives . As of now the 2019 Mac Pro with 14.4 I just did the update a few hours ago literally came up and lost two drives on its first boot up once again turned it off unplugged it turned it on notice there's no punctuations here LOL and finally the drives came back. So it's safe to say the 2019 Mac Pro is not being fixed as of yet my opinion is they're not going to fix it and we're all screwed I never get rid of my equipment as you can see I'm still using a 5.1 and I even have some g5s and G4S from the good old days. I've been in the professional music business for over 40 years I've seen things like this come and go just like the adoption of firewire and then how quickly they left it behind this will just be another case of Apple being Apple I hope I'm wrong take care

 

[profdraper]

Apple is highly active in the security and protection of individuals hard drives there are a lot of factors that influence how a drive interacts with Mac OS?? I basically believe what he's telling me is we're pretty much going to be out of luck in the next few Renditions of Mac Pro as they keep locking everything down to their own ssds and not allowing us to easily upgrade to large fast storage telling us that Thunderbolt is good enough is an insult to anyone that needs to move 57 gig projects every 10 minutes right now with PCI 4 the Mac will move a 1.8 TB information in under 2.6 minutes that's what I need

Sounds about exaclrty in my experience ... these days Apple are pretty much clueless unless it applies to their overpriced fashion devices. They do exaclty this, continuously, & break everything 3rd party. FWIW, OS Monterey works perfectly in this respect.

 

[profdraper]

Oh, well...

After a week or so of mostly all 8 Samsung SSDs/two Sonnet cards mounting every startup, I updated to Sonoma 14.4

On the next startup only seven of the SSDs mounted, then two of those took turns dismounting and disappearing with the usual error messages about dismounting improperly.

At this moment there are five SSDs mounted but I'm expecting one of those to dismount any time now. The other four are on a different card and will likely remain mounted.

As of now I'd say Sonoma 14.4 is either a step backward or at least no better.

It does seem that Apple has chosen to avoid spending time updating anything to do with Intel. If that continues there'll be one more (me!) returning to Windows after a five year detour into Mac-Land.

And that's the name of that tune,

Tom

Roll back to Monterey? That works. Then lock down your machine and don't ever add any more Apple apps, updates, drivers of anything else ... after all, we did our work just as well as last year and this year with this garbage.

 

[profdraper]

If this is some kind of security crackdown, they're doing a pretty half-assed job of it, given it only prevents the PCIE drives appearing every other boot or so. What's the extra security risk of PCIE drives vs Thunderbolt or SATA drives supposed to be anyway?

Exactly. Apple just make this ***** up. When some fan boy says 'but Apple are so much more secure than everything else', ask them where they got that information and the only answer is 'Apple'. Influencers & bs artists ... check elsewhere in the realworld with research, white papers etc and the answers are quite different; pretty much the same as everywhere else & everyone has them share of ongoing security concerns and new exploits.

Applejust do this for their own benefit & sharholders above customers.

 

[profdraper]

Thank you Radium, for the post and those 2 marks of punctuation 😊
Have you tried to reset the nvram ?

For those who know this trick, can it be done at each boot without damaging or compromising anything?

Since you guys mentionned some security related issues during the boot sequence, I have found some leads :

Disabling System Integrity Protection​

As mentioned in the glossary, SIP functions as a method of system protection. Apple describes it as follows:


Before Apple implemented SIP, any software that was granted root access (by the user entering her/his password) could modify/edit system files. Generally, a user shouldn't disable SIP unless there's a specific reason. That said, there are plenty of reasons to disable SIP, such as certain boot managers or for unsupported hardware cases. SIP can always be re-enabled.

Disabling Secure Boot​

Disabling Secure Boot on the 2019 Mac Pros is generally a common behavior as the only volume the Mac Pro will boot off of is the factory SSD if you do not enable the SSD. You cannot remove the Apple-provided SSD(s) and still boot regardless if Secure boot is on or off.


What do you think? Could there be anything useful here?

Nope, been there done that.

 

[aventham]

What I'm trying to tell you is what I'm learning as I'm going the 2023 Mac Pro that came with 14.3.1 which was a refurbished never exhibited losing the drives it loads them right after login screen each drive would appear one at a time never disconnect or fail to load. The other 2023 Mac Pro which I purchased new had 13 something on it I can't remember and it was exhibiting from day one drive dropping they would always Mount but you get the warning that they were disconnected sometimes you go for weeks and then other times you couldn't go a day without shutting the machine off and rebooting. The 2023 new Mac Pro now has 14. 3.1 and it's doing the same thing the other 2023 Mac is now and not dropping the drives . As of now the 2019 Mac Pro with 14.4 I just did the update a few hours ago literally came up and lost two drives on its first boot up once again turned it off unplugged it turned it on notice there's no punctuations here LOL and finally the drives came back. So it's safe to say the 2019 Mac Pro is not being fixed as of yet my opinion is they're not going to fix it and we're all screwed I never get rid of my equipment as you can see I'm still using a 5.1 and I even have some g5s and G4S from the good old days. I've been in the professional music business for over 40 years I've seen things like this come and go just like the adoption of firewire and then how quickly they left it behind this will just be another case of Apple being Apple I hope I'm wrong take care

Still a glimmer of hope though, if you haven't yet tried a clean install of 14.4 on the 2019?!

 

[Radium]

I'm literally doing it right now just wanted to make sure I backed up my original Drive. This is just pissing me off to no end I love my Intel Mac. I have a feeling though it's nothing to do with the clean install it's simply not going to work until Apple addresses the t2 chip and the way it allows the drive to be mounted on the desktop. One other quick question has anyone ever noticed that they can eject the drives on their Mac Pro 2019 I didn't think it was possible? Apparently now on the 2023 I can eject them at any time and then remount them I never knew that was an option in 2019 because they were PCI mounted drives just wondering.

 

[rhoulian]

Exactly. Apple just make this ***** up. When some fan boy says 'but Apple are so much more secure than everything else', ask them where they got that information and the only answer is 'Apple'. Influencers & bs artists ... check elsewhere in the realworld with research, white papers etc and the answers are quite different; pretty much the same as everywhere else & everyone has them share of ongoing security concerns and new exploits.

Applejust do this for their own benefit & sharholders above customers.

I can’t believe that. I mean ok, Apple has pissed me off when I understood I had to buy adaptors (preferably theirs) to plug TB1 or TB2 or even USB in their new machines. I had to buy those, but it was worth it : even after 10years I have been using my old firewire audio interface on new macs with no problem so far.

But making the 2019 MP and be proud to say « hey you were looking for an evolutive machine that you will keep many years, able to accept plenty of PCI cards ? Ok the price is 2 times the cylinder model was, but the MP2019 is for you! »

…. And they would **** us up just 1year after that ???? No I am sorry it seems impossible, or maybe I am too naive

 

[H2SO4]

I'm literally doing it right now just wanted to make sure I backed up my original Drive. This is just pissing me off to no end I love my Intel Mac. I have a feeling though it's nothing to do with the clean install it's simply not going to work until Apple addresses the t2 chip and the way it allows the drive to be mounted on the desktop. One other quick question has anyone ever noticed that they can eject the drives on their Mac Pro 2019 I didn't think it was possible? Apparently now on the 2023 I can eject them at any time and then remount them I never knew that was an option in 2019 because they were PCI mounted drives just wondering.

I think you can do all that disk ejection prior to the 19.

 

[H2SO4]

For those wanting to look at the boot process itself as a way of debugging I found a lovely command on stack exchange.
It posts the whole days messages all nicely arranged in columns and colour coded;

log show --predicate "processID == 0" --start $(date "+%Y-%m-%d") --debug

On the back of this I decided to switch off the Mac overnight to naturally reduce the size of the log.
I plugged everything back in so the unit was as per my day to day use case, (3080 installed, a few USB devices), and then booted.
I've noticed that when I verbose boot I get everything connecting every time, and no panics or prohibited symbols mid boot - not sure why.
So once I ran that terminal command my statistics are thus;
Log - Default: 8,690, Info: 0, Debug: 919, Error: 2,460, Fault: 139
Activity - Create: 0, Transition: 0, Actions: 0

I'm going to pop it into a spread sheet and filter to see what's most common.
The first faults/errors just at a cursory glance are USB related;

Fault 0x0 0 0 kernel: (IOUSBHostFamily) [com.apple.usb:all] SS06@14700000: AppleUSBHostPort::registerCompanionPortGated: cannot register HS06@14300000 for speed 2 since a companion port HS02@14100000 is already registered

Error 0x0 0 0 kernel: (IOUSBFamily) [com.apple.usb:all] AppleUSBLegacyRoot@(null): AppleUSBLegacyRoot::usbServiceCallback: controller <private> (XHC1) usbServiceArray <private>(count 1) options 0x00000000

It might be worth someone else alse seeing if they have similar results?

The majority of faults contain, "oid *, IOFramebuffer *, IOIndex, void *) event = <1> at <43261291744>", which suggests to me that the 3080 might be related.

 

[rhoulian]

I've noticed that when I verbose boot I get everything connecting every time, and no panics or prohibited symbols mid boot not sure why.

I'll try this if the NVRAM reset on startup is not 100% successful. As someone said there, better have a solution, even if annoying, than nothing

It might be worth someone else alse seeing if they have similar results?

I will test this, definitely !

 

[Radium]

Well the results after wiping out 700 gigs of information and using the internal drive to make sure that it has no excuse as the operating system not to see the other drives. I failed to mention on my 2019 Mac Pro I have two sonnet cards one is a 32 TB. The other one is 3/4 TB Drive as a raid and one 2 TB Drive as a booting disc. I carbon clone to that disc all the time and back up my entire system so I never have to worry. It's been working perfect for years until recently after os 13 the non-loading HD was rare, now it's just impossible to get the machine to ever boot up reliably. The funny part when I boot off that drive with the os , it always mounts the other disc is hit or miss. However at this point I'm just going to conclude that 14.3.1 resolve the issue for the Mac Pros of 2023 after the machines boot up the drives are loaded one at a time every time they never drop out during work time all is well including 14.4 both machines . Now as far as the 2019 7.1 were still in trouble after the first boot up of the fresh install it lost the 32 TB Drive OWC and also the sonnet 16 TB it makes no sense I guess I'm glad I purchased the 2023s that my partner yelled at me we should have waited and just dealt with the 2019 constant drive failure. I have nothing more too add I'm a musician first a producer second and very old and tired thank you for dealing with my punctuation. I have nothing else I can offer. The engineer at sonnet thank me for my time and the value that I had so many Mac Pros on hand which is a rarity to do so much random testing . At this point it all points back to the 2019 T2 chip which controls security something has to be fixed on Apple side. Once again after the engineers discussion on Friday 36 minute conversation on the phone it left me feeling sorry for them and us too. That's all I can say take care I'll be watching The Forum hopefully there's something else I could try.

 

[aventham]

Appreciate your persistence in giving it a thorough try anyway, and reporting back!
There are a few things left on the "menu":
1) NVRAM or SMC+NVRAM resets
2) card order - just my pet theory, that if there's a problem with the power management or the timing/threading of the boot sequence between CPU and T2, then it might make a difference to have the storage cards in "lower-numbered" slots than other third party (e. g. USB) cards
3) secure boot settings - worth a re-test at 14.4 perhaps, even if they haven't done the trick with earlier versions https://support.apple.com/en-gb/102522

 

[ZombiePhysicist]

On the back of this I decided to switch off the Mac overnight to naturally reduce the size of the log.
I plugged everything back in so the unit was as per my day to day use case, (3080 installed, a few USB devices), and then booted.
I've noticed that when I verbose boot I get everything connecting every time, and no panics or prohibited symbols mid boot - not sure why.
So once I ran that terminal command my statistics are thus;
Log - Default: 8,690, Info: 0, Debug: 919, Error: 2,460, Fault: 139
Activity - Create: 0, Transition: 0, Actions: 0

I'm going to pop it into a spread sheet and filter to see what's most common.
The first faults/errors just at a cursory glance are USB related;

Fault 0x0 0 0 kernel: (IOUSBHostFamily) [com.apple.usb:all] SS06@14700000: AppleUSBHostPort::registerCompanionPortGated: cannot register HS06@14300000 for speed 2 since a companion port HS02@14100000 is already registered

Error 0x0 0 0 kernel: (IOUSBFamily) [com.apple.usb:all] AppleUSBLegacyRoot@(null): AppleUSBLegacyRoot::usbServiceCallback: controller <private> (XHC1) usbServiceArray <private>(count 1) options 0x00000000

It might be worth someone else alse seeing if they have similar results?

The majority of faults contain, "oid *, IOFramebuffer *, IOIndex, void *) event = <1> at <43261291744>", which suggests to me that the 3080 might be related.

So just verbose booting solves the problem on the 7,1?!

 

[CanPhantom]

So just verbose booting solves the problem on the 7,1?!

That actually makes some sense if the problem is caused by a timing issue.

 

[mattspace]

So just verbose booting solves the problem on the 7,1?!

Yes, because Apple's new feature is Heisenberg Boot, where you can only know if your drives are going to connect if you're observing the boot process directly.

 

[H2SO4]

So just verbose booting solves the problem on the 7,1?!

lol, I wish.
All I can say is that everytime I've gone the command V route so far it's been good. I googled around a bit and it seems that the system process is evvvvvvver so slightly different when you go verbose.
Probably a red herring but I'll keep doing that for now.
I have to enable a parallels extension so I can use WIndows quite soon. So that'll be my next restart. Hopefully I'll be goo still.

 

[mattspace]

lol, I wish.
All I can say is that everytime I've gone the command V route so far it's been good. I googled around a bit and it seems that the system process is evvvvvvver so slightly different when you go verbose.
Probably a red herring but I'll keep doing that for now.
I have to enable a parallels extension so I can use WIndows quite soon. So that'll be my next restart. Hopefully I'll be goo still.

Tried setting verbose mode as a permanent thing?

 

[H2SO4]

Tried setting verbose mode as a permanent thing?

Not yet. When I'm not regularly on site I do a lot of booting to different OS, it's only a 5-10 second key press each time so it's not a bind to do it this way.
I have software that won't run on Sonoma, (or even Mac OS in some cases)..

 

[aventham]

Yes, sorry - verbose boot should have been number 4 (or maybe number 1) on my "menu" above.
Here's the command for anyone who wants to set it to happen automatically:

How to use verbose mode on Mac

Verbose mode allows you to watch extensions and other items load as your Mac boots, making it possible to identify problems. Here’s how to use it.

macpaw.com

I suppose anything that alters the sequence or timing of the normal boot procedure *could* have an improving effect on whatever is otherwise going awry!

 

[mattspace]

Yes, sorry - verbose boot should have been number 4 (or maybe number 1) on my "menu" above.
Here's the command for anyone who wants to set it to happen automatically:

How to use verbose mode on Mac

Verbose mode allows you to watch extensions and other items load as your Mac boots, making it possible to identify problems. Here’s how to use it.

macpaw.com

I suppose anything that alters the sequence or timing of the normal boot procedure *could* have an improving effect on whatever is otherwise going awry!

Tinkertool System also lets you set that. I'd assume it's just a front end for changing the boot args.

 

[H2SO4]

Tinkertool System also lets you set that. I'd assume it's just a front end for changing the boot args.

No doubt.
Just done another V-boot, all good. The system will likely stay up now till next Friday so I'll be keeping an eye.

 

[H2SO4]

No doubt.
Just done another V-boot, all good. The system will likely stay up now till next Friday so I'll be keeping an eye.

Just thought.
I have 4 Mac OS and one windows drive installed.
The built in SSD is a slightly out of date Sonoma.
One is Win10.
One is an up to date live Sonoma.
One is a clone of the above.
Also an old Mac OS.
So. No one has mentioned yet but it’s never the Kernel panic, ie the boot drive crashing.
Only difference is one is blessed.

 

[macguru9999]

Yes, because Apple's new feature is Heisenberg Boot, where you can only know if your drives are going to connect if you're observing the boot process directly.

Is there a black cat involved ? Oh wait, that was Schrodinger ......
I noticed the problem and went back to Monterey (solved), but that was not an issue its just a spare machine for me at this point.

 

[aventham]

Since 14.4 beta 4: all SSDs "show up and stay" (no raid, two PCIe cards with one SSD per card)

Has it carried on in the same trouble-free vein for you @Iomahs?
I have the same PCIE drive setup as you and have just upgraded to the release version of 14.4.
Initially things looked good, but then I had a kernel panic on restart, and two more shortly after the desktop reappeared. After the first one, my drives went missing, but were present last time and are present again this time.
The info at the start of the "send a report" window referred to the keyboard driver the first time, but the last couple of times it has referred to the NVME driver:

Backtrace (CPU 0), panicked thread:
...
Kernel Extensions in backtrace:
com.apple.iokit.IONVMeFamily(2.1)

Mercifully, these kernel panics, which I'd also been getting with 14.3.1, still only seem to strike on shutdown or shortly (a minute or two) after startup; in between, everything seems to be stable.
I suppose it counts as an improvement with 14.4 (so far) that the only time the drives haven't shown up is after a mid-shutdown kernel panic!