MP 6,1 2013 Mac Pro Restoration - Kernel Panic

[TheValleyOfDefilement]

Greetings. I'm currently restoring a 2013 MP. Having a few general issues, but can boot from seeded external boot medium in safe mode.
With regular boots the system crashes down in a loop.
I've done a lot of digging and can make some assumptions, but looking for some opinions and insights.

Here's the relevant output. Thanks in advance!


Sun Oct 3 08:51:55 2021

*** Panic Report ***
panic(cpu 0 caller 0xffffff80028dab9d): Kernel trap at 0xffffff7f85c5eafe, type 14=page fault, registers:
CR0: 0x000000008001003b, CR2: 0xffffff9121afb000, CR3: 0x0000000015412000, CR4: 0x00000000001626e0
RAX: 0xffffff9121afb000, RBX: 0x0000000000005434, RCX: 0x0000000002000000, RDX: 0x0000000003000000
RSP: 0xffffff9121aebbe0, RBP: 0xffffff9121aebc00, RSI: 0xffffff8028f93350, RDI: 0x0000000000000000
R8: 0x0000000000000002, R9: 0x0000000000000002, R10: 0x00000000e0000000, R11: 0xffffff8028f9335a
R12: 0xffffff8024a7ae80, R13: 0xffffff80e37d0000, R14: 0x0000000000000000, R15: 0xffffff80245b5690
RFL: 0x0000000000010246, RIP: 0xffffff7f85c5eafe, CS: 0x0000000000000008, SS: 0x0000000000000010
Fault CR2: 0xffffff9121afb000, Error code: 0x0000000000000002, Fault CPU: 0x0, PL: 0, VF: 1

Backtrace (CPU 0), Frame : Return Address
0xffffff9121aeb6b0 : 0xffffff80027ad5cd
0xffffff9121aeb700 : 0xffffff80028e9245
0xffffff9121aeb740 : 0xffffff80028da97a
0xffffff9121aeb7b0 : 0xffffff800275a9d0
0xffffff9121aeb7d0 : 0xffffff80027acfe7
0xffffff9121aeb8f0 : 0xffffff80027ace33
0xffffff9121aeb960 : 0xffffff80028dab9d
0xffffff9121aebad0 : 0xffffff800275a9d0
0xffffff9121aebaf0 : 0xffffff7f85c5eafe
0xffffff9121aebc00 : 0xffffff7f85c6e03b
0xffffff9121aebc20 : 0xffffff7f85c5e9e8
0xffffff9121aebc60 : 0xffffff7f85c7f0d3
0xffffff9121aebcb0 : 0xffffff7f85c6de0c
0xffffff9121aebcd0 : 0xffffff7f85c0573c
0xffffff9121aebd20 : 0xffffff7f85c03726
0xffffff9121aebd60 : 0xffffff7f85b898e8
0xffffff9121aebde0 : 0xffffff7f85c01d6e
0xffffff9121aebe40 : 0xffffff8002e2df6b
0xffffff9121aebe80 : 0xffffff8002e2dc25
0xffffff9121aebf00 : 0xffffff8002e2d1e7
0xffffff9121aebf50 : 0xffffff8002e2ecd6
0xffffff9121aebfa0 : 0xffffff800275a0ce
Kernel Extensions in backtrace:
com.apple.iokit.IOAcceleratorFamily2(404.15)[54AC3DDA-1C4B-3884-80BA-9D9E87DDC370]@0xffffff7f85b5b000->0xffffff7f85c00fff
dependency: com.apple.driver.AppleMobileFileIntegrity(1.0.5)[117C5BD8-7DBD-3563-9B28-87E7278B379A]@0xffffff7f83522000
dependency: com.apple.iokit.IOSurface(255.6.1)[0216A8C3-8E7C-358A-BEC5-4D103C421C39]@0xffffff7f859ea000
dependency: com.apple.iokit.IOPCIFamily(2.9)[CE5E5806-48AB-39C0-8C1F-B57921B37D71]@0xffffff7f83095000
dependency: com.apple.iokit.IOGraphicsFamily(530.68)[35DF864A-81F1-30E7-AFA2-7DB2B37B6E1D]@0xffffff7f853f1000
dependency: com.apple.iokit.IOReportFamily(47)[4A62301D-0778-317C-91DA-56E63FF1D1AD]@0xffffff7f83ac4000
com.apple.kext.AMDRadeonX4000(2.1.1)[FDF6F8FD-03A0-3443-BF89-A3AB1E7C80F2]@0xffffff7f85c01000->0xffffff7f86040fff
dependency: com.apple.iokit.IOSurface(255.6.1)[0216A8C3-8E7C-358A-BEC5-4D103C421C39]@0xffffff7f859ea000
dependency: com.apple.iokit.IOPCIFamily(2.9)[CE5E5806-48AB-39C0-8C1F-B57921B37D71]@0xffffff7f83095000
dependency: com.apple.iokit.IOGraphicsFamily(530.68)[35DF864A-81F1-30E7-AFA2-7DB2B37B6E1D]@0xffffff7f853f1000
dependency: com.apple.iokit.IOAcceleratorFamily2(404.15)[54AC3DDA-1C4B-3884-80BA-9D9E87DDC370]@0xffffff7f85b5b000

BSD process name corresponding to current thread: kernel_task

Mac OS version:
18G9323

Kernel version:
Darwin Kernel Version 18.7.0: Tue Jun 22 19:37:08 PDT 2021; root:xnu-4903.278.70~1/RELEASE_X86_64
Kernel UUID: 041B6A6D-CD16-36FA-88B9-E32FF46EF89F
Kernel slide: 0x0000000002400000
Kernel text base: 0xffffff8002600000
__HIB text base: 0xffffff8002500000
System model name: MacPro6,1 (Mac-F60DEB81FF30ACF6)

System uptime in nanoseconds: 72597103006
last loaded kext at 72589859226: com.apple.kext.AMDRadeonX4030HWLibs 1.0 (addr 0xffffff7f86041000, size 7655424)
loaded kexts:
com.apple.kext.AMDRadeonX4000 2.1.1
com.apple.driver.AppleGraphicsDevicePolicy 3.50.16
com.apple.AGDCPluginDisplayMetrics 3.50.16
com.apple.driver.AppleHV 1
com.apple.iokit.IOUserEthernet 1.0.1
com.apple.driver.AudioAUUC 1.70
com.apple.fileutil 20.036.15
com.apple.iokit.IOBluetoothSerialManager 6.0.14d13
com.apple.driver.ApplePlatformEnabler 2.7.0d0
com.apple.driver.AGPM 110.25.11
com.apple.driver.X86PlatformShim 1.0.0
com.apple.driver.pmtelemetry 1
com.apple.driver.AppleHDAHardwareConfigDriver 282.54
com.apple.Dont_Steal_Mac_OS_X 7.0.0
com.apple.driver.AppleHDA 282.54
com.apple.driver.eficheck 1
com.apple.driver.ACPI_SMC_PlatformPlugin 1.0.0
com.apple.driver.AppleThunderboltIP 3.1.2
com.apple.iokit.IOBluetoothUSBDFU 6.0.14d13
com.apple.kext.AMD7000Controller 2.1.1
com.apple.driver.AppleFIVRDriver 4.1.0
com.apple.driver.AppleOSXWatchdog 1
com.apple.driver.AppleMGPUPowerControl 3.50.16
com.apple.iokit.AppleBCM5701Ethernet 10.3.3
com.apple.driver.AppleLPC 3.1
com.apple.driver.AirPort.BrcmNIC 1400.1.1
com.apple.driver.AppleIntelMCEReporter 107
com.apple.driver.AppleIntelSlowAdaptiveClocking 4.0.0
com.apple.filesystems.autofs 3.0
com.apple.filesystems.apfs 945.275.10
com.apple.iokit.SCSITaskUserClient 408.250.3
com.apple.driver.AppleFileSystemDriver 3.0.1
com.apple.driver.AppleVirtIO 2.1.3
com.apple.filesystems.hfs.kext 407.200.4
com.apple.AppleFSCompression.AppleFSCompressionTypeDataless 1.0.0d1
com.apple.BootCache 40
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib 1.0.0
com.apple.AppleSystemPolicy 1.0
com.apple.driver.AppleAHCIPort 329.260.5
com.apple.private.KextAudit 1.0
com.apple.driver.AppleACPIEC 6.1
com.apple.driver.AppleACPIButtons 6.1
com.apple.driver.AppleRTC 2.0
com.apple.driver.AppleHPET 1.8
com.apple.driver.AppleSMBIOS 2.1
com.apple.driver.AppleAPIC 1.7
com.apple.driver.AppleIntelCPUPowerManagementClient 220.0.0
com.apple.nke.applicationfirewall 202
com.apple.security.TMSafetyNet 8
com.apple.driver.AppleIntelCPUPowerManagement 220.0.0
com.apple.kext.AMDRadeonX4030HWLibs 1.0
com.apple.iokit.IOAcceleratorFamily2 404.15
com.apple.kext.AMDRadeonX4000HWServices 2.1.1
com.apple.iokit.IOAVBFamily 760.6
com.apple.plugin.IOgPTPPlugin 740.2
com.apple.iokit.IOSkywalkFamily 1
com.apple.driver.AppleSSE 1.0
com.apple.iokit.IOSurface 255.6.1
com.apple.driver.DspFuncLib 282.54
com.apple.kext.OSvKernDSPLib 528
com.apple.driver.IOPlatformPluginLegacy 1.0.0
com.apple.AppleGPUWrangler 3.50.16
com.apple.kext.AMDSupport 2.1.1
com.apple.AppleGraphicsDeviceControl 3.50.16
com.apple.driver.AppleGraphicsControl 3.50.16
com.apple.driver.AppleBacklightExpert 1.1.0
com.apple.iokit.IONDRVSupport 530.51
com.apple.driver.AppleUSBAudio 315.6
com.apple.driver.AppleHDAController 282.54
com.apple.iokit.IOGraphicsFamily 530.68
com.apple.iokit.IOHDAFamily 282.54
com.apple.iokit.IOAudioFamily 206.5
com.apple.vecLib.kext 1.2.0
com.apple.iokit.IOEthernetAVBController 1.1.0
com.apple.driver.AppleSMBusPCI 1.0.14d1
com.apple.iokit.IO80211Family 1200.12.2
com.apple.driver.mDNSOffloadUserClient 1.0.1b8
com.apple.driver.corecapture 1.0.4
com.apple.driver.X86PlatformPlugin 1.0.0
com.apple.driver.IOPlatformPluginFamily 6.0.0d8
com.apple.iokit.IOSlowAdaptiveClockingFamily 1.0.0
com.apple.driver.AppleThunderboltEDMSink 4.2.0
com.apple.driver.AppleThunderboltDPOutAdapter 5.6.9
com.apple.kext.triggers 1.0
com.apple.iokit.IOUSBHIDDriver 900.4.2
com.apple.driver.AppleXsanScheme 3
com.apple.iokit.BroadcomBluetoothHostControllerUSBTransport 6.0.14d13
com.apple.iokit.IOBluetoothHostControllerUSBTransport 6.0.14d13
com.apple.iokit.IOBluetoothHostControllerTransport 6.0.14d13
com.apple.iokit.IOBluetoothFamily 6.0.14d13
com.apple.driver.AppleHIDKeyboard 208
com.apple.driver.usb.IOUSBHostHIDDevice 1.2
com.apple.driver.usb.AppleUSBHub 1.2
com.apple.driver.usb.cdc 5.0.0
com.apple.driver.usb.networking 5.0.0
com.apple.driver.usb.AppleUSBHostCompositeDevice 1.2
com.apple.iokit.IOSerialFamily 11
com.apple.filesystems.hfs.encodings.kext 1
com.apple.driver.AppleThunderboltDPInAdapter 5.6.9
com.apple.driver.AppleThunderboltDPAdapterFamily 5.6.9
com.apple.driver.AppleThunderboltPCIDownAdapter 2.1.5
com.apple.driver.AppleThunderboltNHI 4.7.9
com.apple.iokit.IOThunderboltFamily 6.8.7
com.apple.driver.AppleUSBMergeNub 900.4.2
com.apple.driver.usb.AppleUSBXHCIPCI 1.2
com.apple.driver.usb.AppleUSBXHCI 1.2
com.apple.iokit.IOAHCIFamily 288
com.apple.driver.usb.AppleUSBEHCIPCI 1.2
com.apple.driver.usb.AppleUSBEHCI 1.2
com.apple.driver.usb.AppleUSBHostPacketFilter 1.0
com.apple.iokit.IOUSBFamily 900.4.2
com.apple.driver.AppleUSBHostMergeProperties 1.2
com.apple.driver.AppleEFINVRAM 2.1
com.apple.iokit.IOSMBusFamily 1.1
com.apple.driver.AppleEFIRuntime 2.1
com.apple.iokit.IOHIDFamily 2.0.0
com.apple.security.quarantine 3
com.apple.security.sandbox 300.0
com.apple.kext.AppleMatch 1.0.0d1
com.apple.driver.DiskImages 493.0.0
com.apple.driver.AppleFDEKeyStore 28.30
com.apple.driver.AppleEffaceableStorage 1.0
com.apple.driver.AppleKeyStore 2
com.apple.driver.AppleUSBTDM 456.260.3
com.apple.driver.AppleMobileFileIntegrity 1.0.5
com.apple.iokit.IOUSBMassStorageDriver 145.200.2
com.apple.iokit.IOSCSIBlockCommandsDevice 408.250.3
com.apple.iokit.IOSCSIArchitectureModelFamily 408.250.3
com.apple.iokit.IOStorageFamily 2.1
com.apple.kext.CoreTrust 1
com.apple.driver.AppleCredentialManager 1.0
com.apple.driver.KernelRelayHost 1
com.apple.iokit.IOUSBHostFamily 1.2
com.apple.driver.usb.AppleUSBCommon 1.0
com.apple.driver.AppleBusPowerController 1.0
com.apple.driver.AppleSEPManager 1.0.1
com.apple.driver.IOSlaveProcessor 1
com.apple.iokit.IOTimeSyncFamily 740.2
com.apple.iokit.IONetworkingFamily 3.4
com.apple.iokit.IOReportFamily 47
com.apple.driver.AppleACPIPlatform 6.1
com.apple.driver.AppleSMC 3.1.9
com.apple.iokit.IOPCIFamily 2.9
com.apple.iokit.IOACPIFamily 1.4
com.apple.kec.pthread 1
com.apple.kec.Libm 1
com.apple.kec.corecrypto 1.0

EOF
Model: MacPro6,1, BootROM 129.0.0.0.0, 4 processors, Quad-Core Intel Xeon E5, 3.7 GHz, 8 GB, SMC 2.20f18
Graphics: kHW_AMDFireProD500Item, AMD FirePro D500, spdisplays_pcie_device, 3 GB
Graphics: kHW_AMDFireProD500Item, AMD FirePro D500, spdisplays_pcie_device, 3 GB
Memory Module: DIMM1, 4 GB, DDR3 ECC, 1866 MHz, 0x802C, 0x394A53463531323732415A2D314739453220
Memory Module: DIMM4, 4 GB, DDR3 ECC, 1866 MHz, 0x80AD, 0x484D54343531553741465238432D52442020
AirPort: spairport_wireless_card_type_airport_extreme (0x14E4, 0x135), Broadcom BCM43xx 1.0 (7.77.61.3 AirPortDriverBrcmNIC-1305.12)
Bluetooth: Version 6.0.14d13, 3 services, 27 devices, 0 incoming serial ports
Network Service: Wi-Fi, AirPort, en2
PCI Card: AMD FirePro D500, Display Controller, Slot-2
PCI Card: AMD FirePro D500, ATY,KaniParent, Slot-1
USB Device: USB 2.0 Bus
USB Device: Hub
USB Device: BRCM20702 Hub
USB Device: Bluetooth USB Host Controller
USB Device: USB 3.0 Bus
USB Device: External USB 3.0
USB Device: USB Receiver
USB Device: Apple Keyboard
Thunderbolt Bus: Mac Pro, Apple Inc., 19.2
Thunderbolt Bus: Mac Pro, Apple Inc., 19.2
Thunderbolt Bus: Mac Pro, Apple Inc., 19.2

 

[TheValleyOfDefilement]

Additional Information:
Running Mojave 10.14.6
Unable to get system to boot from internal ssd at this time, FW isn't updating.
Boot ROM Version: 129.0.0.0.0
SMC Version (system): 2.20f18
Illumination Version: 1.4a6

 

[MisterAndrew]

I don't think very many people have experience restoring a 6,1 from a non-functional state. At minimum you'll need the appropriate Apple Service Diagnostics. Your best option is probably to take it to the Apple store and have them fix it for you.

 

[TheValleyOfDefilement]

I don't think very many people have experience restoring a 6,1 from a non-functional state. At minimum you'll need the appropriate Apple Service Diagnostics. Your best option is probably to take it to the Apple store and have them fix it for you.

That's what I've semi discovered along this path. Not ready to pitch in the towel yet. Off to find S3159 I guess!

 

[TheValleyOfDefilement]

Update: Got it to boot briefly on High Sierra via seeded boot drive, without safe mode. Tests pass on gpu A for Geekbench 4. Gpu B hangs the test. Going to do more tests.

 

[TheValleyOfDefilement]

Update:
Killed a GPU card needlessly (probably) for the purposes of understanding the failure methods. Not sure what that will yield - yet.
Replaced interconnect cable on GPU A, and interconnect cable on GPU B with a replacement card.
Still had trouble with High C, and Mojave, but still could safemode boot.
Tore the unit down completely.
Popped in a new BR2032 battery (NOT CR2032!) while I was in there and cleaned everything up.
It wasn't until I replaced the CPU with an out of band model that I got a declared memory error with the 5 second beep code.
No tools I ran nor the built in diag tool were able to catch it.
Symptom was general slowness and random odd behavior.
Pulling stick and reconfiguring RAM resulted in an operational state.
Started at rev 129.0.0.0, went to 133.0.0.0 through initial Mojave, then moved into Big Sur.
Firmware updated to current position without an issue through Big Sur.
System has been stable over 4 days.
Wakes and sleeps no issue. Works fantastically.
Hope this helps someone find their way.

Side notes:
I followed the apple service manual with a science, and moved through the troubleshooting flowcharts to solve my problems.
One standoff did come out of the core on the CPU riser, however I was able to remove it with a clamping wrench gently attached to the ring of the standoff, not the threads, and wrapped the board with static proof bag, held in place with Kapton tape. This process was done carefully to make sure there was no damage, excessive force, or torsion applied to the board.
The core was repaired by chasing the threads, and certifying they were fine, then applying Loctite 242 to the threads of the standoff and securing. Level measurements were taken across all 4 standoffs during several steps of tightening the standoff, and a final level measurement taken on all 4 to ensure it was true. I stopped applying pressure just slightly after the standoff bottomed out in the core to avoid completely stripping the core and rendering it useless. The core was allowed to have the 242 cure for over 24 hours before additional re-assembly steps were taken.
It is vital that any core repairs are done with utmost concern and caution.
Clean board fingers with a lintless cotton bud and 91% isopropyl alcohol. I also used this to clean the GPUs up. All boards were given ample time to dry out, with verification in several ways prior to re-assembly.
Every step was taken with extreme care and a great attention to detail.
All specs followed completely and tightening patterns to the apple service manual.
All fasteners that had traces of blue on them received a very small amount of Loctite 242 on reassembly.
Separators were made for the IO board and others out of cardboard surrounded by antistatic bag material for reassembly.
MX4 thermal compound was used for both GPUs and the CPU.
OEM VRAM thermal pads were discarded after measuring in every possible way. These were replaced with Thermalright 2MM type. I used 2x of the 85x45x2MM packs. These were cut down to match the entirety of the GPU VRAM copper interface areas, test fitted, and then applied to the GPUs before re-assembly.

I run the system with an advanced fan curve through Macs Fan Control, using Sensor-based value, assigned to GPU 2 Diode. Exhaustive testing revealed that this was the best temp sensor to key off of for all cases I could figure out. Fan speed starts to increase from 55c, and Maximum temperature has been defined as 75c. The sound footprint is louder in some cases, but the system stays much cooler. This is a trade-off I am happy with and doesn't require 3rd party solutions to be involved.
What's the fan noise like with these settings? Idle duty is indistinguishable from ambient noise (With the meter on the top this was mid 20s DB) virtually silent. Light duty catches in the low 30s DB. Moderate usage is high 30s DB. Heavy usage is mid to upper 40s DB.

I take no responsibility for what you do with this information, as the 6,1 is a fragile wonder of the world. Don't do any of this if you aren't moderately or highly advanced in electronics repair and have a strong ability for attention to detail. This information has been provided because I couldn't find the answers I needed.

Overall, I love this thing, even if I facepalmed at the engineering excessively. It was worth it.

 

[TheValleyOfDefilement]

Additional information:
If anyone is wondering what processor was installed and why:
I settled on the 8 core E5-2667 V2.
This processor has the highest single core performance index, and has the same amount of cache as the 10 core E5-2690 V2. Meaning more possible cache per core. It can also punch in at 4GHz. It's a clock for clock winner.
12 core was out of the running quickly, having only really pondered the E5-2697 V2 loosely.
Most applications need higher single core scores, and the ones I use are pretty poor at multiprocessor utilization anyway. I'm not using it for rendering. YMMV.
No <115w or 150w processors were considered. The E5-2687W V2 isn't worth the extended TDP when weighted against the design of the thermal core and single fan, for such mild gains. I scored the E5-2667 V2 for about half the cost of that model.

 

[TheValleyOfDefilement]

Storage information:
The 6,1 arrived with no ssd. Picked up a reasonably sized official SSUAX ssd, and it was cheap. Good for testing. Going to probably use it another few days then replace it with a 1tb SSUBX. I didn't want to mess with any adapters or anything during rebuild, as it was stuck on 129.0.0.0 at the beginning. Not currently planning on using anything but OEM drives on the inside. I'll support that with solid nvme drives as externals and network attached storage.

 

[TheValleyOfDefilement]

Storage information:
The 6,1 arrived with no ssd. Picked up a reasonably sized official SSUAX ssd, and it was cheap. Good for testing. Going to probably use it another few days then replace it with a 1tb SSUBX. I didn't want to mess with any adapters or anything during rebuild, as it was stuck on 129.0.0.0 at the beginning. Not currently planning on using anything but OEM drives on the inside. I'll support that with solid nvme drives as externals and network attached storage.

Update: I switched to the 1TB SSUBX toward the beginning of November. No issues.