Experiences with SFP28?

[TrumanLA]

I'm wondering if you can help me out. I also made a Hackintosh using Supermicro with the X11SPA-T (customized firmware I had to get authenticated with Supermicro) and I did this prior to prices dropping on used Mac Pro 7,1's (2019). So, I ended up turning that X11 into my new server board on my 24 bay. I also have other Hackintoshes, but I would ideally like to get this going in the 7,1 and with an enclosure with my Mac Studio.

Just a note about getting SFP+ performance on Catalina using an older MacPro with any SFP+ // SFP28 card ... the only thing that actually worked for me (contrary to what everyone says ... that SMB (samba) is the best protocol) was AFP in Peer-to-Peer and transfers to and from my TrueNAS. FWIW, I will always rule out which protocol works best in the future.

 

[DCswitch]

Just a note about getting SFP+ performance on Catalina using an older MacPro with any SFP+ // SFP28 card ... the only thing that actually worked for me (contrary to what everyone says ... that SMB (samba) is the best protocol) was AFP in Peer-to-Peer and transfers to and from my TrueNAS. FWIW, I will always rule out which protocol works best in the future.

This is simply false. I tested them all thoroughly, got them all to work, and SMB is the clear winner.

 

[DCswitch]

@metacollin @Kris Kelvin

I'm hoping you can help me out a bit. If you're short on time, metacollin could you simply upload your EFI? Perhaps, there is something in OpenCore that's allowing it to work. In a 7,1, I'm just not getting the kext/driver to load.

 

[DCswitch]

Success! I have the TP580-LP-CR running on Ventura 13.2.1!

Bad news is you will have to either turn off SIP to use the kext as it no longer passes signature verification, OR you could inject it using a bootloader like OpenCore (harder but probably the more secure option as you don't have to turn off SIP).

No word on speed yet, but it works fine with cxgbtool at least.

@metacollin @Kris Kelvin Hope you're well. I'm doing something that is really important and could use your help.

 

[Kris Kelvin]

I'm hoping you can help me out a bit.

I'm afraid I currently don't have the hardware to test any of this.

What you could try is using verbose kextload (kextload -v) and report back with the output.

 

[DCswitch]

I'm afraid I currently don't have the hardware to test any of this.

What you could try is using verbose kextload (kextload -v) and report back with the output.

italy@Italys-Mac-Pro Extensions % kextload -v cxgb.kext
Executing: /usr/bin/kmutil load -p /Library/Extensions/cxgb.kext

Error Domain=KMErrorDomain Code=71 "Insufficient permissions for action: Unable to load 'com.softraid.driver.SoftRAID' in unprivileged request." UserInfo={NSLocalizedDescription=Insufficient permissions for action: Unable to load 'com.softraid.driver.SoftRAID' in unprivileged request.}

italy@Italys-Mac-Pro Extensions % csrutil status
System Integrity Protection status: disabled.

italy@Italys-Mac-Pro Extensions % sudo kextload -v cxgb.kext
Password:
Executing: /usr/bin/kmutil load -p /Library/Extensions/cxgb.kext

I forgot sudo the first time, but with sudo there is no output and nothing happens.

 

[Kris Kelvin]

italy@Italys-Mac-Pro Extensions % sudo kextload -v cxgb.kext
Executing: /usr/bin/kmutil load -p /Library/Extensions/cxgb.kext

There is no error shown, so I assume it's loaded.

Can you run the command kextstat to verify that?
If "com.chelsio.driver.cxgb" is listed, it is loaded and we can proceed to the next issue.

PS: Just to be sure, you're on Intel, right? The Chelsio driver currently isn't available for Apple Silicon.

 

[Kris Kelvin]

Sonnet introduced a 25G adapter for macOS, too:
https://www.sonnettech.com/product/twin25g/overview.html

Soon to be available as PCIe card or external Thunderbolt adapter (with separate power brick, unfortunately).

It's based on a Mellanox chipset and may currently be the only solution with a modern driver (based on DriverKit).
Apple Silicon and macOS Ventura are supported.

 

[DCswitch]

There is no error shown, so I assume it's loaded.

Can you run the command kextstat to verify that?
If "com.chelsio.driver.cxgb" is listed, it is loaded and we can proceed to the next issue.

PS: Just to be sure, you're on Intel, right? The Chelsio driver currently isn't available for Apple Silicon.

Thank you for helping me out. Intel, yes. It's an Intel Xeon W processor in a Mac Pro 7,1 2019.
I don't see it, but here are two screenshots.

I also have it working with the official driver from Chelsio on Big Sur (they list it for Catalina, but I managed to get it working in Big Sur, but not in Monterey or Ventura) if that's helpful.

 

[DCswitch]

Sonnet introduced a 25G adapter for macOS, too:
https://www.sonnettech.com/product/twin25g/overview.html

Soon to be available as PCIe card or external Thunderbolt adapter (with separate power brick, unfortunately).

It's based on a Mellanox chipset and may currently be the only solution with a modern driver (based on DriverKit).
Apple Silicon and macOS Ventura are supported.

Good to know, but they do cost $900, and only 25G. I like to keep 40G at my minimum these days.

 

[DCswitch]

There is no error shown, so I assume it's loaded.

Can you run the command kextstat to verify that?
If "com.chelsio.driver.cxgb" is listed, it is loaded and we can proceed to the next issue.

PS: Just to be sure, you're on Intel, right? The Chelsio driver currently isn't available for Apple Silicon.

@metacollin - not sure if you could upload your Supermicro EFI. What model motherboard is it?

@Kris Kelvin Any thoughts to proceed?

 

[Kris Kelvin]

Any thoughts to proceed?

The kext doesn't seem to be loaded. You'll have to find out why. Maybe because it is (invalidly) signed, maybe it is in the wrong location, maybe SIP isn't really off. There are many possibilities, but this isn't really the thread for that. Google is your friend.

 

[m1carbon]

The Mellanox MCX4121A-ACAT that you got working ... is that with the OEM Myllenox ROM or did you swap the ROM out for an ATTO ROM..? Also, I tried using both the ATTO and ConnectX-4 on a MP5,1 ... as well as a Chelsio S320E-SR and while the ATTO and ConnectX-4 were both limited to about 1Gb for me also (in the MP5,1) ... ATTO said it's bc Mojave's SMB stack is crap. But ... the S320E actually did give me 10GbE ... for 10Gb and then it **** the bed and lost connectivity. I tried installing my T520-SO ... but the cMP wouldn't even chime (post) with the card installed (in any PCIe slot). Very goofy considering Chelsio makes a driver that supposedly works with everything from El Capitan thru Mojave and a 2nd driver for Catalina (and presumably newer OS also). I'm waiting for a power cable to test these via TB3 with an OEM Myllenox, ATTO-ROM Myllenox, ATTO SFP28 card ... to see what really works etc. I don't think I'll bother checking the diff between 10Gb and SFP28 for now (I hate turning on my SFP28 switch bc it's much louder than the Mikrotik).

I don't imagine [how] the MCX412A is connected (PCIe vs TB3) really matters ... right..?

Also, re MTU 9000 ... I went through a thread on STH a couple of years ago by a guy who documented a crusade he'd gone on trying to get all of his gear working with MTU 9000 settings and ultimately found it to be more effort than it was worth. Did you come to a different conclusion..?

Thanks for the info you posted.

It is a Mellanox CX4. I didn't modify anything. I settled with just using my X520-DA2 since it's Plug n Play with 10G working.

 

[DCswitch]

The kext doesn't seem to be loaded. You'll have to find out why. Maybe because it is (invalidly) signed, maybe it is in the wrong location, maybe SIP isn't really off. There are many possibilities, but this isn't really the thread for that. Google is your friend.

That's a bummer, I was hoping you had more to offer than that, but I understand you may be busy. SIP is definitely turned off as I showed you the screenshot, the kext is definitely in the correct place, so that leaves us with signing it. I think someone mentioned needing a Developer account to sign it, SO if anyone is reading this... does anyone have a Developer account with Apple?

@metacollin Still hoping to hear from you as your EFI or advice may solve this.

 

[Kris Kelvin]

Some people figured out that macOS Ventura already contains DriverKit-based drivers for additional ethernet chipsets out of the box: Mellanox (ConnectX-4, ConnectX-5 and ConnectX-6) and Intel (X500 and X700).

This post here says it works fine (except for not supporting jumbo frames), even in a Thunderbolt PCIe enclosure.

This is great news and now clearly the best option going forward. No tinkering with SIP or third-party drivers and no risky system upgrades!

I'm off to find some used Mellanox gear.

 

[jnewell]

Success! I have the TP580-LP-CR running on Ventura 13.2.1!

Bad news is you will have to either turn off SIP to use the kext as it no longer passes signature verification, OR you could inject it using a bootloader like OpenCore (harder but probably the more secure option as you don't have to turn off SIP).

No word on speed yet, but it works fine with cxgbtool at least.

Edit: nevermind, the issue was on my side with the switch. Works great, thanks again!

Hey Colin, nice work! I used your kext and was able to also get Ventura to (at least) "see" my TP580-LP-CR card too. However, were you able to get it to actually talk on the network? Mine while showing a link up in ifconfig, doesn't actually connect to the network - can't DHCP - and I think it is because the link speed is not correct, but I cannot change it in the "hardware" section of network utils. Did you make any progress here? Feels like we are so close. I am using intel hackintosh with OC bootloader.

 

[Kris Kelvin]

A $300 small external Thunderbolt 25G Ethernet Adapter for Macs:

Enclosure: Wikingoo M1 (~$200 from Aliexpress, search for "Mini egpu")
Adapter: Nvidia Mellanox ConnectX-4 Lx (MCX4121A-ACAT with low-profile bracket) (~$100 from Ebay)
Driver: Apple's Mellanox driver (integrated in macOS Ventura)

Some notes:

• The ethernet adapter came with a Huawei firmware; I flashed the regular Mellanox firmware to it (using a Linux box).
• The ConnectX-4 Lx consumes around 10 watts, so it needs some airflow (see manufacturer specifications). The Wikingoo M1 doesn't have any active cooling, so I had to find way to install a fan. Luckily, the board contains an EPS12V connector (alternative power input?). It can be utilized as a 12V power source for fans. I've installed two 60x60x10mm fans that ensure both the ethernet chip and the pluggables will stay cool (at the expense of some noise, despite using Noctua NA-SRC10 low noise adapters). Maybe the QNAP QXG-25G2SF-CX6 PCIe card would be a better fit – same Mellanox chipset but with an integrated fan.
• I'd assume that performance is not quite as good as the (pretty expensive) solutions from ATTO and Sonnet. However, I like the ability to use the device without any additional driver.
• The Thunderbolt bridge is bus-powered, but the PCIe card is powered by an external 12V power supply. The Thunderbolt spec only guarantees 15 watts of power anyway, so powering a ConnectX-4 over a Thunderbolt connection is probably impossible.

 

[TheRacerMaster]

Success! I have the TP580-LP-CR running on Ventura 13.2.1!

Bad news is you will have to either turn off SIP to use the kext as it no longer passes signature verification, OR you could inject it using a bootloader like OpenCore (harder but probably the more secure option as you don't have to turn off SIP).

No word on speed yet, but it works fine with cxgbtool at least.

Oh nice, I'm not the only one who looked into this 🙂 I ended up doing the same thing in a Lilu plugin: https://github.com/al3xtjames/CXGBFixup

I also saw that the link would drop after a sleep/wake cycle when using jumbo frames. I ended up hacking around this by reconfiguring the MTU on wake (by hooking functions with Lilu).

 

[unphased]

A $300 small external Thunderbolt 25G Ethernet Adapter for Macs:

Enclosure: Wikingoo M1 (~$200 from Aliexpress, search for "Mini egpu")
Adapter: Nvidia Mellanox ConnectX-4 Lx (MCX4121A-ACAT with low-profile bracket) (~$100 from Ebay)
Driver: Apple's Mellanox driver (integrated in macOS Ventura)

Some notes:

• The ethernet adapter came with a Huawei firmware; I flashed the regular Mellanox firmware to it (using a Linux box).
• The ConnectX-4 Lx consumes around 10 watts, so it needs some airflow (see manufacturer specifications). The Wikingoo M1 doesn't have any active cooling, so I had to find way to install a fan. Luckily, the board contains an EPS12V connector (alternative power input?). It can be utilized as a 12V power source for fans. I've installed two 60x60x10mm fans that ensure both the ethernet chip and the pluggables will stay cool (at the expense of some noise, despite using Noctua NA-SRC10 low noise adapters). Maybe the QNAP QXG-25G2SF-CX6 PCIe card would be a better fit – same Mellanox chipset but with an integrated fan.
• I'd assume that performance is not quite as good as the (pretty expensive) solutions from ATTO and Sonnet. However, I like the ability to use the device without any additional driver.
• The Thunderbolt bridge is bus-powered, but the PCIe card is powered by an external 12V power supply. The Thunderbolt spec only guarantees 15 watts of power anyway, so powering a ConnectX-4 over a Thunderbolt connection is probably impossible.

Hey thanks for posting this, it was really helpful to know that an mlx5 driver is in macOS now and it allowed me to pull the trigger on a ConnectX-4 card. I was able to order a MCX455A-ECAT for around $70, this is actually a 100Gbit card with the 16 PCIe lanes as well, I felt good about one of the cheapest available options being a 100Gbit capable card, so I don't have to even flash it to run that speed. Also, it is known that almost all of these CX4 cards can be flashed to unlock 100Gbit on them!

Still, the dual port cards are all at around the $140 mark or higher so I am grateful to get something functional for my Apple Silicon macbook without breaking the bank, since this time all I needed to buy was the CX4 card, which is perfect. It works with this old Sonnet eGPU enclosure I have. it's comically big for this tiny card, and the card I have does not have a full height bracket, but it works! It did not show any signs of life until I set it to Ethernet mode with `mlxconfig` with my linux machine. But I am sure this workflow only affects VPI cards that are capable of Infiniband, they tend to come configured for IB by default.

I haven't configured packet size and I am getting 12Gbit between this CX4 and a CX3 on my linux box. I'm not sure if that's to be expected or not.

What configuration did you do to push nearly 20Gbit with yours? I'm worried that jumbo packet size configuration is the only knob we have to tweak. It does not seem like RDMA will come to macOS any time soon. Not sure.

At any rate even with roughly 10Mbit it's already worth drilling some holes and running fiber into my living room, so I can finally have a handy USB-C cable that charges the macbook while also giving it full gigabit internet speed as well as let me actually transfer data faster than I can with external storage.

 

[Kris Kelvin]

What configuration did you do to push nearly 20Gbit with yours?

Default configuration, actually. No Jumbo frames or anything.
Note that the 19Gbit/s were from "remote" (PC) to "local" (Mac). The other way may have been slower.

What was the negotiated Ethernet link speed between the ConnectX-3 and the ConnectX-4?

 

[unphased]

Ah ok. I didn't actually test MTU 1500, in reverse, but I'll test next time, yes I had 9000 MTU set on the Linux connectx-3 side, and with iperf3 -R (to send from linux) I do get 19.9Gbps so that lines up with your result.

With MTU 2034 (the max allowed by Apple's driver) I can only send at 16Gbit/s. It's not bad, but a bit frustrating knowing the hardware can do more.

Negotiated speed is 40Gbit.

I suppose I'll just have to settle for this. As I can only really afford 4 PCIe lanes to put these NICs on... even if there wasn't this bottleneck, there'd be the thunderbolt inherent bottleneck which I seem to recall is around 22Gbps, and then, shortly after that comes the 4 PCIe lane bottleneck which I measured at 26Gbit between linux and windows back when I was testing that. Where both ends were connected on M.2 slots with 4 lanes.

 

[devon807]

Success! I have the TP580-LP-CR running on Ventura 13.2.1!

Bad news is you will have to either turn off SIP to use the kext as it no longer passes signature verification, OR you could inject it using a bootloader like OpenCore (harder but probably the more secure option as you don't have to turn off SIP).

No word on speed yet, but it works fine with cxgbtool at least.

I used your kext, and was able to get it working (ish) in Sonoma, but couldnt get it in Ventura for whatever reason. Card shows as "Terminator 4" lol. Card is identified but can get the link up

 

[sash11]

I just tested two 25GB sfp28 nics through Sonnet echo express single slot low profile expansion TB3 enclosure.
Frist card was Intel XXV710-DA1. I do not recommend using it. It did not properly work with my mikrotic CRS510-8XS-2XQ-IN switch. I could access the internet and some hosts on my network, but it did something, that I could not even ping my ESXi server and all VMs running on it. What is even more bizarre is that other computers on the network could not ping ESXi host and VMs on it when this card was active on the network. I tried all possible configurations for the SFP28 port on the switch, but nothing helped.
Second card I tried was Mellanox MCX4111A-ACAT, also single port 25GB sfp28. This card had no issues what so ever. I could access all my devices on the network. So I am sticking with it, just need to find a low profile bracket for it.
Intel card used native macOS driver com.apple.DriverKit-AppleEthernetXL while Mellanox used com.appleDriverKit-AppleEthernetMLX5.