MP All Models Mellanox ConnectX-3 40 GbE using ATTO FastFrame macOS driver

[jamesq]

Easy to check: Look into /System/Library/DriverExtensions/
(Terminal: ls /System/Library/DriverExtensions/)
ixgbe is the driver for Intel X500 series of NICs (10 GbE)
ixl for Intel X700 series (10/25/40 GbE)
mlx5 for Mellanox ConnectX-4 to 6 series
A French blogger reported on using an Intel X520 in Thunderbolt enclosure with an iPad! M1 Mac should be a breeze.


Apparently, Sonnet ported the driver independently of Apple (and has jumbo frames which are reportedly missing from the Apple driver for now). For this to make commercial and technical sense, Sonnet likely changes some ID on the cards they sell, and lock the driver to their modded cards, as with their 10 GbE offers. It should be possible to patch the driver, or alter IDs of a generic card, but why would one attempt that as first intention rather than using Apple's native driver with genuine, unaltered, cards?

Thank you for your reply! The issue is that we need 25Gbe thunderbolt 3 adapter and all of them are like $1K and higher. My plan was:
"To buy Sonnet Echo Express SEL Thunderbolt 3 to PCIe 3.0 x8 expansion enclosure and then attach a Mellanox ConnectX-4 Lx MCX4121A-ACAT (dual port) or MCX4111A-ACAT (single port) NIC."

This would be like $450

But from what you have wrote it feels like this will not work.

What is the problem with drivers that are limited at 1500-2000 MTU and not have jumbo frames? Will it affect speed or bandwidth during file transfers?

 

[etorix]

But from what you have wrote it feels like this will not work.

On the contrary, I do not see any reason why a generic 25 GbE ConnectX or Intel XXV710 in any Thunderbolt enclosure would not work with the Apple driver. Just don't use the Sonnet driver with generic cards.

What is the problem with drivers that are limited at 1500-2000 MTU and not have jumbo frames? Will it affect speed or bandwidth during file transfers?

Jumbo frames may provide a slight improvement but should not be a deal breaker.
After all, Ethernet evolved from 10 Mbits/s to (currently) 800 Gbits/s in many steps; and up to 10 Gbits/s, these steps were ten-fold increases. During all this time there has been ONE single increase in maximal frame size, a meagre six-fold increase from 1500 to 9000: One can wonder why. And the short answer is that "jumbo frames" brought much trouble for limited results; improving the processing of an ever increasing number of small packets proved more effective, and scalable, than enlarging the packets.

 

[metacollin]

I did manage to flash the firmware again on the mac but it doesn't change anything. Looks like the same issue with NVRAM. Do you have any ideas on how to proceed?

Edit: I got a bit further. I could restore the NVRAM to default settings with atinicnvr and now the NVRAM not found message disappeared. Still the same in the ATTO config tool. Maybe we would need a proper ATTO Nvram file to copy onto the card?

Ok. I finally got my hands on an MCX-314A-BCCT an I'm here to hack firmware and chew bubblegum. I'm not out of bubblegum, I have quite a bit actually, but it's cool I can multitask.

So first discovery so far is that ATTO is pretty liberal with their use of the word "NVRAM" and in some documents can and will refer to the flash memory where the firmware lives as NVRAM as well.

I found some documentation on the NQ41/42 that explicitly state that the MAC addresses are saved in flash memory.

So this isn't a case of anything being locked. What's happening is the ATTO firmware is looking at a specific memory address on the flash chip, one that is outside the addresses consumed by the firmware image. Unless it is an original ATTO card, this address will have nothing there and read all zeros by the firmware.

Now, we could try to determine which flash memory address we would need to write to but it would require someone dumping the entire flash chip contents of a proper ATTO card which may not even be possible without using a hardware spi memory reader/burner and desoldering the chip to do it. We could find someone keen on doing this to their $700 ATTO card...

But let's not do that. I'm just going to write a bunch of random garbage bits to the unused part of flash memory and see what address the ATTO driver comes up with. A 48 bit MAC address should be unique and I just have to search for the location of those same bits in what I burned to the flash memory.


Then it's just a matter of writing an ATTO vendor id'd MAC to said address and bob's your auntie.

Assuming the driver even cares about that, there is a good chance it doesn't. It probably just checks that there is something there, not if it looks correct but I could be wrong.

The larger question is if this can be done from the host computer or if it can only be done using a flash burner. I would very much like to find a solution that wouldn't require the user performing surface mount hot air rework and desoldering chips.
Ideally, a solution that requires running some commands from Linux at the worst.

I'll be posting a follow up once I have more info. Wish me luck!

 

[aurimasniekis]

After I figured out that Ventura contains mlx5 drivers, I grabbed some cheap PCIe dock from AliExpress and one spare 100gbe nic I had on had ConnectX4 100gbe. It worked out of the box, max with the short 100gbe dac cable I had to my Windows machine I was able to push out 21gbit through iperf3

I even tried with my iPad Pro M2 to see if it will work as it's basically the same OS, and yes it works but the max I was able to push out was ~16.5gbit and a few sec peaks of 19gbit.

Regarding MTU with MLX5 I dunno but appears that Apple hard-coded the 2034 MTU as max when the driver itself says 9000 inside where it checks interface limits.

I hope Apple will change this in future updates...

I tried to insert ConnectX 3 card but for some reason MLX5 kernel panics with it...

 

[xjn819]

After I figured out that Ventura contains mlx5 drivers, I grabbed some cheap PCIe dock from AliExpress and one spare 100gbe nic I had on had ConnectX4 100gbe. It worked out of the box, max with the short 100gbe dac cable I had to my Windows machine I was able to push out 21gbit through iperf3

I even tried with my iPad Pro M2 to see if it will work as it's basically the same OS, and yes it works but the max I was able to push out was ~16.5gbit and a few sec peaks of 19gbit.

Regarding MTU with MLX5 I dunno but appears that Apple hard-coded the 2034 MTU as max when the driver itself says 9000 inside where it checks interface limits.

I hope Apple will change this in future updates...

View attachment 2215725

I tried to insert ConnectX 3 card but for some reason MLX5 kernel panics with it...

View attachment 2215726

well, all the network interfaces are now under IOSkywalk, mlx5 and other NIC driverkits are still developing to work with skywalk. Tested on MacOS14 beta, nothing changes

 

[neonlost0]

thanks for this info, i had some connect x3 cards laying around and couldn't get it working but picked up a used connect x4, once i got it out of IB mode Mac OS found it and it works pretty well

 

[xjn819]

thanks for this info, i had some connect x3 cards laying around and couldn't get it working but picked up a used connect x4, once i got it out of IB mode Mac OS found it and it works pretty well

Are you referring to IPoIB ? Or more than that with RDMA ? This is big news to me!

 

[etorix]

thanks for this info, i had some connect x3 cards laying around and couldn't get it working but picked up a used connect x4, once i got it out of IB mode Mac OS found it and it works pretty well

Mlx5 supports ConnectX -4, 5, and 6 series, not 3.

Are you referring to IPoIB ? Or more than that with RDMA ? This is big news to me!

In the context of Mellanox cards, "IB" must be "InfiniBand", as short-latency data centre infrastructure connection is a big application for them. But asking about obscure acronyms by throwing in more acronyms… Hmm, how could I say…

 

[xjn819]

Mlx5 supports ConnectX -4, 5, and 6 series, not 3.


In the context of Mellanox cards, "IB" must be "InfiniBand", as short-latency data centre infrastructure connection is a big application for them. But asking about obscure acronyms by throwing in more acronyms… Hmm, how could I say…

Well, I have been using IP O InfiniBand with RDMA on macOS for a long time. I have an ATTO 40gb NIC, and I ported RDMA core myself. However, it seems that these cannot be achieved via dk, so I have to use ATTO's NIC, which is a kext driver that will be deprecated soon.
It is really good thing for me if apple already done rdma things

 

[neonlost0]

Are you referring to IPoIB ? Or more than that with RDMA ? This is big news to me!

yes i meant the card was in IPoIB mode, switched it to ETH then it connected.

 

[xjn819]

yes i meant the card was in IPoIB mode, switched it to ETH then it connected.

My bad. hope someone can help test the RoCE functionality of the eth mode

 

[klzsysy]

Hi, can you please confirm that you have this exact model "mellanox CX4212A-ACAT"? I cannot find information about this nic anywhere. Maybe you meant this one?
Mellanox ConnectX-4 Lx MCX4121A-ACAT

I want to try a similar combination, but afraid that M1 macOS will require some drivers that does not exist. Did your 25GbE mellanox nic worked out of the box without any drivers or you had tweaked something?

Thank you in advance.

I want to buy Sonnet Echo Express SEL Thunderbolt 3 to PCIe 3.0 x8 expansion enclosure and then attach a Mellanox ConnectX-4 Lx MCX4121A-ACAT (dual port) or MCX4111A-ACAT (single port) NIC.

Recently Sonnet announced and started to sell a new product called Sonnet Twin25G thunderbolt 3 adapter:
https://www.sonnettech.com/product/twin25g/overview.html

and it looks like it uses the Mellanox 25gbe network adapters from the same family ConnectX-4 Lx. They prodive some drivers here:
http://www.sonnettech.com/support/kb/kb.php?cat=635&expand=_a3&action=b1219#b1219

But they require macOS 13.2+ Ventura. Does anyone know if their driver can work with any Mellanox ConnectX-4 Lx MCX4121A-ACAT or they can somehow vendor lock their driver to work only with their version of this card?

Sorry, I made a typo, its model is mcx4121a-acat

No need to do any extra operations, it works out of the box

 

[Dandu]

Hi

I have two Connect X4 (25 Gb/s), a DAC cable (25 Gb/s, 25G BASE CR) and two Mac with Thunderbolt enclosure. And i can't have more than ~12 Gb/s. Is there a special way to test with iPerf ?

 

[Kris Kelvin]

I have two Connect X4 (25 Gb/s), a DAC cable (25 Gb/s, 25G BASE CR) and two Mac with Thunderbolt enclosure. And i can't have more than ~12 Gb/s.

Are your Thunderbolt enclosures and cables rated for 40Gbit/s? Especially the longer cables (>1m) often are 20Gbit/s only. You can check that in the System Profiler.

Do both Macs agree on the negotiated network speeds shown in System Preferences?

Is there a special way to test with iPerf ?

What version of iPerf are you using?
If using IperfUtil, do you have "Mode" set to "Bandwidth Local <- Remote"?
If using the CLI version, what parameters did you specify?

 

[Dandu]

The two are 40 Gb/s, no problem with that.

Actually, i have 19 Gb/s from a PC to the Mac with the -P 8 parameters.

I dont use special parameters : iperf -s for the server, iperf -c IP with the client (and -P 8).
I have tried with two Mac mini M1 and a MacBook Pro 2017, but the MLX5 driver is sigle threaded and the x86 version is very slow (100 % from a core).

I have tried with iperf3, the last version installed with homebrew

 

[enamjuhoure]

Monterey 13.3.1 M1Max macbookPro
thunderbolt 3 dock + mellanox CX4212A-ACAT 25G nic Work very well

The only problem is that only 2034 largest mtu

View attachment 2201816

View attachment 2201815

Can it work in 25GBase Mode？
It can only work in 10GBase Mode in my devices.

My devices:
Macbook Air M2 Ventura 13.5 <---> thunderbolt 3 dock <--->mellanox CX4121A-ACAT 25G nic <---> mellanox CX4121A-ACAT 25G nic <---> NAS(NUC9)

The cable is a SPF28 DAC cable.

 

[mkush]

On the contrary, I do not see any reason why a generic 25 GbE ConnectX or Intel XXV710 in any Thunderbolt enclosure would not work with the Apple driver. Just don't use the Sonnet driver with generic cards.

Just to confirm... plain old Intel cards are working with no added drivers. Specifically, I'm running a Mac Studio (M1 Ultra) with Sonoma 14.0 Beta (build 23A5337a, public beta 6). I put an Intel XL710 (40Gbe, full model number XL710-QSR1) in a Sonnet Echo Express SE IIIe Thunderbolt chassis. MTU looks to be limited to a range of 1280-2034. It's going to be talking to a FreeBSD server with an identical card, direct connected with fiber. All I've done so far is ping, so no idea of performance. I know Thunderbolt is not going to let the 40Gbe card get anywhere near full performance but I found these cards and the fiber on eBay for pretty cheap ($150/ea for the cards including Intel transceivers so I just went ahead and didn't bother with 25Gbe cards).

 

[janissammetti]

Hello,

I flashed my card with the ATTO firmware and it will work under Windows. But unfortunately not on Mac Ventura, see screenshorts.

 

[unphased]

My MCX455A-ECAT is working well so far, I will need to drill holes to run fiber before I can use it more regularly to report if it is solid or flaky. So far the 2034 max MTU has been the most disappointing as it's clear the system can handle more (19.9Gbit when receiving, since on the other side I have set the CX3 card to have MTU 9000). When sending I can only reach 16.4Gbit with MTU 2034. It almost feels like an artificial driver limitation done by apple for market segmentation or something.

@xjn819 when you say "help test the RoCE functionality of the eth mode" how would we go about doing this? I'd love to see RDMA working on macOS and what you have stated so far in various places is enticing. If we're going to be hamstrung by artificial MTU limitations, RDMA is the only hope it would seem.

Does anyone know what it would take to hack the driver to work around the 2034 limitation?

Either way I'm still glad to get 16Gbit, since that is still faster than anything I can get from 10Gbit and I don't think you can even get 10Gbit thunderbolt networking for under $250, and I only had to spend $70 for this CX4 VPI card. Yes, you have to use linux to reconfigure it to ethernet mode with mlxconfig before it will show signs of life, but that won't be a concern if you get an ethernet only CX4 card.

 

[unphased]

Is there some way we can hack ATTO drivers to work on a setup like this? Don't those have full 9000 mtu?

 

[reasonsandreasons]

Also playing around with these cards in a Thunderbolt enclosure and the new native driver. I have the MCX4121A-ACAT and everything's reporting as appropriate with Sonoma--I need to do more performance testing, though.

A side note is that the MCX4121A-ACAT is the same card that the Sonnet Twin25G is based on, to the point that it's identified as one in a Windows host and uses Nvidia's drivers. If anyone has a Twin25G card and could provide an NVRAM dump I'd be interested to see if it's possible to refresh it to get it to work. As-is the Sonnet driver doesn't activate with this card, but I did get occasional kernel panics with it enabled so something is happening. I don't think the MTU and lack of jumbo frames are an issue for my purposes--the extra 6Gbps or so I can likely get is a meaningful improvement--but it'd be nice to juice the performance higher. Hopefully macOS 15 improves the situation.

I will check in on SMB multichannel, too. That might be an avenue to get higher speeds in the meantime.

 

[klzsysy]

Can it work in 25GBase Mode？
It can only work in 10GBase Mode in my devices.

My devices:
Macbook Air M2 Ventura 13.5 <---> thunderbolt 3 dock <--->mellanox CX4121A-ACAT 25G nic <---> mellanox CX4121A-ACAT 25G nic <---> NAS(NUC9)

The cable is a SPF28 DAC cable.

You can try short-circuiting the two ports of the network card to verify the speed.

 

[srgz]

Hi...I was just perusing this thread and I am still wondering -- is it possible to get any form of RDMA working currently on MacOS with ANY manufacturer's nics?

SMB direct, RoCE, iWarp, NFS 4 using RDMA for the transport layer, even iSCSI with iSER...do ANY of these work on any version of MacOS with any NIC from any manufacturer....?

I don't see what good getting a card that is faster than 10GbE working will do if you can't have some kind of RDMA instead of TCP for the transport...it will just be bottlenecked from transport up to the application layer, won’t it?

 

[unphased]

Hi...I was just perusing this thread and I am still wondering -- is it possible to get any form of RDMA working currently on MacOS with ANY manufacturer's nics?

SMB direct, RoCE, iWarp, NFS 4 using RDMA for the transport layer, even iSCSI with iSER...do ANY of these work on any version of MacOS with any NIC from any manufacturer....?

I don't see what good getting a card that is faster than 10GbE working will do if you can't have some kind of RDMA instead of TCP for the transport...it will just be bottlenecked from transport up to the application layer, won’t it?

I am wondering the same thing re: RDMA. I will be working on bringing up various RDMA solutions going between the Linux machines and to Windows, since it seems like there is some degree of support there between NFSoRDMA and SMB Direct (for which I understand that linux has an implementation as well). macOS on the other hand is a huge question mark.

With that said, I can also report that SMB Multichannel from my M1 Max via ConnectX-4 via thunderbolt to an NVMe behind a ConnectX-3 on the other side does achieve greater than 1GB/s throughput, so it does go faster than 10Gbit. I believe, but I am not sure about this, that thunderbolt itself limits to somewhere around 22Gbits of usable bandwidth. I was getting around 15 or so Gbit up and almost exactly 20Gbit down to the mac.

Currently getting my Mellanox 40Gbit SX6036 switch set up... Didn't do much speed testing yet, trying to get to the bottom of this issue where once disconnected for a few hours, the line would stay down until I reseat the transceiver at the switch end, which is no fun.

 

[perplx]

That's a known issue with Alpine Ridge and Titan Ridge thunderbolt chips, the old intel chips were limited at about 22Gbits. There is a new USB4 chip by asmedia the ASM2464PD used in the Zike drive that supports upto 32Gbits, which is only achievable on newer intel chipsets, on apple it's limited to about 25Gbits. There are no pcie enclosures that use the new asmedia chip yet but you could make one using a riser on the Zike drive.

links

https://www.reddit.com/r/Thunderbolt/comments/16x4yrq https://www.reddit.com/r/UsbCHardware/comments/16bqgtg

List of fastest USB4 ASM2464PD & TB3 JHL7440 SSD enclosures (January 2024)

[2024/01/07 – Additional models] IntroductionFastest external M.2 SSD enclosuresUSB4 ASM2464PD SSD enclosuresRetail modelsODM modelsThunderbolt 3 JHL7440 enclosuresFor TB3/TB4/USB4 hosts and …

dancharblog.wordpress.com