I dont think the display settings would be the problem/solution. What would be needed is that the M3 max has actually inplemented support for, say 1080p at 480 Hz.
Does someone know? There are commercially available high refresh rate monitors, ASUS have a 540 Hz gaming monitor. We might see monitors with even higher refresh rates unveiled soon at CES.
As said, if the specific display setting combination is within bandwidth specification of the given DisplayPort version, it has to be possible. Apple never lists DP version on their machine specs but Thunderbolt mandates this, so by proxy all modern Macs especially the ones with TB4 has to support that.
Now why OS setting matters though? Because Apple does not want to clutter the OS display settings interface to the point being unusable by normal people, so they only list a few typical use case settings by default. This is why you may need a 3rd party app to "unlock" those settings.
You mentioned doing this for a project that's where I inferred that you may have been planning to use a custom / industrial display. But even with that ASUS, it is relatively obscure, though we can expect the EDID to probably be done correctly. Whichever the case, there is a good likelihood that you may need to edit custom EDID in order to force the communication between the display and the Mac the way you want.
You are on the right track in seeking 1st hand experiences instead of listening to conjectures like from myself. But using high refresh displays on Macs has always been a really obscure use case, you may end up being the first to test the 480Hz, at least on this forum.
