Having worked in Scientific Computing and Visual Effects/Post Production, no one ever upgrades machine mid-cycle. In addition, one would have to replace the PCIe slots every other cycle as they would need to have their specs increased.While the recent Mac Pro has been criticized by some as a poor value with fuzzy market targeting, I think that Apple can make this a highly desirable machine by doing the following:
1) Make the processor modular. By doing this, businesses would now be buying into an infrastructure, and would save money in the long run since they could get a credit for their current module trade-in, and upgrade at a price below a Mac Studio. Also, time is saved since expansion cards and drivers don't need to be reinstalled.
If you are arguing for two Ultra CPUs in one box with some custom PCIe switch to enable connecting either CPU to any slot, it will never happen. The engineering effort to support that, would be immense and would make such as system cost prohibitive.2) Make dual CPU modules. That's right- go back to the dual g4 days. Of course you're not going to get double the speed of an M3 Ultra, but the OS and CPU bus could in theory be tweaked to yield 1.75x which is what the original M1 Ultra was giving over the M1 Max anyway. Moreover, Apple can set the module prices so that the "dollar per performance" graphs just make a fully tricked out Mac Pro a better overall value.
In a quick search, I do not see that they are binned chips, nor would I expect they would pay TSMC extra for them, as they are unlikely to sell enough to require that they through away any to get their required yield. The existing cooling infrastructure likely already provides enough additional cooling to mean that they can use regular chips.3) Make the modules liquid cooled. Yes, this is an added expense, but hear me out. You're buying a Mac Pro because you want the best of the best, the cre'me de la cre'me, the - you get the point. And for this reason, the chips for the MP are typically binned for higher frequency which makes them expensive. If however, the modules are more like a liquid cooled graphics card, then 15% performance gains could be achieved with standard chips while also preserving chassis space. This in turn, would partially mitigate the less than 2x performance gain from dual configurations.
PCIe 5 is already out. PCIe 6 (or whatever comes next) will be here soon. Newer machines will want newer busses. No one other than hobbyists upgrade the way you describe. It is just creates too many problems.4) While not presently critical, PCie 4.0 at full speed per slot would be a practical forward thinking move. As module speed increases and new cards increase their bus speeds, businesses would likewise increase the value of their chassis investment through longevity.
No, you would not, because it would cost so much more than a current Mac Pro that you could not afford it. That machine would not be twice the speed of an Ultra, as it would not have unified memory and developing for it would be quite difficult. It would be cost prohibitive, as it would require a great deal of custom engineering that would benefit very few users. It would not really be upgradable, as newer machines would have newer busses.If Apple did this, then I would buy a Mac Pro, because I could have one machine with near double the speed of an Ultra, I would have the flexibility for internal SSD and AV expansion, and I could easily upgrade as needs demand.