I was on small for a long time but realized it was very difficult to create large windows or large TV/movie screens because it gimps the maximum window size (and IMO it's far too small when watching stuff in Apple TV unless you go to a dedicated theater environment which also is not ideal), so I recently switched back to large and have stayed there.
I also found that some apps totally break in small (Polycam being one) because devs haven't fully accounted for it yet as it changes the dimensions of volumes which isn't well known yet.
Where do you go to make adjustments?
Settings/Display/Appearance. This is a separate thing to text size, and is more akin to the "More Space" option on Macs where the whole UI scales up and down.