As far as DLNA software goes, the "file not supported" messages are from the compatability of the TV, not from the server. The server's only job is to present the files it knows are videos to the client in a certain manner, changing the files requires different software.
I can't say whether there is any such thing on Mac, but on Windows, TVersity does the job. It detects what device you are using to play it, has a library of compatability, and transcodes the files into a compatible format before streaming. The disadvantages of this is that it takes a little longer to buffer each individual file, and the setup is a little more complicated, but it means that your TV will play anything that the server can.
If you don't want this, then I would suggest getting a separate player. For standard-def stuff, an original Xbox modded (2 minute job) with XBMC will play pretty much anything, from pretty much anything, and has an amazing set of features (such as making a library of TV shows, and pulling in episode information on each one). It doesn't have the processing power to cope with HD stuff, though.
If you want HD stuff, a Mac Mini with PLEX (same program, but a Mac OSX development branch), or a number of other players will do the job (Popcorn Hour, Icy-Box from RaidSonic).
With regards to the WHS thing, the HP offererings do not have a large footprint. Possibly slightly deeper than if you were to stack a drobo onto a Mac Mini, but there isn't much in it (the dimensions are 25cm x 14cm x 23cm (9.75" x 5.5" x 9.875")), and then that would support Tversity (I know, I have used it on it).