drjack, you are asking an extraordinarily subjective question and can't get good input from anyone here telling you what "they" do, as they would be making very subjective decisions too.
And you can't judge dark Batman vs. dark Aliens as any kind of objective test. It doesn't matter that they are the about the same movie length. Again, go max dark- use something like Keynote to render the (one) color BLACK at 1920 x 1080 for the exact same amount of minutes as either Batman or Aliens. (Every pixel) Black is as dark as a "film" can get. Then, run that render through Handbrake and see what you get.
The file size of movies (subjectively judged as "both are dark" or not) is going to vary widely if you want to key on quality. Grainy films are going to be big files because HB is trying to capture all the detail in the grain. It's that variety of detail (variety of pixel colors) that is going to yield a larger file.
Here's an easy metaphor to help you think about what HB is doing: pull out a pad of paper, take a pencil and reproduce a picture with much detail as good as you can. Turn the page (to a new blank page) and draw a picture of a single color: let's say white, so you don't have to actually even touch the pencil to the paper. Which one took the most time & effort to draw? Obviously, the first. The first required a lot of "data store" because there was a lot of detail in the frame to "capture." The second didn't require any detail. Go scan each picture. The first is going to end up a much bigger file than the second for the same reason.
Now think about a moving picture: 24-30 frames (sketches) per second. If in the first one you were drawing what you see looking out the window as you drive down a highway, those frames are going to vary widely as the scene changes. If you are going to capture what you see in great detail (quality), it's going to take a LOT of careful drawing (HB processing) of each frame to capture all that detail for "playback" later. All those pages (frames) of the pad are going to be full of ever-changing detail. The pad represents the "storage" of the movie you are capturing as individual frames.
Let's imagine each of the drawing pads has 1,000 pages (frames). In pad #2, the movie is just the (one) color white. Rather than needing a fresh frame by flipping to page 2, you can just hold on page 1, as the image is exactly the same. In fact, in pad #2, you never need a new frame to reproduce the pure white you see. So, you can simply "play" page one 1,000 times to create an equal length "movie" of pad #1.
Both pads have the exact same horiz and vert resolution. You've subjectively put in the level of quality you want in drawing the individual "frames" in pad #1 and pad #2 is a perfect quality reproduction of it's "movie" (the color white). What's the storage like? Pad #1 is the "file" storage of 1000 individually-drawn pages at whatever level of quality you chose to put into drawing each frame. Pad #2 can be "stored" as just page 1 shown 1000 times. The "file size" of pad #1 is 1000 pages and pad #2 is 1 page.
As to your question of quality vs. file size, no one can help you get it right because only you can decide how much quality you want to trade off to get the file size you desire. Anyone else offering suggestions are basing it on what they subjectively choose as a balance of quality vs. file size. The easiest answer for you is to choose the
TV3 preset and move the quality slider up a notch, render, move it up another notch, render, another, render, etc... and then compare each render until quality falls below your own threshold. Each notch up should yield a smaller file. So when you get to your quality threshold, you'll have the one best answer to your question.
To paraphrase Apple PR spin, you're thinking this wrong. HB is reducing massive file sizes with minimal losses of quality at the default setting. If you check the file size of the original BD vs. the version rendered by HB, you'll likely see a big cut in file size without a noticeable loss in quality. Sometimes that's going to yield a 20+GB file size for some movies and other times it might squeeze something down to <3GB. It's taking the amount of "pad" pages necessary to store the movie detail so that a viewer will probably not be able to see a difference.
What you're wanting is much more compression without a loss of quality and that's not available. HB is doing as good as it can leveraging THE codec for that exact task. To get smaller file sizes, you have to start sacrificing quality, or resolution, or color depth, etc (all of which could fall under "quality"). The next gen codec- H.265- has some claims (or maybe spin) claiming it can compress much better than H.264 such that it can yield up to 50% smaller files without a loss of quality. But H.265 is not fully here yet and both Apple and the HB crew will need to make changes to embrace it when it is finally adopted. In the meantime, H.264 is THE way and it's either maximum quality OR minimum file size. There is no MAX Quality AND MIN file size solution much better than what you can get out of HB.
