Having a tough time with this.
Bands with lugs make me lean toward the smaller one, but sport bands and loops make me lean towards the larger one. My wrists are roughly 165mm.
My wrist is the exact same size and I decided to upgrade from a 42mm S3 to a 44mm S5. I personally think that 42mm was the perfect size for me but I am very happy I went with the 44mm S5 over the 40mm one.