I don't think the OP is saying the rubber is pulling on his hair. He's saying that his hair is getting caught in the "clasp" or whatever its called on the sport strap whilst trying to put it on one handed.
I though apple said the sport strap was soft and smooth, did not think it would pull hairs. Looks like ill have a hairless wrist.
I didn't notice before but I don't have hair on the bottom side of my wrist where the band connects. The top yes and some on the sides but nothing on the bottom to get caught. Lucky me.