So...you only wear your gloves around other people?
Just think a little bit can you? When you enter a supermarket were you touch a lot of things such as gates, the cart, basket, and so on .... You really think it is a good idea not to use any protection?
When walking outside when I don't have to touch anything, I do not need to wear gloves. I do wash my hands and other precautions when I get home.