If you have the light cotton ones in your pockets, you can match, but under no circumstances exceed those at the event of higher rank than yourself.

Never with anything less than a black Argyle jacket, and when you shake hands remove or not to match the person whom you are greeting. Traditionally you would never remove your gloves to greet a lady, but I notice a lot of folks forgoing that tradition too. It's shaky ground and a slippery slope.