In my experience, people will often notice if you're wearing one, and some pins may attract more interest than others.

On the other hand, again in my experience so far, no one has ever noted the absence of a kilt pin.

My own usual preference is to not wear one.