If you like the badge, then keep it; it's your kilt. Perhaps you could brainstorm for ways to weigh down that area of the kilt; maybe you could sew in a small weight under the badge itself. I just think that a pin AND a badge would simply be too much.