I have used a bent safetypin to hold the loop shank buttons on my jackets so the buttons can be removed and changed easily. It is not great for buttons because the pin can fold the fabric a little, and also show. It might work out for a cap badge though...

I'm trying to come up with a different design of safetypin for the buttons that does not require the tiny buttonhole, and also does not cause the fabric to fold.

I've also heard of split rings rather than cotter pins being used with that kind of shank on removable buttons.