How badly do you want to keep the job? If you could afford to lose it then you could press the issue. He sounds insecure in his 'manhood' and threatens to check the rules if you wear it again. Sounds like a challenge to me!