Why not a simple "group"? A Group of kilted men, a group of men in kilts. It seems to leave nothing for anyone to be confused about.
A 'rabble' sounds like a put down or an implied insult.

Chris.