The glengarry is of military origin, and some folk will say that only soldiers should wear them.
I like wearing mine, and even took the step of replacing the black Hanoverian cockade with a white Saltire one.

Just like with the kilt, if you choose to wear it, wear it respectfully and with pride and you'll be fine.