Sure wear what you want as long as you are happy.
To be honest most kilt wearers wouldn't know the difference from one pair of hoes and another.