To each their own, I guess. A collared shirt with tie isn't casual. Not even with a sweater or sweatshirt pulled over it. In the Pacific NW of the US, it's a bit on the dressy side. Might be fine for someone who kilts only on special occasions, but, not fine for someone like me, who wears a kilt pretty much every day, including to classes at the local community college, where I'm probably the best dressed out of all the male students, even though I usually am wearing a t-shirt with my kilts.

-J