I would say it all depends on the venue and the level of casual, what exactly is your definition of casual that is the key as I see it. You may want to look at the 10 looks thread where Jamie and I took one kilt each from casual to white tie, it gives several different levels of casual. to me anyway casual can go from this



To something like this



Depending on the occasion I would not feel out of place in either outfit, of course I would not wear the t-shirt to a wedding but also I would not dress in the jacket and tie for a barbecue.