I don't think it was the fact that you were wearing a sweater, but that sweater in particular. It does look a little odd. The thing on the shoulders make it stand away from the body at an odd angle and when I looked at the picture that is what I saw first.
That's all I can say.