I tend to have problems with my shirt sleeves moving up inside my jacket. That said, I happen to notice when others adjust their clothing. Whether it is too often or not often enough. But I like to see a gentleman (or a dude, if Alan H has dared to read this far) make minor adjustments to his appearance. Since, as others have stated, minor adjustments are almost always needed at some point.

For myself, I try to be subtle, yet purposeful when adjusting my sleeves-be it shirt or jacket.