Sadly, we live in an age where your look defines you... in other peoples' eyes, anyway. (Come to think of it, I suppose it's not just our present age; it's pretty much always been the case that the 'clothes make the man'). You may not intend to look "goth", but obviously your manner of dress is what people associate with it. Since you apparently aren't presenting a traditional Scottish Highlander type image, "goth" is what people default to in their minds.

Personally, given your description, I'd probably think "punk rocker" before "goth", but that's probably because I don't see any "goths" where I live.

Did you take time to converse with these people and politely correct them?