I've known two men that changed names later in life: one took his wifes surname when they were married, and the other took his mothers' maiden name. Both were for similar reasons to yours. So, it may be more common than you think.