Certainly not an issue with me. I've received a "Semper Fi" from other US military, foreign military, and civilians.

Seems many view this as a way to say "hi" or "thanks" to a Marine. I once had an Army gate guard use it. I heard him say "Airborne!" to the vehicle in front of me so took it as a sign of respect.