I don't believe you'll get in any trouble; however, you will probably have a repeat of what happened to you. Military folks are fairly possessive of their symbols and the Ranger beret is one of those.

From a distance, your balmoral may be confused for a Ranger beret, and Rangers may approach you, thinking you are one of them. So, be prepared to explain.