I've only been wearing kilts for a year, so that's the basis of my opinion as I differ with others here who present themselves quite handsomely.

If you like the belt buckle, keep wearing it. I wear big buckles like yours with all my kilts. Heck, I used to wear big buckles like that with my blue jeans. So counting you and I, there are at least two of us that don't mind wearing a nice buckle casually.

Abax