I think he's banging his head in frustration as a joke. A lot of hard-core Colombo junkies get frustrated having to explain about the first name over and over again. Regardless of what you saw, Lieutenant Columbo cannot possibly have a first name.
The creators of the character never wanted you to see him at home or even to see his home; They didn't want you ever to see his wife despite his constant talking about her; And they certainly didn't want you to ever know his first name. It's a gimmick that the creators and producers used from the beginning of the show.
So if you ever happen to see an ID with Columbo's first name on it or something like that, it's only because an art director randomly picked a name and/or a director made a mistake and accidentally showed the prop too clearly. Columbo is a fictional character and he was never given a name by his creators.