Isn't there a strong hint that it is from the Great Santini episode, Now You See Him. The quote about being not much to look at (the first one) seems to me as though it could have been said to the Great Santini...
You see what he does Paul?...do you see?....Headache comes in and out of his cave and then proceeds to leave us in the lurch.....torturing us to the bitter end!!!
I thought about Now You See Him...great answer...but where and when?...I don't know.......Headache!! the torment must end!!
"Well, there you are sir. You see, I'm not as dumb about these things as I thought."
Yes, it comes from "A Matter Of Honor". Columbo says it to Luis Montoya while the two of them are in the stable discussing the sort of "stick-thingy" that victim Hector Rangel would have had with him out there in the bullring.