Mike is right in that he probably should have thrown both the gun and gloves overboard.
But he could half hide the gloves (as he did) and threw them overboard later.
Columbo, although was first suspicious because of the feather from the bed pillow.
But Abe is right because if the story was written differently with the gun and gloves going overboard, the writers would have to have concocted other things to give Columbo the necessary format leading to a solution.
Of course, this episode, as others, really involved a trap where columbo forced Danzinger to implicate himself by now producing the 2nd pair of gloves with no reasonably defendable purpose. Not really direct proof..but enough for us, the great masses.