He didn't let a perpetrator get away with anything, because his first duty was to process a confession to the crime. His claim on Grace was only circumstantial anyway. He had NO iron clad proof, just probable cause to arrest her on suspicion.
In my view, that closing scene made the episode and among the most poignant ever produced: