Theodore Roosevelt actually said...
We do not have to propose to 'turn the rascals out', for we have shown in very deed that whenever by diligent investigation a public official can be found who has betrayed his trust he will be punished to the full extent of the law.
Context
Remarks Accepting the Republican Nomination for President
07/26/1904