Benjamin Harrison actually said...
I have endeavored to hold up the law as the one single admissible rule of conduct for good citizens.
Context
Letter to the Virginia State Baptist Convention on Lawlessness in the Southern States
05/20/1892