Andrew Johnson actually said...
Congress may, under the Constitution, admit new States or reject them, but the people of a State can alone make or change their organic law and prescribe the qualifications requisite for electors.
Context
Veto Message
01/28/1867