Andrew Johnson actually said...
Congress may, under the Constitution, admit new States or reject them, but the people of a State can alone make or change their organic law and prescribe the qualifications requisite for electors.
01/28/1867