Abraham Lincoln actually said...

a full pardon is hereby granted to them and each of them, with restoration of all rights of property, except as to slaves and in property ca...

To now abandon them would be not only to relinquish a lever of power, but would also be a cruel and an astounding breach of faith.

This recommendation is specially made in order to comply with the requirements of the ninth section of the aforesaid act

so far as it relates to State governments, has no reference to States wherein loyal State governments have all the while been maintained.