Andrew Johnson actually said...
On the contrary, candor compels me to declare that at this time there is no Union as our fathers understood the term, and as they meant it to be understood by us.
Context
Third Annual Message
12/02/1867