Andrew Johnson actually said...

Thus far and upon this vital policy there was perfect accord between the Cabinet and myself.

This language is very significant.

Under a sense of public duty I am compelled to deny your right under the Constitution and laws of the United States.

Public considerations of a high character constrain me to say that your resignation as Secretary of War will be accepted.