Andrew Jackson actually said...
Nor do I claim the right in any manner to supervise or interfere with the person intrusted with such property or treasure, unless he be an officer whose appointment, under the Constitution and laws, is devolved upon the President alone or in conjunction with the Senate, and for whose conduct he is constitutionally responsible.
Context
Message to the Senate
04/20/1834