James Buchanan actually said...
By what authority have they inquired into the causes of removal from office, and this from the parties themselves removed, with a view to prejudice his character, notwithstanding this power of removal belongs exclusively to the President under the Constitution?
06/22/1860