James Buchanan actually said...
In the latter I state that--The executive government of this country in its intercourse with foreign nations is limited to the employment of diplomacy alone.
Context
Special Message
02/17/1859