James Buchanan actually said...
I certainly had no right to make aggressive war upon any State, and I am perfectly satisfied that the Constitution has wisely withheld that power even from Congress.
Context
Special Message
01/07/1861