Millard Fillmore actually said...
My own views of the national liability upon this subject were expressed in the note of the Secretary of State to Mr. Calderon of the 13th November, 1851.
Context
Special Message
06/13/1852