James Buchanan actually said...
The practice of making such appointments, whether in a vacation or during the session of Congress, has been constantly followed during every Administration from the earliest period of the Government, and its perfect lawfulness has never to my knowledge been questioned or denied.
01/15/1861