Benjamin Harrison actually said...
I have rather felt that the flag should be at the peak, because those whose dying we commemorate rejoiced in seeing it where their valor placed it.
Context
Remarks During Decoration Day Ceremonies in Independence Hall, Philadelphia
05/29/1891