Assume? No. Conclude? Maybe. If by this time you mean "After the Civil War, but before the wings were built".
Grover Cleveland was the only President to leave the White House and return for a second term four years later? He was also the only President to be married in the White House?
Verum audaces non gerunt indusia alba. - Ipsi dixit MCMLXXII