Born second in a family of nine children in Maine and schooled in Boston, Willis attended Yale and traveled extensively in Europe. His sister was Fanny Fern, a member of the Pfaffian crowd.