Fred Gray, the son of prominent New York doctor Dr. John F. Gray and Elizabeth Hull-Gray, was born in New York in 1840. His two siblings died before they reached adulthood.