Though many details about his early life are in dispute, scholars agree that Arnold was born in New York City and that his father may have been the Reverend George B. Arnold. The family relocated to Illinois and then to Monmouth County, New Jersey where Arnold enjoyed a country upbringing. Though he apprenticed himself to a portrait painter in New York in 1852, Arnold soon determined that literature would be his true calling.
Charles Kingsley was one of several men who were associated with the Fred Gray Association, "a loose confederation of young men who seemed anxious to explore new possibilities of male-male affection" (Folsom & Price “Walt Whitman”). E. Miller explains that “during this period Walt [Whitman] was intimate with a group he called the Fred Gray Association. Not much is known about this circle. Gray, the son of a noted New York physician, took a medical degree after the war. Nathaniel Bloom became a successful merchant. Hugo Fritsch was the son of the Austrian consul.