Local lore has Cappahosic House built on property patented by John Stubbs in 1652 and 1702. While the Stubbs family owned the property in the mid-18th century, it is unclear if someone from the Stubbs family actually built or lived in the house in 1751. The Stubbs family remained in possession of the property until 1787.
Of classic Georgian design, Cappahosic house was likely built around 1751 on the York River in Gloucester County. The historic marker claims the house was constructed in 1712, but there is no documentary evidence for this date (however, an archaeological dig in the basement produced items from the 1720s, which adds to the mystery). It was likely built by descendents of John Stubbs. The house was near a popular ferry and was likely the home of the ferry's operator. In 1857 the house was deeded to John C. Baytop and was known after that as the "Baytop House."