As with the Manor house, the gardens at Sylvester Manor have seen great changes over the centuries as dictated by evolving tastes and needs. The garden planted by the Sylvesters in the 1650s would have included vegetables and herbs, and records show that an orchard of fruit trees was laid out at the northeast corner of the current garden site. Although ornamentals would not have been a priority at a site so raw, remote and dedicated to commercial food production, tradition has held that Grizzell Brinley Sylvester brought to Shelter Island the first boxwood introduced in America – in spite of there being no evidence to support that claim. So while when and by whom it actually was added remains speculative, Sylvester descendants clearly embraced boxwood as an almost iconic manor plant, taking cuttings to root at their own homes, and including it many times over in various renovations to the garden landscape.
The current 2-acre garden is what remains of Cornelia Horsford’s grand 1908 reshaping of the property as a Colonial Revival showpiece. Over the 18th century axial plans laid out by Brinley Sylvester, who had planted the gardens for esthetic and commercial use, Cornelia cut back and transplanted the box hedges, built terraced and parterre flower beds, and, perhaps in homage of the Sylvester’s Amsterdam origins, included a sunken Dutch flower bed, accessed by a graceful staircase descending from the rose garden. In the northeast corner of the former rose garden still stands the 18th century privy (or outhouse), painted the same soft yellow as the house, and paneled inside with wainscot removed from the Ladies parlor during a late 19th-century remodeling.
By 1950, Alice Hench was already a passionate gardener at her family’s Dering Harbor property when she met Andrew Fiske, and came to humorously credit their marriage on a shared love of tending plants: “Mr. Fiske came to visit one weekend and said, ‘You have some very fine boxwoods. Why don’t you bring them to Sylvester Manor?’ By Monday, we were engaged.” Thus Alice Fiske began a long and famous tenure as mistress of the manor gardens, often driving the tractor herself. She indelibly left her own mark on the Manor gardens with massive plantings of her signature daffodils and well-tended roses, while faithfully caring for the beds laid out by Cornelia in 1908.