Oliver Culver was born in 1778 in Connecticut and came to Rochester, New York as a surveyor. One of the Rochester's early pioneers with diverse interests, Culver undertook surveying, road building, fur trading, shipbuilding and worked as a coroner, businessman, politician, and community leader. His legacy for the city of Rochester includes establishment of St. Luke's Episcopal Church on Fitzhugh Street, laying down the East Avenue in the heart of the national historic district, and early log-school house near the Indian Landings. He built one of the first canal boat and shipping vessels for a trade route connecting the Great Lakes along the canal system. Later in his life, he also served in State Legislature.
The house was built by Oliver Culver on the southwest corner of East Avenue and Culver Road within the East Avenue Historical District East Avenue Historic District. The original part of the house was built in 1805 as a smaller frame house. In 1816, a new house was built in front of and attached to the older house, the structure that largely remains today. The house was used as a tavern which featured both the front entrance for men and a separate side entrance for women. On the second floor, there is a ballroom with an elliptical ceiling adapted from ship hull design, two symmetric mantles, and a springboard floor constructed to accommodate large number of occupants. The Oliver Culver House has been described as "one of the most charming houses in the Genesee Valley"
In 1906, the owner of the house at the time, third-generation Caroline Culver and her husband relocated the house from the corner of East Avenue and Culver Road to its current location on East Boulevard by East Avenue, about 1,000 ft in distance along the East Avenue. During this move, the old part of the house in the back was partially destroyed. This part was later restored and extended to enlarge the room that is now the library of the house. The house exemplifies the federal neoclassic elements such as slenderness, symmetry, and restraint typified by the American house builders of the era.
From its construction in 1816 through 1945, three generations of Culver family resided in the house. In 1945, the house was purchased by Elizabeth Gibson Holahan to be used as her residence. She served as president of the Landmark Society of Western New York (1952-1961), and as president of Rochester Historical Society (1977-2000). She served as director of the restoration of the Revolutionary War General Philip Schuyler Mansion (1764), supervisor of the excavation and restoration of Stone-Tolan House - the oldest house (1792) in Monroe County, director of the restoration of the Patrick Barry Mansion (1857) and undertook relocation and restoration of the George Eastman's boyhood home from Waterville to the George Eastman House.
- "Oliver Culver House, 70 East Boulevard, Rochester, Monroe County, NY | Library of Congress". Loc.gov. Retrieved 2018-02-05.
- "Oliver Culver: 1778–1805 Brighton's Man for all Seasons". Historic Brighton. Archived from the original on 2018-02-06. Retrieved 2018-02-05.
- Schmidt, Carl F. (1975). The Arts in the Genesee Valley, Vol 4: The Early Architecture of the Genesee Valley. Geneseo, NY: The Genesee Valley Council on the Arts. pp. 75–76.
- Comstock, Helen (1958). 100 Most Beautiful Rooms in America. NY, NY: Viking Press. pp. 176–177.
- The Fine Estate Antiques Collection of Elizabeth Holohan. Rochester, NY: Cottone Auctions. 2005. pp. 2–3.