The Elms was originally the family home of William and Sarah Powell, built in 1896 to a design by T. P. Worthington. Elmslie School was founded in 1918 by sisters Elizabeth, Polly and Peggie Brodie. The school, called Ellerslie School, which occupied the former home of the Powell family, had just 11 pupils to start with; Elizabeth Brodie was the first headmistress. She remained headmistress until 1952. The school moved to the Elms in 1922 and changed its name to Elmslie Girls' School. In 1941 the school became a day school and by 1945 had 353 pupils. The Diocese of Blackburn's Board of Finance ran the school from 1948. In 1987 the school became an associated Woodard school. Elmslie closed in 2000.
The Elms is a two-storey house built on an L-shaped plan, of red brick, with a sandstone front façade. There are three bays at the front, the left-hand of which is gabled. Both the left and right bays have small turrets on either side of the upper windows; the left turrets are topped with pinnacles, the right with crenellated caps. The central front entrance has a porch with marble columns. Internally, many of the building's original fittings are intact.
During the building's use as a school the following additions were made to the original house: gymnasium and assembly hall (1938), more classrooms in the Fisher block (1952), a chapel (1959) and VI Form Centre and music rooms (1988).