It is located just off the Commercial Road (A13) approximately one mile (1.6 km) east of the City of London. The square is currently laid out as a formal garden with mature trees and planting and is surrounded on two sides by elegant early 19th century townhouses.
Arbour Square was laid out as a garden enclosure in 1819. By 1830, contemporary maps show buildings on all sides of the square and many of the surrounding roads. The south and west sides of the square are still made up of the original three and four storey Georgian townhouses which are Grade II listed. The east side was demolished and replaced by the Raine's School building (also Grade II listed) in 1913, while the terrace on the north side was replaced by a block of flats in 1937. The Raine's School building now houses part of Tower Hamlets College. The school building was entirely refurbished in 1994 by Sprunt Architects, featuring a new open learning centre within the listed heart of the building.
Running north from the square on the west side is West Arbour Street; and running north on the east side is East Arbour street which is a smaller two storey Georgian terrace. The former Arbour Square police station which once held the Kray Twins and alleged Provisional Irish Republican Army terrorists is located between these two streets; it closed in 1999 and has recently been redeveloped into flats.
Arbour Square is mirrored on the opposite side of commercial road by Albert Gardens, and impressive and almost fully intact late-Georgian residential square made up of Grade II listed houses.
Originally named Albert Square, it was built in the early 1840s, and renamed Albert Gardens in 1937. In 1899 there was a proposal to build houses on the central garden in the square but the London County Council compulsorily purchased the land and opened Albert Square Garden to the public in 1906. In the north of the garden is a 19th-century drinking fountain topped by a statue, 'Shepherd Boy', which was erected in 1903 by the Metropolitan Public Gardens Association 
Together with the neighbouring Havering Street which contains smaller but also Grade II listed houses, and several other early 19th century terraces on the Commercial Road, Albert Gardens and Arbour Square became part of the Albert Gardens Conservation area in 1969.
Havering Street, Arbour Square, and Albert Gardens are some of the last remaining original late-Georgian houses in the E1 postcode area of London's East End.
- Greenwood's Map of London, second edition (1830)
- London Borough of Tower Hamlets Conservation Area Character Appraisals and Management Guidelines
- O’Neill, Sean; Nicola Woolcock (2005-01-17). "Squatters find police station is good home without any charges". Times Online. London: News International Group. Retrieved 2008-04-06.
- [London Gardens Directory http://www.londongardensonline.org.uk/gardens-online-record.asp?ID=THM001]
- [Albert Gardens Conservation Area http://www.towerhamlets.gov.uk/idoc.ashx?docid=5def9caa-7d71-4143-8832-5dcfd8eb09f2&version=-1]
- Historic England. "East Side:Raine's Foundation School (205714)". Images of England.
- Historic England. "South Side:Nos 50 to 60 (205715)". Images of England.
- Historic England. "West Side:Nos 5 to 15 (205716)". Images of England.
- London Gardens Directory http://www.londongardensonline.org.uk/gardens-online-record.asp?ID=THM[permanent dead link]
- Albert Gardens Conservation Area http://www.towerhamlets.gov.uk/idoc.ashx?docid=5def9caa-7d71-4143-8832-5dcfd8eb09f2&version=-1