The hamlet name derives from the Old Englishhlose-leah, and means 'pigstye-clearing'. It is often said that the hamlet was named after the Loosley family (fairly prolific in the area) but it is, in fact, the other way round - the family was named after the place.
The Domesday book talks of the large number of pigs at (Princes) Risborough. Pigs were often found in clearings at the edges of the woods, the animals allowed to wander through sectioned areas of the woods. The row of piggeries in the lea of the woods became the start of a settlement called Loosley Row. Loosley Row stretched from Brimmer's Farm, east of Princes Risborough, round the side of the hill to the hamlet we know today. It included Wardrobes farm and Wardrobes House.