In the 16th century the main house on the island was inhabited by the MacLeods of Lewis, and it was here that Roderick Macleod of Lewis ('Nimheach' - the venomous) implemented his plan to ensure that his grandson would inherit Raasay and the lands of Gairloch. He convened the two powerful families (names do not appear to have been recorded) of the time at Isay house for a banquet which he suggested was to inform them of good news. During the feast he invited each person present to accompany him outside of the banquet hall in order to inform them of this news. Upon leaving the hall the victim was promptly stabbed to death. In this way both families were wiped out.
The island was home to a small fishing community in around the 1830s, which included a general store and fishing station, and by 1841 supported 15 families "in considerable comfort". However, like many other small islands, the population of around ninety were removed during the Highland clearances to make way for sheep. The remains of the 18 or more cottages can be clearly seen from the Waternish peninsula, which overlooks the three islands.
The island was briefly owned by the singer Donovan in the 1960s.