James Milson

Postcard of Milsons Point. The caption reads: "Milson’s Point – Probably in the early Seventies [1870s]. Kirribilli on the farther shore. Eastern Road Wharf is front. Alfred St to the right, the Arcade and clock are within (illegible)ing distance of the spreading tree."

James Milson (1785–1872)

James Milson
Born	James Milson (1783-11-25)25 November 1783 Grantham, Lincolnshire, England
Died	25 October 1872(1872-10-25) (aged 88) Milsons Point, New South Wales, New South Wales, Australia
Nationality	English
Occupation	Farmer
Known for	Milsons Point named after him
Spouse	Elizabeth Kilpack (1793–1850)
Children	Sophia Milson (1811–1877) James Milson (1814–1903) David Milson (1817–1890) Elizabeth Milson (1819–1872) John Milson (1821–1891) Robert Milson (1824–1886)

James Milson (25 November 1783 – 25 October 1872) was an early settler on the North Shore of Sydney, Australia.

He was born on 25 November 1783 at Grantham, Lincolnshire, England and died at the age of 88 on 25 October 1872 at Milsons Point, Sydney, New South Wales, Australia.

23-year-old James Milson arrived in Port Jackson (Sydney) on the Albion on 19 August 1806 as one of the earliest free settlers in the Colony of New South Wales. His motivation for immigrating was the same as the many who followed him, the promise of free land. He was a native of Lincolnshire experienced in farming and was welcomed by the colonists, desperate for men with agricultural knowledge. He married in 1810 and subsequently raised a family of 6 children.

Milson did well in the Colony of New South Wales, and established a number of prosperous businesses, which included supplying ships with stone ballast, fresh water, and the produce of his dairy, orchard, and vegetable gardens.

In his own words, in the years before 1825 Milson had "principally resided in District of Parramatta" (more specifically in the area in the District of Parramatta then called the "Field of Mars").

In the early 1820s Milson settled in the District of Sydney in the vicinity of today's Jeffrey Street, Kirribilli, a Sydney suburb on the North Shore of Sydney Harbour. He was a prominent resident of the area for more than 50 years until his death in 1872 at home at "Gratham" in the modern suburb of Milsons Point in what was then called the Municipality of East St Leonards.

Milson and his sons and grandsons built several homes in the area: "Brisbane House" (James Milson), "Grantham" (James Milson), "Fern Lodge" (James Milson), "Wia Wia" (John Milson), "Elamang" (James Milson Jnr), "Coreena" (Alfred Milson, son of James Milson Jnr), and "Wayala" (Arthur Milson, son of James Milson Jnr). Milson's son-in-law William Shairp also built "Carabella". Fern Lodge (heritage listed) and Elamang (in the grounds of the Loreto Convent) were still standing in 2008.

Milsons Point, the headland into Sydney Harbour on which the north pylons of the Sydney Harbour Bridge stand, and the suburb on this headland to the west of Kirribilli, are both named after James Milson. A retirement village at North Sydney that includes a nursing home is also named after him.

In addition Milson Island in the Hawkesbury River (previously known as Mud Island, renamed in 1976), Milson Passage the passage in the Hawkesbury River between Milson Island and the southern bank (previously known as South(ern) Channel or The Gutter, renamed in 1976), and Milson Passage the suburb on the southern bank of the Hawkesbury River opposite Milson Island (previously known as Prickly Point, renamed in 1995) are all named after Milson's son Robert Milson (1824–1886).

Surname

In many records James Milson's surname is shown as "Milsom". "Milsom" was the surname by which James signed letters, in which legal proceedings were transacted, and which appears on early maps such as the 1840s map shown below. By the time that James died in 1872 he was known as Milson, the name by which he and his family are known today.

Family tradition

In the words of the North Sydney Historical society:

James Milson is a controversial figure in local history, having made many claims during his lifetime (mostly in attempts to claim land) which either cannot be substantiated, or in some cases have been positively disproved

There is a Milson family tradition which includes that:

• (1) James Milson was "promised" all the land on the North Shore of Sydney Harbour between Hulk (Lavender) Bay and Rainbow (Neutral) Bay by Governor Philip Gidley King. Despite Milson's wife's continuous urging, he did not press the Governor for a written-out grant.
  • However, James Milson did not arrive in Port Jackson until after Philip Gidley King had ceased to be Governor on 12 August 1806. James Milson arrived on the Albion on 19 August 1806 during the period that Captain William Bligh was Governor from 13 August 1806 to 26 January 1808 (until the military coup by the NSW Corps). James Milson's arrival and his marriage on 8 January 1810 were both events that occurred after Philip Gidley King ceased to be Governor of New South Wales. Therefore there is no way that Philip Gidley King could have made Milson a land grant, or that Milson's wife could have urged Milson to press Philip Gidley King for a written-out grant. Milson marriage occurred during the time that Lachlan Macquarie was Governor of New South Wales. Lachlan Macquarie was Governor from 1 January 1810 to 1 December 1821. James Milson was instead given land by Governor Macqaurie. In James Milson's memorial to Governor Macquarie dated 5 June 1820 it was stated that James Milson was living with his family at Pennant Hills "following agricultural pursuits" on a 100-acre grant previously given to him by Governor Macquarie.
• (2) Milson was shown a plan delineating the land in his name and that this sufficed as legal ownership in the early colony.
  • However there is no legal precedent for this ascertion.
• (3) Milson built his house on "Milson's Point" shortly after his marriage in 1810.
  • However in the years after his marriage James Milson lived at the "Field of Mars". in addition to the memorial of 5 June 1820 establishing residence for himself and his family at that time at Pennant Hills (in the "Field of Mars", district of Parramatta), in Milson's own words in the years before 1825 he had "principally resided in District of Parramatta". He did not build a residence at Milsons Point in the District of Sydney shortly after his marriage in 1810.
• (4) No actual acreage was stated and this fact weakened Milson's case in the subsequent legal dispute between himself and Campbell over title to the land that included Milson's Orchard.
  • However this case did indeed mention acreage, and was not about who owned the land. Milson lost his case against Campbell based on non-payment of a lease agreement between himself and Campbell. The case mentioned that Milson leased a farm of 120 acres (48ha) from Robert Campbell. In this case Milson did not purport to own the land that he was leasing. Milson instead referred to Campbell's absence of proof of title which made it impossible to ascertain the boundary between the 120 acres (48ha) that Milson leased from Campbell, and the adjoining "very large portion of land" (acreage not stated in the case but known to be 50 acres (20ha) that Milson had received as a grant from Governor Sir Thomas Brisbane in August 1824. It is true, however, that Milson did dispute Campbells ownership of the land during the nearly 3 years from 1828 that he ceased paying rent. During this time Milson wrote to the Surveyor-General and Governor Ralph Darling in an attempt to have the land recognised as his own instead of Robert Campbell's. The reply that Milson received on 13 March 1829 was that he could not obtain the land as it was the property of Robert Campbell, and that the land that Milson owned (the adjoining 50 acres) was already measured and marked out.

Versions of this family tradition, or parts thereof, have been repeated as fact in many publications about the life of James Milson. The most common part of the family tradition that is repeated is that the court case between Campbell and Milson was a dispute over land title. Examples of repetition of the family tradition, or parts thereof, are found in the Australian Dictionary of Biography, the publication by Elizabeth Warne, the publication of R.H. Goddard, publications of the North Sydney Council and the North Sydney Historical Society, and the Dictionary of Sydney.

Newspaper report that contains the family tradition

An example of a version of the family tradition follows from the Sydney Morning Herald of 19 May 1926.

Sydney Morning Herald, 19 May 1926	Evidence from documentation
AN OLD FAMILY THE MILSON STORY (BY R. H. G.) The death recently of Mr. Arthur James Milson at his residence, Walaya, Milson's Point, removes another link connected with the early days of New South Wales.
His grandfather, James Milson, arrived in Sydney as a free settler in 1804	James Milson arrived aboard the "Albion" on 19 August 1806.
with letters of introduction to Governor King from several influential friends in England.	Governor King was Governor when James Milson sailed from England but was not Governor by the time he arrived.
On arrival he built a small cottage on the North Shore opposite to Dawes Battery and here cultivated extensive gardens and an orchard at Castle Hill for the supply of vegetables and fresh fruit for the ships in Svdney Cove.	Milson did build a small cottage "on the North Shore opposite to Dawes Battery" but it was in the early 1820s, not "on arrival" in 1806. In the early 1820s he also built there a dairy, orchard and vegetable gardens to supply the ships in Sydney Cove. Milson did not have an orchard at Castle Hill. (The orchard at Castle Hill was owned by Milson's son Robert who in 1867 was advertising the sale of his crop of oranges from his Castle Hill orchard, and advertising the sale of this orchard in 1868.) Milson had grazing land at Pennant Hills, which is close to Castle Hill.
On receiving a “promise of the land fronting Sydney Cove" he remarked that the area was not fit to grow anything as it was nothing but sandstone. The Governor reasoned with him that although he would not be able to grow much then the stone on the ground would certainly be valuable in years to come for building purposes. A prophecy that was true indeed.	The Governor being discussed here is Governor King who was not Governor during the time that James Milson was in the Colony. This conversation with Governor King has no basis in fact. Milson did receive land "fronting Sydney Cove", a 50-acre grant from Governor Brisbane in August 1824 that fronted both Lavender Bay (then known as Hulk Bay) and Careening Cove (see MAP 4 in this article). He established his dairy & orchard on land leased from Robert Campbell (see MAPS 1-3 in this article).
James Milson held the position of confidential land steward to Governor Macquarie, and had the superintending of the Government Farm (now Botanic Gardens).	This has no basis in fact. Lachlan Macquarie ceased being Governor on 1 December 1820. In June 1820 James Milson had a memorial written to Governor Macquarie stating that he was living at Pennant Hills on a grant that Governor Macquarie had given him and requesting more land. A referee for this memorial was Robert Campbell. None of this agrees with Milson being a "confidential land steward" or "superintending of the Government Farm (now Botanic Gardens)" during the time of Governor Macquarie.
Governor Brisbane having known Milson in England, they were fast friends, and he held a similar position in Sir James's household.	It is not known if James Milson knew Governor Sir Thomas Brisbane (his name was not Sir James) in England or if they were fast friends. It is true, however, that James Milson was a steward (not a land steward) and keeper of Government House in Sydney during the time of Governor Brisbane. Milson's salary as keeper of Government House from the period 22 August 1823 to 27 February 1824 had been £89. In addition his wife received an assigned convict servant, John Wallans (Agamemnon, 1820, 7yrs) on 24 November 1823 when she is recorded as Mrs Milson of Government House. James Milson himself received an assigned convict servant on 17 September 1824 when he was recorded as John (sic) Milson of Government House. The servant he received at that time was Edward Sharley ("Martha", 1818, 7yrs). Then on 1 November 1825 James Milson was again recorded as a Steward of Government House.
James Milson married Elizabeth, a daughter of David Kilpack, a member of the New South Wales Corps, and one of the first grantees of land in the Paramatta district, and an early sheep breeder of the colony.	James Milson did marry Elizabeth the daughter of David Kilpack but most of the rest is incorrect. David Kilpack had arrived in the colony as a convict in the First Fleet aboard the Scarborough in 1788. He was never a member of the New South Wales Corps. He did receive an early grant in the Parramatta district receiving his first grant of land in 1794. He died in 1797 after having earlier sold his farm to John Macarthur. He was not involved in sheep breeding like John Macarthur and his friends.
"King's Despatches" mention the number of sheep in the colony in 1805 to be 20,617, and the sheepbreeders were S. Marsden, John Macarthur, R. Hassall, Edward Lamb, T. Rowley, David Kilpack, Edward Robinson, James Sheppard, and Thomas Arndell.	It is well known from Australian History that Samuel Marsden, John Macarthur, Rowland Hassall, Thomas Rowley, and Thomas Arndell were involved in early sheep farming in the Colony of New South Wales. As an ex-convict David Kilpack did not move in this social circle, and did not have enough land on which to graze sheep. Other names are associated in some way with the Milson history. For example James Sheppard (sic) was a man with whom James Milson had a dispute over land in the early 1820s; Edward Lamb had worked for Robert Campbell’s nephew Robert Campbell Jnr, and James Milson Jnr had later been a partner in the firm of Robert Campbell Jnr & Co. It is not known who the name Edward Robinson could be referring to, though there was an Edward Robinson who had arrived as a convict on the "Admiral Barrington" in 1791 and who had farmed cattle in the Hawkesbury district. That David Kilpack was a sheepbreeder, or was included in "King's Despatches" with the likes of Samuel Marsden, John Macarthur, Rowland Hassall, Thomas Rowley, and Thomas Arndell has no basis in fact.
Milson's house on the North Shore was the rendezvous of many such gentlemen as Dr. McLeod, Dr Ivory Balcombe, Brabyn, Sir James Brisbane and Judge Bent who would be found there on Sunday afternoons.	It is not known if this has any basis in fact. "The Monitor" of 8 December 1826 states that Milson’s farm on the North Shore of Port Jackson was "a well known resort for Sunday-goers" but no names are given. Again the Governor’s name was Sir Thomas Brisbane. Sir James Brisbane was the Commodore of the British Navy who travelled on his ship H.M.S. Warspite which was in Port Jackson from 19 October 1826 to 30 December 1826 with Sir James Brisbane dying during this time on 19 December 1826 while the ship was in the harbour.
In 1826 bush fires raged over the whole of the North Shore and Milson's house was destroyed. Everything was lost including some valuable mares he had recently imported. The occupants were rescued by officers and men from H.M S. Warspite.	There were bushfires that raged in 1826 and destroyed Milson’s house. There was no mention of valuable mares in the newspaper report about the effect of the fire on the Milsons. There was also no mention in the newspaper reports of rescue. The H.M.S. Warspite was, however, in Port Jackson between October 1826 and December 1826 (see above) and there may be some truth on this part of the family tradition as the family retained the name of this ship and the name of its Commodore Sir James Brisbane (even though confusing his name as the name of the Governor of New South Wales Sir Thomas Brisbane).
An account of the disastrous bush fires that occurred on November 28, 1826, appeared in the "Monitor" on December 8 of that year and was recorded as follows:-"It Is our painful duty to record a melancholy instance of the mutability of human affairs. There dwelt upon the North Shore a reputable family named Milsham the heads of whom had long held confidential positions in the households of Governors Macquarie and Brisbane. Having acquired an easy competency they had retired to this sequestrated spot to enjoy the fruits of their labour, a comfortable neatly-furnished dwelling; a garden, cattle, and a well-known resort of Sunday-goers known as 'The Milk House,' which produced a considerable weekly rental, constituted their possessions. Milsham was absent, and his wife and daughter saw with fearful solicitude the distant smoke rising o'er the hill; towards the set of sun the blazing heath presented to their view an appalling prospect. Two male domestics then on the promises made active endeavours to cut off the communication of the flames by clearing a space as ample as time would admit to check its destructive progress, but the attempt was vainly made. The impelling breeze wafted ignited masses through the air and the only recourse now left was to remove the most necessary and valuable portables. The house contained apparel, beds. etc., which wore hastily conveyed to a distance; but the unfortunate owners were compelled to witness the demolition of their peaceful habitation. The small remaining hope of saving the residue of their property was not for long. The destructive element made rapid progress towards the depository of their movables, and quickly was the wreck complete."	The bushfires occurred on 25 November 1826, not the 28th. The report that is quoted from the "Monitor" of 8 December 1826 is not the complete story but was missing the last sentences. Other edits have been made including changing the spelling of the surname from Melsham to Milsham.
The "Monitor," December, 1826, reports other serious bush fires: -"The northern point or the hemisphere presented a vivid glare from the conflagration." A similar fire took place at Woolloomooloo.
Governor Brisbane having taken up his residence at Rosehill (Parramatta), he offered Government House to James Milson until he rebuilt.	It is not known if there is any truth in this assertion.
His new home he called Brisbane Cottage, erecting it on the former site, situated just above the new cutting recently made to the Milson's Point vehicular ferry.	A new cottage was built within the bounds of the orchard that James Milson had established on the land that he leased from Robert Campbell and is seen on a map from the 1840s. It is believed to have been called "Brisbane Cottage" and the home that Milson built in 1831 on his own 50 acres granted to him by Governor Brisbane in 1825 was called "Brisbane House".
The late Mr. Arthur Milson, when speaking of the old days of Milson's Point, recalled the small creek that ran through the garden to the west of Brisbane Cottage, emptying itself over a waterfall into this part of the harbour (known then as Cockle Bay), where ships' boats used to fill their casks when getting supplies of milk, fruit, and vegetables.	This creek can be seen on MAP 3 in this article, a part of a map from the 1840s. The waterfall is on land just outside the area being leased by James Milson in the 1840s.
On the cliff near the waterfall there were several round holes in the rock, and his father told him that the natives in the early days used these holes to mix up a punch, and get intoxicated with the mixture.	It is not known if there is any truth in this assertion.
In these grounds there used to stand three headstones under some lemon trees, marking the position of the remains of the chief officer, surgeon, and one of the crew of the ship Surry (belonging to Captain Raine), who succumbed to smallpox on August 12, 1814. The authorities would not allow the interment In a public burial ground, and hearing this, Mr Milson allowed the internment to be made in his grounds.	The graves are seen in the orchard on MAP 3 in this article, a part of a map from the 1840s. The people buried were 3 typhoid victims from the convict transport Surry. The notation on the map says "Spot where three of the Surry's crew are interred". James Milson was not living on this land as early as 1814, and the graves were already in place before Milson moved there and began to build his orchard. His permission was never needed to be sought to perform the burials on that land. One of those buried is known to have been the ship's surgeon who attended the typhoid victims.
James Milson (Snr) was one of the original members of the first Boating Club formed in 1836; Mr. Burton Bradley was the commodore. The Obituary of James Wilson Jnr from 1903 states that it was instead James Wilson Jnr who was one of the original members of the first Boating Club formed in 1836.
His Son, James Milson junior, was one of the founders of the Royal Sydney Yacht Squadron in 1862. When the late Mr. Arthur Milson came to reside in Sydney he, like many other members of this old yachting family, took a great interest in yachting. He [Arthur] built the yacht Mischief, which was named after the old boat of the same name owned by his father [James Milson Jnr], and which was well known as far back as the early 'fifties. Mr. [Arthur] Milson did not take an active part in yacht racing, but held office in the Royal Sydney Yacht Squadron, first as rear-commodore, afterwards as vice-commodore, and finally as commodore. He [Arthur] retired from the latter position to offer the position of commodore to Lord Forster.	This appears to be factual.
Referring to the naming of the proposed new railway station at Milson's Point (Kirribilli), by the Railway Commissioners, the late Mr. [Arthur] Milson never ceased, up to the time of his death, to draw attention to the obviously unnecessary alteration of the name of Milson's Point.

Early years in the Colony of New South Wales

1806 to 1810: Employment and marriage

Fellow passengers on the Albion with James Milson included Robert Campbell (1769–1846) and his family, and his nephew Robert Campbell Jnr. In the long voyage the Campbells and James Milson got to know each other well. Robert Campbell was a successful merchant who also had farming interests. Robert Campbell owned land that including 900 acres (364ha) at Canterbury within six miles south-west of Sydney where he ran the largest private cattle herd in the Colony, and an undeveloped 120 acres (48ha) on the headland on the north side of Port Jackson that includes both Kirribilli Point and Milsons Point.

James Milson was immigrating to begin life in the Colony as a farmer. Initially before he could obtain a land grant of his own he needed to obtain employment. A court case between Campbell and Milson in 1831 stated that Campbell was a former employer of Milson. It therefore appears that James Milson had a job with Robert Campbell on Campell's Canterbury farm even before stepping onto the shore of his new homeland.

Milson next found employment at the Field of Mars in the District of Parramatta. References do not agree as to which farm he worked on in the area.

On 8 January 1810 at St Philip's Church, Sydney, Milson, describing himself as a described himself as "servant and labourer", married Elizabeth Kilpack (1793–1850), the daughter of David Kilpack (c. 1757 – 1797) and Eleanor McDonald (c. 1754 – 1835), who at the time of her daughter's marriage was the wife of Thomas Edward Higgins (c. 1740 – 1829).

The Australian Dictionary of Biography states that Elizabeth Kilpack was the daughter of Milson's employer, without stating who Milson's employer was. Some interpret this to mean that Milson was working on the farm of Elizabeth's father David Kilpack. This, however, is impossible as David Kilpack died in 1797 which was 9 years before Milson arrived in the Colony, and before Kilpack died he had sold his farm to John Macarthur (1767–1834). Elizabeth Warne says that Milson worked on the farm of Elizabeth's mother, the "widow Kilpack", a farm of 170 acres. This again is impossible as prior to 1816 Elizabeth's mother never owned a farm, and she was also only the "widow Kilpack" for 14 months till her second marriage to Thomas Higgins on 31 January 1799, although she was still at times identified by her maiden name of Eleanor McDonald (1822 and 1825 musters) and her former married name of Eleanor Kilpack (land grant of 1816). (Elizabeth Warne also incorrectly states that Milson worked on this farm before 1810 at the same time as working on his own farm of 50 acres opposite Sydney Cove when he did not receive the grant for this 50 acres until August 1824.) There then only remained Elizabeth's step-father Thomas Higgins. The 1806 muster, however, shows that he only had a farm of 30 acres and records in the Colonial Secretary's Index show that he did not acquire another 100 acres until July 1809.

The Beecroft and Cheltenham History Group states instead that James Milson had worked on Cornish Hills Farm when he first arrived as a free settler in 1806. With the understanding that James Milson's first employer in the colony had been Robert Campbell at Canterbury, this is the most likely scenario of Milson's second workplace at the "Field of Mars". The Cornish Hill Farm was owned by John Macarthur. By 1803 John Macarthur had acquired a total of over 1000 acres at the "Field of Mars" being a number of smaller farms that he called the "Cornish Hill Farms", in an area that extended from today's Dundas through Carlingford and Beecroft to Thompson's Corner in Pennant Hills at the junction of Castle Hill and Pennants Hills Road. On Macarthur's Cornish Hills Farms, which included the farm that had originally belonged to the father of Elizabeth Kilpack, John Macarthur employed farm managers for the various farms and for various farming tasks. Two of these farm managers had been David Kilpack, the father of Elizabeth Kilpack, from the sale of his farm until his death in 1797, and Andrew Murray, a friend and brother-in-law of James Milson, after Murray's arrival in the Colony in 1817.

Other collaborating evidence that James Milson had been employed by John Macarthur includes that:

• When James Milson received his first land grant in 1816 he was recommended for this grant by "H McArthur", the nephew of John Macarthur.
• When James Milson received a subsequent land grant of 100 acres at Pennant Hills in 1821 (he already had 100 acres in that area), it had been 100 acres of the Cornish Hills Farm that John Macarthur had surrendered to the Crown in May 1821, and land that James Milson stated that John Macarthur had requested be granted to James Milson.

• In a dispute with James Shepherd over ownership the 100-acre land grant that had been part of the Cornish Hills Farm, Milson chose John Macarthur as his arbitrator.

James Milson met his wife Elizabeth Kilpack while working on Macarthur's Cornish Hills Farms, as Elizabeth Kilpack was living prior to her marriage at the age of 16 on her step-father's 30-acre farm in the area. Her family also continued to have a close association with the Macarthurs as when her mother received a grant of 60 acres in 1816 the person who recommended for the grant was Mrs Macarthur. Andrew Murray, who married in 1818 a year after arriving in the Colony, also met his wife Eleanor Kilpack, Elizabeth's sister, while working at the Cornish Hill Farms. According to the Beecroft and Cheltenham History Group it is said that Andrew first met Eleanor while out herding cattle at Cornish Hills Farm.

1811 to 1820: Fatherhood and land ownership

James Milson's first land grant, of 50 acres, was granted to him on 16 January 1816 when it was recorded that he was living in the District of Parramatta (which included the "Field of Mars") and had a wife and 2 children. These 2 children were Sophia (1811) and James Jnr (1814). They were to be followed by David (1817), Elizabeth (1819), John (1821) and Robert (1824).

This first grant was for 50 acres at Broken Bay. Broken Bay is the entrance of the Hawkesbury River into the sea. It appears that Milson never did farm this land and was able to instead convince the Governor Macquarie to change this to a larger land grant of 100 acres at Pennant Hills in the "Field of Mars" (see below). Milson was successfully farming before 24 August 1819 when he was paid £7 14s remuneration from Police Fund for damages done him by Government cattle.

On 5 June 1820 an unsigned Memorial was written for James Milson to Governor Maquarie requesting further land. (Milson did not write this memorial himself.) This memorial states that James Milson came to the Colony with the family of Mr Robert Campbell, and Robert Campbell also wrote and signed a recommendation for James Milson to receive "favourable consideration" for his request on the bottom of the Memorial. The Memorial states that James Milson was residing on and farming a 100-acre grant that he had previously received from Governor Macquarie ("your Excellency was pleased to extend") at Pennant Hills. Of this 100 acres he had 20 acres under cultivation and an additional 10 acres ready for cultivation. He also had 40 head of Horned Cattle "and very little pasturage lands to graze them upon" and this was the basis of his request for a further land grant. As a result of this Memorial James Milson was granted a second 100 acres at Pennant Hills.

Property at North Sydney: 1821 to 1831 and beyond

Milson's 50 acres adjoining 120 acres Milson leased from Campbell

Sketch map first published in 1955 in the book "The Life and Times of James Milson" showing approximate location in the vicinity of the modern Jeffrey Street of "Milson's residence" in the 1820s which included his orchard and dairy. Also shows approximate location of the residence of Milson's son-in-law William Shairp who married Milson's daughter Sophia in 1827. The perforated line marks the approximate extent of the 120 acres (48ha) leased by Milson from Robert Campbell in the 1820s. Above the perforated line was the adjoining 50 acres (20ha) that was granted to Milson in August 1824. Milson's residence was burnt down in the bushfires of November 1826. In 1831 Milson built a new residence overlooked Hulk Bay (now known as Lavender Bay) located above the perforated line on his 50-acre (20 ha) grant.

In 1821 (the year ended 30 June 1822) Milson began to lease 120 acres (48ha) of land from Robert Campbell being the headland that includes Kirribilli and Milsons Points. There is a note held by the North Sydney Historical Society that states that Milson at first thought this land to be "nothing but rocks and stones, not enough soil to grow grass to feed stock." This does not agree with Milson's desire to lease the land from Campbell, or his attempting to establish a dairy and orchard there, something that he was able to do successfully. The headland was indeed partly rocky but also contained areas of useful farm land. The rockiness was also useful in providing building materials, and from a quarry on Careening Bay a business of supplying sandstone ballast to shipping. Warne's variation of this story is that Milson found the land to be "rocky and uncompromising" but he was assured that the soil was good for farming and grazing, and he was provided two convicts to assist with the clearing". This does not agree with Milson's experience as a man who had already been farming in the Colony for 15 years, or, in his own words, that Milson had been assigned convict servants from about 1810.

Milson initially leased this 120 acres of land for 7 years which extended to 30 June 1828 for a rent of £10 per annum "as a matter of favour" from Campbell. A lease agreement was signed by both parties with each taking a copy. The lease was then extended in a memorandum between the parties for a further period of 3 years to 30 June 1831 on the same terms. Non-payment of the rent on this lease between 1828 and 1831 became the basis of a court action in August 1831. By the time that Milson had stopped making rent payments on the lease, however, both parties had accidentally lost their copies of the original lease agreement, and Campbell had also accidenatlly lost his title deeds to the land. Once the court case was concluded with Milson being ordered to pay the rental arrears as damages, Milson continued to lease land from Robert Campbell until Campbell's death in 1846, and then from subsequent owners. A map from the 1840s show that Milson was still then leasing a smaller part of the 120 acres.

It is on part of the 120 acres (48ha) that Milson began to lease from Robert Campbell in 1821 that Milson built his dairy and his orchard and also grew vegetables in the area of the modern day Jeffrey Street. It is also on the small portion of the leased 120 acres where Milson established his dairy, orchard and vegetable gardens that Milson built his first home in the area, and another dwelling called "The Milk House" which became "a well known resort of Sunday-goers" from Sydney. It was from the 120 acres of leased land that Milson built a good business supplying ships in Sydney Harbour with fresh fruit and vegetables, milk and water, as well as ballast from a quarry near Careening Cove.

There is no way of knowing exactly when in the early 1820s Milson and his family started to reside on the leased 120 acres. In the 1822 muster of New South Wales, however, he is shown as a resident of the District of Sydney rather than the District of Parramatta showing that the move probably had been made before then. Then in at least the period 22 August 1823 to 1 November 1825 Milson and his wife lived at least part-time at Government House as they were employed there, with James acting as a Steward and Keeper of Government House.

In August 1824 Milson received from Governor Sir Thomas Brisbane a land grant of 50 acres (20ha) further up the headland and adjoining the 120 acres that Milson leased from Robert Campbell. (Elizabeth Warne incorrectly states that this 50 acres was received not as a land grant but as a promissary note.) This land grant gave Milson additional land on which to graze his dairy herd. This land grant also allowed Milson to be shown as a landholder in the District of Sydney in the 1825 muster of New South Wales rather than a resident as in the earlier 1822 muster. As Milson owned this 50 acres rather than leasing it, it was on this 50 acres that he built his family homes of "Brisbane House" (1831) and "Grantham". This land grant also made Milson not just one of the areas most prominent businessmen but also one of the area’s most prominent landholders.

Devastating bushfires occurred in November 1826, with "a house in the occupation of Milsom, a milkman on the North Shore" being burnt down on Saturday 25 November 1826. Elizabeth Warne also correctly records that the bushfires destroyed what was the other dwelling on the property, the "Milk House", but incorrectly calls it a dairy building. The Australian Dictionary of Biography reports that Milson lost in the fire the title deeds of his land at Pennant Hills (close to Castle Hill) and his 50-acre (20 ha) grant at "Hunter's Hill" (then the name for the area of North Sydney that includes today's suburb of Kirribilli and not to be confused with the today's modern day suburb of Hunters Hill) which adjoined the 120 acres (48ha) that Milson leased from Robert Campbell where Milson's dairy and orchard stood. If he lost the title deeds to these lands he also lost the title deeds to his other landholdings including those at Wollombi. (Loss of these title deeds did not force Milson to buy "some 890 acres" of land including land at Wollombi as claimed by Elizabeth Warne. Milson already owned this land prior to the fire, having, for example, purchased Wollombi in 1825. Loss of his title deeds had no long term consequence on Milson and those lands owned prior to the bushfire were also included in his will of July 1829. The subsequent 1831 court case between Campbell and Milson over Milson's non-payment of rent on the lease of 120 acres of land at "Hunter's Hill" discloses that Milson's copy of the lease document was accidentally lost, probably in the same bushfire.

In 1827 Milson had built for his family a replacement cottage on the leased 120 acres that he called "Brisbane Cottage" on or close to the site of his home that had been destroyed by the bushfires. According to notes held by the North Sydney Historical Society this cottage was built by Martin Regan, one of Milson's assigned convict servants who lived until he was 105. This replacement cottage, by then being used as a worker's cottage, appears in a map of the 1840s.

Afternoon tea on the upper lawns of Carabella in the 1890s (Mitchell Library SLNSW)

In 1827 Milson also built his daughter Sophia as a wedding present a cottage that he called "Carabella Cottage" on his 50 acres, and two years later her husband William Sharp built nearby a much larger home that he also called "Carabella" for £600 and that was situated at modern 34 McDougall St, Kirribilli. Milson was also to build his own homes "Brisbane House" (1831) and then "Grantham" on this land. The Dictionary of Sydney describes "Brisbane House" as "a stone house with two storeys facing Lavender Bay, then commonly called Hulk Bay, after the convict hulks moored nearby" and "Grantham" as "a large bungalow-style sandstone home on a larger plot of land (now occupied by the Greenway Housing Commission flats)." In the November 1828 census of New South Wales Milson is recorded as owning 1600 acres of land that included this 50 acres on which he was to build his home.

Before September 1828 it was known to both Milton and Campbell that neither of them had retained a copy of the original lease agreement for Campbell's 120 acres at "Hunter's Hill", and that both of them had lost the title documents for their respective lands at "Hunter's Hill". Only copies of the memorandum still existed.

Milson then attempted to obtain ownership of the land that he knew was owned by Robert Campbell by surreptitious means. In September 1828 Milson wrote to the Surveyor-General and expressed surprise that the land on which his house stood had been "measured to the Honourable Robert Campbell whose claim has never been properly established." Milson also wrote to the Governor on 3 October 1828 in a letter which stated "I have also to inform your Excellency that during the absence of the Surveyor-General, who is in Argyllshire, Messrs Cordeaux and Finch tool it upon themselves to erase my name which stood in the chart for the land which I am now in possession of, and are endeavouring to trump up a claim in favour of the Honourable Robert Campbell for the said land." Milson received a reply on 13 March 1829 from the Colonial Secretary, Alexander Macleay, which stated "I am directed by His Excellency the Governor (Ralph Darling) to inform you that if you allude to the spot on which you have built you must be aware that you cannot obtain it as it is the property of Mr Robert Campbell, who states that he actually gave a lease of it to you, and that the land to which you are entitled" (the adjoining 50 acres) "is already measured and marked out."

Milton also stopped paying his rent after having paid £4 2s 8d of the £10 rent for the year ended 30 June 1829. By the end of June 1831 Milton owed Campbell £25 17s 4d in rent (£5 17s 4d for the remainder of 1828/29 + £10 for 1829/30 + £10 for 1830/31). In June 1831 Campbell had served unpon Milton a bill of arrears of rent, and had received a promise of payment from Milton that had not been fulfilled. On 12 August 1831 Campbell then began court action against Milton for the arrears of rent. As the original lease agreement no longer existed Campbell sued Milton under an action of assumpsit (or breach of promise to pay) for use and occupation of Campbell's land.

During the court action of 1831 Milton never disputed Campbell's title to the 120 acres, or that a lease agreement had been in place. Indeed a copy of the memorandum that extended the original lease, the "second document", still existed. Milton also did not disagree with the evidence of Campbell's son "that a formal instrument had been executed by the parties, binding them, under their hands and seals, to certain performances of a contract for a specified term of years". Milton instead disputed knowledge of the boundary between Campbell's 120 acres and Milton's adjoining 50 acres (referred to as "very large portion of land") which he contended could not be proved without the title documents ("proof of title") under which Campbell held his land. Lack of knowledge of the boundary Milton contended caused him difficulties and for this reason he had resisted paying rent. Milton contended that not knowing where the boundary lay between the two plots of land meant that Milton did not know where the opposite boundary of his own 50 acres lay, left him open to a charge of trespass if he impounded cattle that wandered onto his land if the cattle were in fact on the land of his other neighbour, and left Milton without knowledge of how far to advance with improvements on his own property either in the direction towards Campbell's land, or on his opposite boundary. (That Milton was also missing the title deeds to his 50 acres was not drawn into his argument.) Milton also contended that he did not have the use of the 120 acres "sacred to the lease" as a man by the name of Cunningham was allowed to come onto the land and take away quantities of soil. Milton also contented that the action of assumpsit (or breach of promise to pay) for Milton's use and occupation of Campbell's land was not a lie, but that the proper cource for Campbell to have pursued should have instead been an action for debt under the second document, the memorandum.

The decision of the court was that even without registered title Campbell was the legal owner of the 120 acres, and that Milton was the permissive occupant. The action for assumpsit (or breach of promise to pay) for Milton's use and occupation of Campbell's land was therefore proved. Milton was ordered to pay to Campbell damages of £25 17s 4d, the arrears in the rent.

Bushfire of Saturday, 25 November 1826

The Australian Wednesday 29 November 1826

…it is calculated that the loss of property by the fire is greater than ever known before in the Colony by a similar event. A house in the occupation of Milsom, a milkman on the North Shore, was burnt down on Saturday.

A second newspaper report in the Sydney Gazette of the same date provides more information. At the time of the bushfire Milson was away from the property but his wife and children were at home. Mrs Milson sustained burns from the fire.

Sydney Gazette, Wednesday 29 November 1826

During the extensive and frightful conflagration that occurred in the woods on Saturday last, the house occupied by Mr. Milsom, an industrious settler, in Neutral Bay, at the North Shore, was consumed. Mr. M. was from home at the time, and his wife and five children were on the spot, destitute of the assistance of a man servant. The whole of the property was destroyed, and the inmates, especially Mrs. Milsom, with difficulty escaped with their lives, though Mrs. M. was much burnt. Never before was any thing equal to the devastation which the fire has effected; the grass, is destroyed for miles upon miles, and the trees are continually falling, to the danger of the traveller beneath, from the effects of being burnt at the roots.

A third newspaper report gives even more information about the events of the bushfire of 25 November 1826 that destroyed Milson's buildings and livelihood. This was co-incidentally the 12th birthday of Milson's son James Milson Jnr. The report ends with the chilling words "chilling Poverty appears to be their only prospect." Fortunately that was not to be the case.

The Monitor 8 December 1826

DOMECTIC INTELLENCE

It is our painful task to record melancholy instance of the mutability of human affairs. There dwelt upon the North Shore a reputable family named Melsham (sic), the heads of whom had long held confidential posts in the households of Governors Macquarie and Brisbane; having acquired an easy competency, they had retired to this sequestered spot to enjoy the fruits of their labour, a comfortable neatly furnished dwelling; a garden, cattle and a well known resort of Sunday-goers, known as "The Milk House," which produced a considerable weekly rental, constituted their possessions. Melsham (sic) was absent, and his wife and daughter saw with fearful solicitude the distant smoke rising o'er the hill; towards the set of sun the blazing heath presented to their view an appalling prospect – two male domestics then on the premises made active endeavours to cut off the communication of the flames, by clearing a space as ample as time would admit of, to check its destructive progress; but the attempt was vainly made. The impelling breeze wafted ignited masses through the air, and the only resource now left, was to remove the most valuable and necessary portables the house contained. Apparel, beds, &c. were hastily conveyed to a distance, but the unfortunate owners were compelled to witness the demolition of their peaceful habitation – the small remaining hope of saving the residue of their property was not left them long. The destructive element made rapid progress towards the depository of their moveables and quickly was the wreck complete – of a watch, the wheels alone remained – an old Kangaroo dog, guarding, it is supposed, the last relics of his master's worldly goods, had the hair literally singed off him, and his legs left in a pitiable state. The fate of their Cattle is not yet known, whether destroyed in the conflagration or driven into the bush. By this unforeseen visitation of Providence, these unfortunate people are once more thrown upon the world, their hopes destroyed, and in the decline of life chilling Poverty appears to be their only prospect.

Note: It is known from other sources that both James Milson and his wife Elizabeth did act as stewards at Government House, Sydney, for Governor Brisbane prior to 1826. They did not act as stewards in the time of Governor Macquarie and that part of the above newspaper report is incorrect.

Campbell's 120 acres from 1794 to 1820

The area of Kirribilli ("Hunter's Hill") was settled by Europeans early in the history of the Colony of New South Wales. The first record of land being granted in this area of the North Shore was on 20 February 1794. This was for 30 acres (12 ha) on the North side of the Harbour of Port Jackson, opposite Sydney Cove for a quit rent of 1s per annum after 10 years to an expired convict, Samuel Lightfoot. The North Sydney Historical Society states that the 30-acre grant extended from where Milsons Point Railway Station is now to Kurraba Point but this is impossible as it is an area of much greater than 30 acres and extends over to Neutral Bay which was never part of the land grant. Revising their description, the original land grant extended from about the position of today's Milsons Point Railway Station across to Lavendar bay, and down to and including Milsons Point, that is the western side of the later 120-acre land grant. Lightfoot was a former convict, born in about 1763 and transported to Australia for seven years for stealing clothing. He arrived with the First Fleet in 1788 on the Charlotte.

In 1794 Thomas Muir, a Scottish constitutional reformer, was sentenced to transportation for sedition. He arrived in the Colony on the Surprise on 25 October 1794. Lightfoot sold his farm to Muir who also had a cottage on what is now Circular Quay. The farm that Muir began to establish on the 30 acres he named, after his father’s home in Scotland, "Huntershill" (not to be confused with the modern day suburb of "Hunters Hill" further up the harbour). Muir is thought to have built a hut marked on an old map, and therefore is thought to have been the first European resident on the North Shore. Thomas Muir escaped from the colony in February 1796 aboard an American brig, the Otter and died in France two years later.

The next record of 26 April 1800 is that the 30 acres that had been purchased by Robert Ryan. (Some references incorrectly state that Ryan was granted the 30 acres.) Ryan had worked in Norfolk Island, both as a soldier and also a settler. It is not known how the ownership of the land passed from Muir to Ryan, or whether there were other unknown intermediate owners.

On 26 April 1800 the Colonial Secretary recorded that the 30-acre land grant to Lightfoot was cancelled. The 30 acres had been purchased by Robert Ryan and was being cancelled to incorporate it with an additional grant of 90 acres (36ha) into a total grant of 120 acres (48ha) On the North side of the Harbour, opposite Sydney Cove for a quit rent of 2s per annum after 5 years. The 90 acres (he already owned the 30 acres) was being given to Ryan for his services in the Royal Marines and the NSW Corps. The corresponding entry in the Register of Land Grants states Cancelled, and a New Grant given to Robert Ryan for 90 acres in addition to this Allotment, by Governor Hunter. See the third Register, Folio 37. The entry against the list of all grants and leases of land registered in the Colonial Secretary's Office for Robert Ryan's 120 acres says 90 acres of this grant is given towards the complement Ryan is entitled to of 160 for his services to the Marines & NSW Corps; the other 30 is a Grant of Samuel Lightfoot's which was purchased by the present proprietor & is now cancelled in the first Register folio 78. Robert Ryan therefore has yet 70 acres of land to receive.(160-90=70)

This 120-acre grant to Ryan included almost all of the Kirribilli headland including both Kirribilli Point and Milsons Point.

Around 1806 Ryan, who knew nothing about farming, sold his the property to Robert Campbell, a wealthy Sydney merchant. Campbell built Australia’s first shipbuilding yards there in 1807, at the site that is now occupied by the Royal Sydney Yacht Squadron, Kirribilli.

1814 Quarantine Station

The uninhabited 120 acres at Kirribilli was also used briefly for quarantine purposes in 1814 for the people who had arrived on the convict ship Surry.

In 1814 the Surry was carrying a "cargo" of 200 male convicts. Passengers on board were 25 men of the 46th Regiment. During the voyage out from England malignant fever (typhoid fever) raged on board and a total of 42 men died before the ship anchored in Port Jackson on 27 July 1814. Of these 42 men: 36 were convicts (listed with their names and dates of death in the Sydney Gazette of 10 September 1814), 2 were of the ship's crew, 2 were soldiers from the 46th Regiment, 1 was the 1st (Chief) Mate, and shortly before the ship anchored the Master (Captain) of the ship also died. (36+2+2+1+1=42).

Once the Governor became aware of the situation on board Surry he ordered that the ship be anchored near the uninhabited North Shore and be placed into quarantine. This quarantine was also placed on both the ship and the part of the North Shore off which the ship anchored to allow those on the ship onto land.

On Wednesday (27 July 1814) arrived the ship Broxbornebury…

Same day arrived the ship Surry, Captain PATERSON, who has since unfortunately fell a victim to a fever that had for some time raged with excessive fury on board that ship, and which has also proved fatal to the chief officer, two soldiers, belonging to a detachment of 25 men, of the 46th Regiment, two of the ship's company also, and 38 (sic) male prisoners, of whom she had originally 200 on board.

By the Surry has arrived Brevet Major Stewart, of the 46th Regiment.

The Broxbornebury fell in with the Surry off Shoal Haven; & on speaking her, learnt that from the relaxed state of the crew, and illness of the surviving officers, her safe conduct into this Port was despaired of, unless assisted from the other ship with some person capable of navigating her in; for which service a seaman capable of the task generously volunteered his service, and brought her in. - As soon as the melancholy report was communicated to the GOVERNOR in CHIEF by the Naval Officer, who had in the mean time prohibited any communication with the vessel from any shore or ships' boats, HIS EXCELLENCY was pleased to confirm the interdiction until the nature of the prevailing malady should be ascertained, in the best manner that the circumstances of the case would admit, by the Principal Surgeon and the other Gentleman of the Medical Department; whose report, after being repeatedly alongside, confirmed the apprehension already entertained of the courageous nature of the distemper; and His Excellency was in consequence pleased to issue his instructions that she should be brought up at a convenient anchorage near the North Shore, where the people might be landed, and remain until the danger of their nearer approach should disappear; the necessary arrangements for which salutary precaution are stated in the General Orders of this day. The benefit of a free circulation of air onshore, and the exercise which the still suffering patients may become progressively capable of enjoying in this salubrious climate, to which the present season is perhaps far from being unfavourable, will, we hope most fervently, very shortly put an end to the pestilential terrors and render the restraint no longer necessary.

— "Ship News", Sydney Gazette, Saturday 30 July 1814

The Government Orders about the Quarantine were clear that no one was to go near the persons who had arrived on Surry. A guard was to be placed on board the ship, another guard on the land, and severe punishment was promised for anyone who broke the quarantine.

Sydney Gazette, Saturday 30 July 1814

GOVERNMENT AND GENERAL ORDERS.

Head Quarters, Government House, Sydney, Saturday 30th July, 1814

The Male Convict Transport Ship, Surry. Which arrived in this Harbour from England on the 28th (sic) Instant, having brought a malignant Fever of a very infectious Nature, of which the Master (Captain), first Mate, and forty other Men (=42) have died during the Voyage, previous to the Arrival of the Ship in this Port; HIS EXCELLENCY the GOVERNOR had deemed expedient to use every possible Precaution to prevent the Danger of the Contagion extending from her to the Population of the Colony; and with this View he has given Orders to institute the most rigid Quarantine Restrictions in respect of the Ship herself, and all Persons on board of her, until such Time as the Fever is entirely subsided, and the People now infected are recovered.

It is therefore the GOVERNOR’S most positive Orders, that no Person whatsoever (the Medical Quarantine Officers and their Attendants excepted), shall have any Intercourse or Communication of any sort with the said Ship or the Persons now on board her, until such Time as she is relieved from Quarantine, which will be announced in Public Orders; and the GOVERNOR further directs and commands, that no Person shall visit, or have any Intercourse what ever with the Military Detachment, Sailors, ad Convicts, who are about to be landed from the said Ship on the North Shore of Port Jackson; where they are to remain encamped under strict Quarantine Regulations till further Orders. – There will be a Guard on board the Ship Surry, and also on the North Shore; to enforce these Orders; and any Person detected in attempting to act in Disobedience of them will be confined, and most severely punished.

By Command of His Excellency The Governor,

J.T.Campbell, Secretary.

Quarantine restrictions were lifted for the ship on 13 August 1814, but the restrictions on the camp lasted longer.

GOVERNMENT NOTICE

SECTRETARY'S OFFICE, SYDNEY

Saturday, 13th August, 1814

It being duly notified to HIS EXCELLENCY the GOVERNOR, that the Disease on board the Male Convict Ship, Surry, which had rendered it necessary to put the Vessel under strict Quarantine on her Arrival in this Port, has now happily terminated, and the Danger of Infection from the said Ship is entirely removed:--This is therefore to give Notice that his EXCELLENCY is pleased to relieve said Ship from any further Continuence of those Quarantine Restrictions: but as the Disease is not altogether ceased in the encampment on the North Shore, the Quarantine Restrictions in Regard to that Place, and the Persons there, shall continue in full Force.

By Command of His Excellency,

The Governor,

J.T.Campbell, Secretary

The uninhabited land on the North Shore off which Surry anchored, and on which the persons who had arrived on the ship were allowed to land and set up camp during the time of their Quarantine, was the 120-acre headland owned by Campbell that included Milsons Point and Kirribilli Point. Thus this headland was the first site in Australia to be used for quarantine purposes. This is known because three typhoid victims from Surry were buried there with their graves being in the area that later became Milson's orchard. These graves are stated to be the site of the first burials on the North Shore. They are shown on MAP 3 in this article, a part of the 1840s map of the area. The notation on the map says "Spot where three of the Surry's crew are interred". One of those buried is known to have been the ship's surgeon who attended the typhoid victims. In 1925-26 when homes were being demolished for the construction of the Sydney Harbour Bridge, the gravestones were found being used as hearthstones in cottages in what is now Bradfield Park (located approximately 50 metres west of the location of the graves shown on the early map), and included the gravestone of the surgeon.

The 42 deaths before the ship anchored, plus the 3 graves brings the death toll to 45 (assuming that the 3 graves were all for deaths that occurred after the ship anchored). Bateson, however, states that there were a total of 51 deaths before the quarantine was lifted and lists the additional 9 deaths as the 2nd Mate, the boatswain, an additional 4 seaman, an additional 2 soldiers, and the surgeon.

Thomas Raine was a junior officer on board the Surry. The epidemic of typhus left him the only surviving officer. He subsequently commanded her for the next three voyages (1816, 1819, 1823).

Raine's grandson Tom Raine founded Raine and Horne, an Australian real estate franchise in 1883. There is a memorial to Raine slightly to the west of the Jeffrey Street Wharf.

The family tradition is that Milson lived on this land from the time of his marriage in 1810, and that Milson gave permission for the victims from Surry to be buried there in 1814. That this piece of land was used for a strictly enforced place of quarantine is proof, however, that the 120 acres was uninhabited in 1814, that James Milson did not live there at that time, and that the graves were already in place before Milson moved there and started to build his orchard.

Subsequent owners of Campbell's 120 acres after his death in 1846

After the death of Robert Campbell in 1846 his lands first passed into the hands of his Estate, and then changed ownership under the terms of his will. It is by Campbell's will dated 11 October 18?? (unreadable) that Campbell's 120-acre property in Kirribilli was broken up into areas that can be readily compared with modern maps.

Campbell left a complex will. Eventually each of the sons received one-sixth of the estate while both daughters' shares passed on to the Jeffreys family by about 1880. The west side of Jeffreys Street was left to George Campbell, the youngest son. The east side of Jeffreys Street was left to the elder sons, John and Robert Campbell. Arthur Jeffreys was left the block of land immediately to the north of Jeffreys Street, on the northern side of Fitzroy Street.

Among Campbell's landholdings bequeathed to his children was a small bequest to his daughter Sarah Campbell, the wife of Arthur Frederick Jeffreys. By a deed of partition dated 15 April 1848 and conveyance of 28 February 1854, Jeffreys received a small portion of the 120 acres being the area approximately bounded by modern Willoughby, Carabella, Fitzroy and Broughton Streets (including Burton Street), an area of land that is immediately north of where Milson's dairy and orchard once stood.

Arthur Jeffreys subsequently disposed of various portions of his holding. In 1873, which was also after the death of James Milson, Jefferys sold 2 roods and 6 perches (.2 ha) to joint purchasers Neil Stewart of Parramatta and Henry Hocke Bligh of North Shore. This is the area of the modern Kirribilli Neighbourhood Centre at 16 Fitzroy Street.

At this time Sydney was still a very small colony and the two new owners of this 2 rods and 6 perches of land were connected to Milson by marriage. Stewart's sister had been the second wife of Milson's eldest son, James Milson junior, and Bligh had married Elizabeth Milson Shair, Milson's grand-daughter.

MAP 1: Earliest detailed map of Kirribilli Point and Milsons Point: an 1840s subdivision map for the Estate of Robert Campbell (1769–1846).

Early maps of Kirribilli

MAP 2: Modern map of Kirribilli Point superimposed on top of 1840s map of the same area (superimposed on top of Map 2). Distortion in original 1840s map can be clearly seen. Marked on the map in green is the location of today' s Jeffrey Street; in yellow are the streets in the area of James Milson's Orchard on the 1840s map; in red (upper) is the location on the 1840s map of the large building shown near the milking bails; in red (lower) is the location on the 1840s map of the worker's cottage built within James Milson's orchard which can be seen to be located on modern day Jeffrey Street; in pink is the location on the 1840s map of the boundary of James Milson's orchard; and in pale blue is the approximate location of heritage listed Bratton.

Lavender Bay in the 1880s as photographed by Bernard Otto Hotermann from the western side of the bay towards Milsons Point. (Mitchell Library SLNSW)

MAP 3: Cropped detailed map of Jeffrey Street area of Kirribilli. Cropped from the 1840s subdivision map for the Estate of Robert Campbell (1769–1846).

MAP 4: Map is from the 1850s but refers to earlier land grants. Caption reads "Early map published in the North Shore Times 8 May 1963 showing the boundary of the Municipality of St Leonards East, proclaimed 1860." Added to the original map are the words "Lavender Bay" which used to be called Hulk Bay, and "Milsons Point". Colours have also been added. Highlighted in yellow is the original 120-acre grant to Robert Ryan in 1800 that was purchased around 1806 by Robert Campbell; in pale blue is the 50-acre grant to James Milson in August 1824; in green is the approximate position that was marked on the map for the orchard that James Milson was still leasing from Robert Campbell in the 1840s (in the 1820s he leased the whole 120 acres); and in pink is the approximate position that was marked on the map for the farm of F.J. Hilly that was leased from Robert Campbell in the 1840s. The boundary of the Municipality of East St Leonards is marked in red.

The North Sydney Historical Society mentions an old map on which a hut is marked but provides no other details.

A road survey map of 1822 identifies a "Point Kiriabilli" with no indications that the area was settled.

Map 1 & Map 2

The earliest detailed map of the area dates from the 1840s and is held by the National Library of Australia. It was a plan for sub-division drawn up for the Estate of Robert Campbell (1769–1846), the owner of the land. The map shows settlement in the area of Jeffrey Street and Milsons Point as well as what is now known as Kirribilli House and Admiralty House at Kirribilli Point. The other parts of the map have streets and property boundaries shown for the subdivision but are otherwise largely blank.

The map shows the extent of the 120 acres (48ha), owned by the Estate of Robert Campbell, which James Milson was leasing from Robert Campbell in the 1820s, and the subsequently smaller area that Milson was leasing in the 1840s where just his orchard, dairy and a worker's cottage were situated. The worker's cottage had been his second home in the area, "Brisbane Cottage". It is believed to near the site of this cottage that Milson had built his first home in the area that was destroyed in the devastating bushfires of November 1826.

Adjoining the 120 acres (48ha) owned by the Estate of Robert Campbell the map also shows the location further up the point of the 50-acre (20 ha) grant of land made by Governor Brisbane to Milson in August 1824. When compared with modern maps of the area, the 1840s map shows that the boundary between Campbell's 120 acres and Milson's 50 acres went across the peninsula in a straight line, and that modern day Willoughby Street forms part of that straight line.

It was on this adjoining 50 acres (20ha) owned by Milson that Milson grazed his dairy cattle. It was also on this adjoining 50 acres (20ha) owned by Milson that Milson had resided since 1831 when he built "Brisbane House" as the family home. The house was named after Sir Thomas Brisbane, Governor of NSW from 1821 to 1825. It was a two storey stone house situated approximately at the block enclosed by today's Arthur, Middlemiss and Lavender Streets, Lavender Bay. The home faced Lavender Bay, then commonly called Hulk Bay, after the convict hulks moored nearby. In addition Milson later built on this 50 acres (20ha) a large bungalow-style sandstone home situated at Greenway Drive, Milsons Point near where the Greenway Housing Commission Flats are today. This home was called "Grantham" after James Milson's place of birth. Both these homes were demolished in 1925-26 to make way for the building of the Sydney Harbour Bridge.

Map 3

The cropped detail map shows that Milson had an orchard, dairy, beehives and a (worker's) cottage on leased land in the vicinity of Jeffrey Street. In the orchard were the 1814 graves of three men from the "Surry", vegetable beds and a gardener's hut. The dairy was in a barn and near the dairy was a well, milking bails, and a calf pen. The cottage had a separate kitchen and near the cottage were sheds, a stable, a store, the bee-hives, and a men's slab hut. The map also shows the tank on the foreshore below the orchard that Milson built in 1832 (according to the Australian Dictionary of Biography) to supply water to shipping. According to one contemporary observer of the times:

Mr Milson, North Shore, has cut a cistern in the solid rock capable of containing 100 tons of water [which is] supplied from an excellent spring constructed so as to fill, by means of a leather hose conductor, ship's boats along the beach where the water is of sufficient depth to float a ship of the line.

Other features of the cropped detail map in the vicinity of Jeffrey Street are as follows:

• The shape of the shoreline has not changed significantly since 1840, the variation is due to the construction of ferry landings in the vicinity of the Jeffrey Street Wharf over the past 100 years and also the construction of the Sydney Harbour Bridge in the 1930s.
• The Sydney Harbour Bridge is now located at the base (south) end of the point.
• The large building shown at the top of the map near the "Milking Bails" and a "Calf Pen" is marked as a "Barn". It is close to Bratton, a Victorian Italianate style heritage listed building at 36 Pitt Street Kirribilli.
• The small stream marked at the south east corner of the "Orchard occupied by J. Milson" is most probably the site of a small timber decked simple steel bridge in what is now called "Captain Henry Waterhouse Reserve".
• The map identifies the 1814 graves for three typhoid victims from the ship Surry. These graves were within the area that Milson leased from Campbell for his orchard.
• Many of the streets and some property boundaries can be readily compared with modern maps.

Map 4

A map from the 1850s for the then proposed Municipality of East St Leonards (which was proclaimed in 1860) shows boundaries of original grants. It shows the extent of the 50 acres that was granted to James Milson in August 1824 which has been highlighted in pale blue. When compared to modern maps it shows that the 50-acre grant extended from Berry Street in the North, Walker Street in the West, abutted Careening Cove and other grants in the East for which no modern street marks the boundary, and adjoined Campbell's 120 acres in the south at Willoughby Street.

Campbell v Milsom: 1831

Sydney Monitor, 5/10/1831:

SUPREME COURT - CIVIL SIDE

SEPTEMBER 30 - Before Mr. Justice DOWLING, and Messrs. McLaren and Ross, J, P.'s, Assessors.

Campbell v Milsom - This was an action of assumpsit (breach of a promise to pay), to recover the sum of £25 17 4d, for rent due from the defendant for the use and occupation of a certain farm situate on the North Shore, leased by the plaintiff to defendant for a term of years. It appeared in evidence that the defendant leased a farm of 120 acres at the North Shore, from the plaintiff his former employer, as a matter of favour from the plaintiff, at an under rent of £10 per annum, for a term of seven years originally, and which was subsequently extended three years for the purpose of growing vegetables, &c. For the fulfillment of the contract on both sides, an Instrument was signed by both parties, each taking a copy, but which instruments was by accident lost by both parties. That the defendant had punctually paid his rent up to the year ending June 1828, and also the sum of £4 2s 8d towards the rent of the year 1829, at which time he stopped payment, without assigning any reason for so doing; in consequence of which a servant of the plaintiff's waited on the defendant in June last, presented him with a bill for the arrears of rent, and received a promise of payment from him, but which promise he had never fulfilled, which was the occasion of the present action.

For the defense, Mr. Wentworth, without calling witnesses, in his address to the Court, stated that the defendant would not have resisted the payment of the rent if he could have discovered for what he was paying such, having frequently applied to Mr Campbell for the boundaries of his farm, but without being able to obtain them. In consequence of which, had the defendant been disposed to impound any cattle, he could not have done so without making himself liable to an action at law for a trespass. Another important fact was, that the defendant carried on a trade in supplying the shipping in the harbour with ballast, by the consent of the plaintiff; and yet although the farm ought to have been held sacred to the lessee, and which in fact the lessor was bound to secure to his tenant, a person of the name of Cunningham was in the habit of taking away quantities of soil for ballast, under the sanction of an order from the Colonial Secretary. In the absence of any proof of the title under which the plaintiff held the land, the defendant who was granted a very large portion of land from Sir Thomas Brisbane, had taken up a part of that grant adjoining the farm in question, but did not know how far he might have advanced for want of the proper boundary, and which the plaintiff also appears not to know. Mr. Wentworth contended, that by the evidence of the plaintiff’s son, it appeared that a formal instrument had been executed by the parties, binding them, under their hands and seals, to certain performances of a contract for a specified term of years, and that the present action of assumpsit for use and occupation, could not lie, but that an action of debt would have been the proper cause to pursue.

The learned Judge over-ruled the objection taken by Mr. Wentworth, as he was advised, that the second document was not formally drawn, but was merely a memorandum between the parties, and put the case to the Assessors on the merits, who immediately returned a verdict for plaintiff. Damages - £25 17s 4d.

Dispute over 100 acres at Pennant Hills

In 1820 James Milson owned, and was living on, a land grant of 100 acres at Pennant Hills. This was not enough land for his needs and he began making attempts to increase the land available to him for farming purposes. First in 1820 he requested a second land grant from Governor Macquarie, and second in 1821 he made arrangements with Robert Campbell to lease 120 acres of undeveloped land at Kirribilli.

Milson's request for a second land grant from Governor Macquarie was successful, and after May 1821 Milson was granted a second 100 acres at Pennant Hills. James Milson applied to the Surveyor General John Oxley (1783–1828) and on 15 September 1821 received an order of possession for 100 acres that had been part of John Macarthur's Cornish Hills Farm and that John Macarthur had surrendered to the Crown in May 1821 as part of a deal to give Macarthur more land at Camden. This second 100 acres at Pennant Hills then became the subject of a dispute between James Milson and James Shepherd.

Sir Thomas Brisbane became Governor of New South Wales on 1 December 1821. In an undated memorial sometime after 1 December 1821 and before 8 November 1822, James Milson dictated a letter to Governor Sir Thomas Brisbane giving his side of the land dispute. This memorial is signed "James Milson", in the same hand that wrote the memorial, on James Milson's behalf. Milson then took possession of the 100 acres, began to clear the land by "felling some timber", and built "the frame of a small dwelling". Another man, James Shepherd of Kissing Point (an area between Ryde and the Parramatta River) also received an order for a grant of 100 acres from Governor Maquarie, and order which Milson maintained (incorrectly) did not state where the land was located. James Shephard then came upon the 100 acres for which James Milson had a possession order and began felling timber. James Shepherd was claiming a right of possession of this land. Milson next mentions that this 100 acres was part of the land given up by John Macarthur to the Government. He then claims that John Macarthur had "begged that the piece of land alluded to if located to anyone might be given to" James Milson. Milson then asked for Governor Brisbane to redress this grievance and to confirm Milson's order of possession from Mr Oxley.

The order of possession that Milson had from the hand of John Oxley said:

The farm originally granted to Cover of 100 acres situated at Pennant Hills, District of Field of Mars, having been surrendered to the Crown; James Milsom, being entitled to receive one hundred acres under an order of his Excellency the Governor, is hereby authorised to take possession of the same subject to the future confirmation of this authorisation by His Excellency the Governor. (signed) J Oxley, Surveyor General by (illegible) 15 Sep 1821

On 8 November 1822, James Shepherd dictated a memorial to Governor Sir Thomas Brisbane giving his side of the land dispute. This memorial is signed "James Shepherd", in the same hand that wrote the memorial, on James Milson's behalf, and in a different hand to earlier memorials of James Shepherd. James Shepherd stated that, after his name was inserted on a list of applicants to receive grants of 100 acres of land, that after learning from John Macarthur that he had relinquished a farm in the District of Dundas adjoining a farm that James Shepherd already owned, that Shepherd memorialised Governor Maquarie respecting this land. (The memorial to which he referred was dated 6 August 1821 and referred to the 100 acres that had originally been granted to James Cover.) to which he received the following answer:

The Governor has received Mr James Shepherd's Memorial of date 6th instant and in reply thereto had now to inform him that in compliance with his request the Surveyor General will be instructed to locate and mark out for him the one hundred acres of land lately relinquished by Mr McArthur to Government in the district of Dundas in lieu of the 100 acres for which Mr Shepherds name was some time since inserted in the list of applicants for land. Sydney. 30 Aug 1821

James Shepherd then mentions that Milson approached Oxley after this date requesting the same land, and Oxley not knowing that the land had already been granted to Shepherd promised it to Milson. As a result Milson took possession and felled about one acre of timber.

James Shepherd next states that he then approached Oxley showing Oxley the order that he had from the Governor, and that Oxley then directed Milson to choose land in some other place. (This is something that Milson did not mention in his memorial.) As a result of Mr Oxley's orders to Milson, James Shepherd then took possession of the land and be way of improvement began to fell timber.

From both Milson's and Shepherd's memorials we learn both men had come upon the 100 acres and begun to clear it by felling timber, and that they had done this that independently, and initially without the knowledge of the other. This was possible because neither man was living on this land. James Shepherd who also owned the 100 acres next door to this land (on which he ran horned cattle) was living at Kissing Point. James Milson was working on his other 100 acres at Pennant Hills, and working at Kirribilli where he was improving leased land to establish what would become his dairy and orchard, and living between the two.

James Shepherd continues his memorial by stating that he next applied to Governor Brisbane for a clearing party to help him clear the land. The answer that Shepherd received was "that James Milsom has represented to his Excellency that he had a prior claim." Shepherd then most humbly begs the Governor to make inquiry or investigation as such will prove that Shepherd has the "right of priority". Shepherd also asks that the Givernor enquire of Mr Meehan the Deputy Surveyor who was consulted by Governor Macquarie before he gave Shepherd the answer that granted Shepherd that 100 acres. Stating that Governor Brisbane has the highest sense of justice and impartiality, Shepherd then "trust"s that the "misunderstanding will be removed".

On the back of James Shepherd's memorial some notes were scribled to say that the Governor had decided that the decision to this dispute should be made by 2 arbitrators, one each to be chosen by the 2 claimants. As a result on 10 February 1823 both James Milson and James Shepherd were written to by Colonial Secretary Frederick Goulburn on behalf of Governor Brisbane with instructions for arbitration based on the scribled notes. The two men acted swiftly to appoint arbitrators as on 20 February 1823 Shepherd had a letter written to Mr Goulburn compalining about Milson's choice of Arbitrator. Milson had chosen to be his Arbitrator John Macarthur. Shepherd disagreed with this choice on 2 grounds. One that due to Macarthur's illness proceedings would be delayed. Two that John Macarthur would not be impartial as John Macarthur had tried to procur the farm for James Milson.

Unfortunately there is nothing contained in the papers of the Colonial Secretary that records the outcome of the arbitration.

Other landholdings

In the November 1828 census of New South Wales Milson is recorded as living at Hunters Hill, then the name for the area of Kirribilli. He is also recorded as holding 1600 acres (646ha) of land in the Colony of New South Wales of which 150 acres (60ha) had been cleared. His declared holding of livestock was 4 horses and 220 horned cattle.

It has been stated earlier in this article that James Milson owned 100 acres at Pennant Hills (Memorial of 1820) and 50 acres at North Sydney (near Kirribilli) that he received in August 1824. The Australian Dictionary of Biography states that in his will signed in July 1829 Milson lists a landholding of 220 acres (89ha) at Castle Hill (that is Pennant Hills, and includes the 100 acres of the memorial of 1820), the 50 acres (20 ha) granted by Governor Sir Thomas Brisbane at North Shore, 640 acres (259 ha) at Wallumbie (Wollombi), and 5 acres (2 ha) on Neutral Harbour (Bay) totalling not 1600 but 915 acres. This list shows, however, that Milson had disposed of some of his lands, and the disposal may have been between November 1828 and July 1829.

In 1824 there is an undated memorial to Governor Brisbane about a grant of land that he had been promised. This largely illegible document states that the place where he had been told to pick a plot of land (the Australian Dictionary of Biography refers to this as being 300 acres at Pennant Hills) no longer had any unallocated land. Milson instead requested land "some unallocated land at the head of Pitt Water back of Mr Wentworth’s farm". It is not known how many acres this grant was for, or if the request for land to be situated Pitt Water was granted. The Australian Dictionary of Biography states that Milson instead sought the 50 acres on the North Shore (that he was granted in August 1824) and 300 acres farther inland. The 50 acres on the North Shore adjoining the 120 acres than was being leased from Robert Campbell appears to be the grant given instead of land at Pittwater (Mona Vale). If Milson also received additional land further inland he had disposed of it before July 1829 as it was not listed in his will.

In a later undated memorial in 1824 written to Frederick Goulburn, the Colonial Secretary, Milson requests a ticket of occupancy for land bordering the Narrabeen Lakes where he wishes to place his horned cattle. This largely illegible document states that James Milson has 100 head of horned cattle. Bordering this land is "(east or west) side by the (illegible), (illegible) side (illegible) Lagoon called Narrow Byne and (north or south) (illegible words) ?amson’s farm." The ticket of occupation was granted on 28 December 1824 when Milson received a letter from Frederick Goulburn in regard to the memorial that Milson had submitted. It is not known how many acres this land was. Milson had disposed of this land before July 1829 as it was not listed in his will.

In 1825 James Milson was allowed to purchase 1000 acres of crown land. On 3 May 1825 his name appears on a list of persons who have received warrants to purchase crown lands which states that he had already paid for 1000 acres. This land was at Wollombi and Milson had disposed of some of this land before July 1829 as only 640 acres is listed in his will.

On 18 June 1825 Milson submitted another memorial requesting land at Lane Cove. The interesting thing about this memorial was that, even though it was prepared for him by another, James Milson signed it. It is signed as "J W Milson". In this memorial Milson stated that he had been in Colony for 19 years, that he came free (that is he did not arrive as a convict), and he has a wife & six children. He also states that he has built up a herd of 120 cattle. He has the means to improve and cultivate another farm if such is granted with which he will be better able to provide for his large family. He asked for a grant at Lane Cove near Mr Wolstoncroft’s and southwest of Higgins’ Farm. This request for more land was successful, and he received 250 acres. In regard to this 250 acres his name appears on a list of persons who have received orders for grants of land dated 25 August 1825 & on a list of lands granted and reserved by Sir Thomas Brisbane dated 17 September 1825. Milson had disposed of this land before July 1829 as it was not listed in his will.

The Australian Dictionary of Biography states that "In 1832 he built on the North Shore a reservoir for watering ships. In later years he acquired more land and was a keen yachtsman." It is recorded in the Sydney Monitor of 2 December 1835 that he bought another 50 acres of crown land at Wollombi.

A public reply to a charge of assault

In October 1825 James Milson was brought before the court in a charge of violent assault towards James Murphy, one of his assigned servants:

Sydney Gazette, 3 November 1825

POLICE REPORT.

MONDAY, OCT. 24.

This day, a prisoner of the crown named James Murphy, a government servant to a Mr. Milsom, preferred a complaint against his master for beating, bruising, and otherwise ill treating him; the marks of which violence were very visible on his person, and the Bench deeming this not only a violation of the Government Orders, as they extend to the protection of servants of the crown, but also an outrage on humanity, they ordered the master to stand committed to take his trial at the Quarter Sessions; bail, was however accepted for his appearance, and the complainant sent to the Barracks until his evidence shall be required.

According to a letter from James Murphy the case had also been reported in an earlier edition of the "Sydney Gazette" dated Thursday 18 October 1825. As a result of this publicly reported assault charge James Milson had a letter written to the Editor of the Sydney Gazette giving his side of the story. Bruising that the magistrates had called "an outrage on humanity" Milson underplayed as a strike on the face.

This letter reveals in James Milson's own words details of his farm and business dealings:

• Milson had horned cattle grazing near Castle Hill. This agrees with the memorial to Governor Macquarie of 1820 stating that then he had a 100-acre farm at Pennant Hills (in the Field of Mars);
• Milson had dairy (milch) cattle in the District of Sydney (which included the North Shore of Port Jackson);
• Milson had a boat from which he daily supplied milk to shipping on Port Jackson;
• Milson for the past 15 years (1810–1825) had employed from 2 to 4 assigned convict servants;
• Milson had in the period before 1825 principally resided in the District of Parramatta (which includes the Field of Mars);
• Milson was then living on the North Shore of Port Jackson opposite Sydney.

Sydney Gazette, 3 November 1825

TO THE EDITOR OF THE SYDNEY GAZETTE.

North Shore, October 30, 1825.

SIR,-Perceiving in your Gazette of Thursday, the 18th Instant, a certain Article, under the Head of Police Report, at Sydney, respecting Mr. JAMES MILSOM being brought forward on a Charge of his assigned Servant, James Murphy, for committing a violent Assault on his person; and as some officious person has falsely stated the Report to you, in justice to myself and character, and that the Public may not be lead astray, I beg leave to state the particulars of the case, and request you w ill insert the same in your Paper:---
James Murphy came to my service in July, 1823, and was assigned to take care of my stock of horned cattle. During such time he was employed, I paid the pound keeper of Castle-hill, for stock impounded through his neglect, the sum of £3 18s 8d I allowed him a ration weekly of 4lbs of salt pork and a peck of wheat, besides £10 per annum. I then employed him to go in a boat, to supply the shipping daily with milk, but found that he neglected that duty, got drunk, and made away with great part of the money; on his expressing contrition for the same, 1 forgave him. I then employed him to mind twelve milch cows, which duty he neglected, and let them stray. He lost four head, and I desired him to go after them; he went for a few hours, and then returned, stating he could not find them. I told him he must find them. He went again, and returned in about half an hour stating he could not go after them as he had no shoes. I had supplied him with a new pair of boots only six weeks before. I told him to go after them, he said he would not, and dared me to make him, and used the most provoking language towards me, when in the heat of passion, I struck him with my hand in the face, when he ran away and called out murder. I came in on Monday morning last to lodge a complaint to the Police Magistrate, and found Murphy there. Murphy has drawn in the time he has been in my service to the amount of £20 16s l0d in cash, and necessaries. He has lost two cows wholly, which he cannot account for. I have been in the habit of keeping from two to four Government servants for the last fifteen years, and can appeal to the Magistrates at Parramatta, where I principally resided, whether any complaint had ever been exhibited before them of any ill-treatment I have used towards them; and shall, on the day of trial before the Quarter Sessions, at Sydney, produce such respectable evidence in vindication of my character, ad will clear it from the aspersion that is attempted to be put on it in the report contained in your Gazette.

I have the honour to be, Sir,

your most obedient Servant,

JAMES MILSOM

James Milson Jnr (1814–1903) & the Royal Yacht Squadron, Kirribilli

James Milson Snr was a farmer and his business interests were associated with the land and its produce. His son James Milson Jnr served a mercantile apprenticeship and became a merchant involved in importing and exporting. He gained a wide knowledge of shipowning.

In 1845 James Milson Jnr invested in the company that owned the paddle steamer the "Fairy Queen". In the 1860s James Milson Jnr also became a director of the Australian Steam Navigation Co. In September 1863 James Milson Jnr was one of the founders of the Milson's Point Ferry Co which inaugurated a regular service between Milsons Point and Circular Quay. Milson's Point Ferry Co was sold in March 1878 to become the North Sydney Ferry Co, and which later merged to become Sydney Ferries Limited.

With the same name, some sources confuse details between father and son and attribute to James Milson Snr some of the activities of James Milson Jnr. This includes the North Sydney Historical Society which states that James Milson Snr was the Milson who co-founded the ferry service. This, however, disagrees with the details in the Obituary of James Milson Jnr. Also often attributed to James Milson Snr is that he was a keen yachtsman and active in local politics, both attributes of James Milson Jnr not James Milson Snr.

James Milson Jnr was a prominent figure among yachtsmen from the early 1830s. His obituary states that he may be described as the "doyen" of Australian yachting.

James Milson Jnr described regattas from as early as 1834 when races were between ship's boats provided with temporary keels. By 1836 there were several small yachts afloat on Port Jackson (Sydney Harbour) and an informal association of yachtsmen including James Milson Jnr, the first approach to a yacting club in Australia.

James Milson Jnr's first yacht was the "Sophia" named after his elder sister. He raced the "Sophia" in the 1st Anniversary Regatta of the informal association in 1837. In 1841 he was winning races in his second yacht the "Friendship" a famous 12 ton decked boat, built especially for him by Mr George Green, of Lavender Bay. In 1855 James Milson Jnr returned from England with the "Mischief", an 11-ton cutter, the first English yacht in Port Jackson, which won many of the races into which it was entered. In 1859 he built the "Era" which came out in framework from England. This 25-ton yacht was one of the best-known craft in Sydney in the 1860s.

In 1856-57 the first formal yachting club in Australia, the "Sydney Yacht Club" was formed with Mr Hutchinson Brown, police magistrate, as commodore, and James Milson Jnr as one of the members. Then on 8 July 1862 was founded "The Australian Yacht Club" when nineteen yachtsmen, including James Milson Jnr, met in the office of The Hon William Walker MLC who became the first Commodore. James Milson Jnr took a leading part in the formation of the association and was the first vice-commodore. Royal patronage was sought. On 27 June 1863 a letter was received from England notifying of the Prine of Wales' willingness to be patron, and designating the club as the "Royal Sydney Yacht Squadron". In 1863 James Milson Jnr was made Commodore.

The Royal Sydney Yacht Squadron first began to lease a property as its base in 1902 when it ontained a lease on a property at Wudyong Point on the eastern side of Kirribilli Point. Wudyong Point is on the 120 acres that James Milson Snr had originally leased from Robert Campbell. The site was formally occupied by the Royal Yacht Squadron on 24 January 1903, just 12 days after the death of James Milson Jnr on 12 January 1903.

(Elizabeth Warne is not used as a reference in this section as the details in her book are incorrect.)

Burial

Both James Wilson Snr (1783–1872) and James Wilson Jnr (1814–1903) are buried in St Thomas' Rest Park, West Street, Cammeray just 3.5 km inland from Milsons Point.