Huddart was born at Whitehaven, Cumberland, the son of William Huddart, a shipbuilder, and his wife Frances, née Lindow. Huddart was educated at St. Bees School 1856–1860, came to Australia in 1860, and joined the coal and shipowning business of his uncle, Captain Peter Huddart, at Geelong, Victoria. In the mid-1860s Captain Huddart retired to England and James took over the business.
The business had expanded and in 1874 James Huddart was the owner of the Medea, a wooden barque of 423 tons, and next year the Queen Emma of 314 tons was also registered in his name. On 1 August 1876 Huddart joined forces with T.J. Parker (a former rival), J. Traill, and Captain T. Webb, and the firm of Huddart Parker and Company was founded, each partner had an equal interest. In 1878 the head office was moved to Melbourne, in 1880 several steamships were added to the fleet, and the business expanded rapidly. Huddart became general manager in 1886, and showed himself to be an enterprising and farsighted administrator. In 1888 the business was turned into a limited company with a capital of £300,000 each of the original partners taking up one-fourth of the shares. At the beginning of the 1890s their steamers were running to the principal ports of New South Wales, Victoria, South Australia, Western Australia and Tasmania, and in 1893 they were also trading with ports in New Zealand.
Huddart had long been interested in a proposal first made by the Canadian Pacific Railway Company in 1885, that an imperial "All-Red" route should be established between Australia and Great Britain via Canada. The suggestion touched Huddart's imagination, and in 1893 he formed the Canadian-Australian Royal Mail Line, with a contract to carry mails between Sydney and Vancouver. Huddart then tried to arrange for a similar line from England to Canada. The Canadian government supported the plan and agreed to pay £150,000, and endeavours were made to persuade the British government to supply a yearly sum of £75,000. The British post-master-general supported the plan but critics in the British government insisted that tenders must be called, and after the tenders came in the question continued to be delayed. Worn out by worry and anxiety Huddart contracted influenza, and died at Eastbourne after a few days illness on 27 February 1901. Huddart's American line had always been carried on separately from the business of Huddart Parker and Company, and he lost much of his private fortune in conducting it. His interest in Huddart Parker and Company was disposed of in 1897. He married Lois Ingham of Ballarat, who survived him with two sons and a daughter. A third son was killed in the Second Boer War.
Huddart had a strong personality, soaring ambition, and great driving power. He may, as The Times notice suggests, "have played for higher stakes than his means allowed" but he was no mere speculator; he was imbued with aspirations for the consolidation of the British Empire, and though he may have been in advance of his time he was nevertheless a great pioneer in colonial progress.