In 1859, George Hepworth, a local Brighouse architect received instructions from Joshua Garside Wilson to design a new mill at Rosemary which would meet the needs of his growing business. An advertisement was placed in the Huddersfield Chronicle on the 30th July where builders were invited to tender for the work of erecting the new building.

The old mill was torn down and a new Rosemary Mill replaced it. The stream that powered the old water mill and fed the public baths was utilised to fill a new mill dam which ensured there was sufficient water for the modern new steam boiler which powered the machinery. Joshua Wilson’s brother-in-law, John Craven was a partner in the business as he is recorded as residing at Ivy House in 1861 and an employer of 30 men, 28 women and 12 boys. A later newspaper report about a fire at the mill in 1868 names it as Messrs. Craven & Wilson, manufacturers. Craven retired in 1870 and went to live in Brighouse at which point Wilson moved into Ivy House to be closer to his business premises.

Wilson resided there until his death in 1898 after which the mill was occupied by Armitage Silk Spinners who eventually sold out to Walshaw Drake, a company that had started by renting a disused weaving shed at Rosemary Mill in 1903.