Why call it Garden Island?

Raising the Union Jack on 26 January 1788, Governor Philip quickly sent a party out looking for areas suitable for growing fresh vegetables for the colonists. It had been a long, arduous journey from Portsmouth and scurvy was rife. Garden Island stood about 300m off Potts Point as can be seen in this undated image taken from The Domain showing Potts Point on the right and Garden Island in the left middle distance. Weathered over the years through neglect but now encased behind perspex and glass, the first rock graffiti by white man (WB, IR and FM) indicates that a party of 3 had endeavoured to till the land as early as 11 February 1788, eventually growing turnips, carrots, lettuces, onions, leaks, parsley, celery, corn, five sorts of cabbages, artichokes and beets. However, as the island was not particularly arable, the colonists not particularly adept and the weather unforgiving, the infant colony was plunged into two years of severe deprivation waiting upon every arrival from "home" for basic foods.
Wednesday: When and why did it cease being an island?