The name Kirribilli is derived from an Aboriginal word Kiarabilli, which means 'good fishing spot'. The suburb initially formed in the vicinity of Jeffrey Street and was subsequently part of a grant to James Milson (1785-1872), after whom Milsons Point was named. The area was largely covered in native bush. As the decades passed, the land was cleared bit by bit and sub-divided, first for the construction of large family residences during the 1840s, secondly for the erection of Victorian terrace houses from the 1860s onwards, and finally for 20th-Century flats.
Prominent early landowners in Kirribilli included members of the Campbell, Jeffrey and Gibbes families. Colonel John George Nathaniel Gibbes (1787-1873), the Collector of Customs for New South Wales, built 'Wotonga' - the forerunner of Admiralty House - in 1842-43 on a 5-acre (20,000 m2) site which he leased, and then purchased, from the Campbells. Gibbes' son, William John Gibbes, lived nearby in Beulah House during the 1840s. Beulah was later lived in by the Riley and Lasseter families. Eventually, the house was demolished and its grounds subdivided into numerous residential blocks which were put up for public auction in 1905.