Kelburn is thought to be the oldest castle in Scotland to have been continuously inhabited by the same family. Orginally the family name was de Boyville but this changed over the years to Boyle . The de Boyvilles from Caen in Normandy came over to Britain with William the Conqueror in 1066 and the present branch of the family settled in Kelburn in 1140.

Nobody knows for certain when a stone building was first constructed on the site, but the original Norman Keep, designed for defence rather than comfort, was probably built by 1200. The original Norman Keep is now enclosed within a grander castle, completed in 1581 by the then Laird, David Boyle, at a time when the family was emerging from relative obscurity and beginning to wield some influence within the local community. The 1581 castle can be clearly distinguished from the more recent parts of the building by its two impressive towers on opposite corners.

During the troubled seventeenth century, the Boyles of Kelburn became wealthy through shipping and shipbuilding. In the later part of the century, they became deeply committed to public service and John Boyle, the father of the first Earl of Glasgow, working for Customs and Excise, attempted to stamp out smuggling on the Ayrshire Coast. He subsequently became a Crown Commissioner, administering the Bute Estates and later the Argyll Estates when these families fell foul of the State.

John's son, David Boyle (1666-1733) was a distinguished Scottish statesman, a privy councillor and Lord of the Treasury among other appointments, and it was he who was created Earl of Glasgow in 1703, one of the last of the Scottish peerages. He was one of the leading figures in the forming of the Act of Union in 1707 which united the English and Scottish parliaments and he is sometimes charged with being responsible for bribing impoverished Jacobites within the Scottish Parliament to vote against their natural instincts. The First Earl was also High Commissioner to the General Assembly of the Church of Scotland and Rector of Glasgow University, an honour which may explain why he chose the name of Glasgow for his title.

It was the First Earl who made the most interesting addition to Kelburn Caste. He planned and ordered a new mansion house to be built and joined to the existing castle, with a new and grander front door facing north. It was completed in 1700, and this larger home was now called Kelburn House, because castles had become unfashoinable at the time and regarded as primitive and uncomfortable. The large drawing room, a double cube, with its high cornice and large sash windows, was designed as the grand dining room, and is regarded today by some architectural historians as the most beautiful room in Scotland. During the 18th and 19th centuries, the Earls of Glasgow became very considerable landowners.

They acquired their land either through legacies from other branches of the Boyle family or through judicious marriages to noble ladies with inheritances of their own. When the sixth Earl of Glasgow inherited the title in 1869, he also inherited all the Boyle Estates. Besides Kelburn, this included land in Dalry, Stewarton, Corshill and Fenwick, and the estate of Hawkhead outside Paisley, plus estates in Dunbartonshire, Fife, Northumberland and the greater part of Cumbrae, the island which lies directly across the water from Kelburn. He also ran six large full-staffed residences. Hawkhead in Paisley, Crawford Priory in Fife, the Garrison on Cumbrae (to which he made several additions), town houses in Perth and Edinburgh, and Kelburn itself.

Unfortunately for the Boyle family, he was caught up in the religious controversies of the day, particularly the Oxford movement, and he ran into debt building and endowing Episcopal churches all over Scotland, including a Cathedral in Perth and one on Cumbrae. By 1888 he found himself owing nearly one million pounds, and his cousin, David Boyle of Stewarton, later 7th Earl of Glasgow, sold his own lands near Irvine in order to raise the money to buy back the Kelburn Estate at auction. All the rest was lost to the family.

However, before disaster struck, the sixth Earl allowed a Victorian wing to be built onto Kelburn enclosing one of the 1581 towers. This new addition includes the impressive dining room with its original William Morris wallpaper, its family portraits and wonderful views over the Firth of Clyde.

The seventh Earl, who was Governor of New Zealand from 1892 to 1898, and the eighth and ninth Earls, were all distinguished naval officers. When not at sea, they spent most of their lives at Kelburn, attempting to find sufficient resources to keep it maintained.

The present tenth Earl of Glasgow and his wife, Isabel, started Kelburn Country Centre in 1977, opening most of Kelburn's grounds and gardens to the public and introducing new attractions. The castle too has recently been open for guided tours at specific times of the year and it is available for functions and private parties at all times.