It is not known who built the structure, but the Donald, Earl of Mar held the lands in 1296. In 1306, the castle was under siege for protecting Robert the Bruce, proclaimed King of Scotland. It was taken through treachery ending the siege.

In 1335, a siege was undertaken at the castle. The lady of the keep held off attacks until her husband, Lord Atholl arrived to battle the sieging army.

1374 arrived with the castle being held by an heiress, Isobel. Alexander Stewart siezed the heiress and married her to become the Earl of Mar and hold the fortress. King James annexed the lands as part of his royal holdings in 1435, to remove some power from the Scottish nobles.

The castle has been held by several families: Stewarts, Elphinstones, Erskines. It was abandoned in 1715 after the first Jacobite uprising.

This is a view of the bake house in the foreground, the chapel after that, and finally the Wardens tower at the rear.