Fyvie was the site of an open-air court held by Robert the Bruce, and Charles I lived there as a child. Following the Battle of Otterburn in 1390, it ceased to be a royal stronghold and instead fell into the possession of five successive families - Preston, Meldrum, Seton, Gordon and Leith - each of whom added a new tower to the castle. Inside, the castle stronghold features a great wheel stair, a display of original arms and armour, and a particularly fine collection of portraits.

Following Victorian trends, the grounds and adjoining Loch Fyvie were landscaped in the 19th century. The American industrialist Alexander Leith bought the castle in 1885. It was sold to the National Trust for Scotland in 1984 by his descendants.

The castle (like many Scottish castles) is said to be haunted. A story is told that in 1920 during renovation work the skeleton of a woman was discovered behind a bedroom wall. On the day the remains were laid to rest in Fyvie cemetery, the castle residents started to be plagued by strange noises and unexplained happenings. Fearing he had offended the dead woman, the Laird of the castle had the skeleton exhumed and replaced behind the bedroom wall, at which the haunting ceased.

Today, the castle is open to tourists during the summer months.