The origin of the name Fraser is believed to lie with a knight called Frezel from the lordship of La Frezeliere in Anjou, who came to Scotland in the 11th century.

In 1160 the name of Simon Fraser is recorded as the owner of Keith lands in East Lothian, and he gave the church there to Kelso Abbey. Simon became a popular name for Frasers. His grand-daughter became the heiress to the Keith lands and they passed to the family who would adopt Keith.