How to update related data using entity framework 6

I have an entity "Person", which has a related entity "Hobby". A person can have several hobbies. I want to create a view model that allows new hobbies to be added and/or existing hobbies to be removed from a Person entity.

The code below works, but I imagine it is rather inefficient.

var newHobbies = new List<Hobby>();

foreach (Hobby hobby in vm.Hobbies)
{
    var originalHobby = db.Hobbies.Find(hobby.HobbyID);
    newHobbies.Add(originalHobby);
}

originalPerson.Hobbies = newHobbies;

I prefer to do something like this:

var newHobbies = db.Hobbies.Where(x => vm.Hobbies.All(y => x.HobbyID == y.HobbyID)).ToList();   
originalPerson.Hobbies = newHobbies;

But I get an error: Only primitive types or enumeration types are supported in this context.

How can I update related data without going to the database multiple times?