Bring a large pot of salted water to a boil over high heat. Add the potatoes, allow the water to return to a simmer. Adjust the heat so the water remains at a simmer and simmer until the sweet potatoes are soft, about 20 to 25 minutes.
Drain the potatoes and either return them to the pot with the garlic and mash with an immersion blender or a potato masher, or put the potatoes through a ricer or a food mill with the garlic, doing this so the riced potatoes fall back into the pot. Add the olive oil, butter, milk, salt, and pepper and stir to thoroughly combine. Serve hot.