Chris Froome could also thank his Team Sky colleagues for keeping him out of danger - and helping overcome a mechanical problem - in the race's first hilly stage.
Greg Van Avermaet mistimed his sprint and Sagan, the three-time reigning world champion with the Bora-Hansgrohe team, took advantage to claim his second victory in this year's event.
"I don't know if he did it on purpose or if he wanted to drop everybody but I have to say thanks," Sagan said.
Van Avermaet still increased his overall lead over BMC teammate Tejay van Garderen to two seconds by gaining precious time in an intermediate bonus sprint.
After a move from Philippe Gilbert, Van Avermaet accelerated out of the final turn with 300 meters (yards) to go but couldn't maintain his pace.
"I tried to win the stage but it was pretty complicated," Van Avermaet said. "Phil went early and he's still pretty close on GC (general classification) so I couldn't let him go. I think I went too early in the sprint. I thought the corner was closer to the finish than it really was."
In a repeat of Stage 2, Sagan held off Sonny Colbrelli in the sprint, with Gilbert finishing third.
Sagan clocked nearly five hours over the undulating leg through Brittany, which contained five categorized climbs over a 204.5-kilometer (127-mile) route from Lorient to Quimper, and resembled a single-day classic.
Gilbert, a former world champion, moved up to third overall, three seconds off the pace.
Froome, the four-time champion who trails Van Avermaet by 57 seconds in 15th spot, finished in the main pack with the other favorites.
There was a nervy moment for the British rider when he had to stop due to a mechanical issue with 70 kilometers remaining.
Teammate Michal Kwiatkowski attempted to fix the problem but Froome was later forced to stop again and change his bike.
Kwiatkowski and two other teammates then quickly escorted Froome back to the peloton.
Froome and Sky were also at the front in the finale, ceding the lead to the stage hunters only in the final kilometer.
"Everyone knew that was going to be a tricky finish today, a very undulating day - never really flat - just up and down," Froome said. "A nice day to have behind us.
"It's the kind of stage where something can be wrong and you quickly find yourself losing 30-40 seconds if you get caught out of position. It was a day to stay up front and really stay on it."
Seven riders broke away eight kilometers into the stage. The group established a lead of about four minutes as they wound along the Atlantic coast through picturesque towns like Riec-sur-Belon - which is famous for its oysters - and Pont-Aven - which inspired artists like Paul Gauguin .
While Brittany is usually rainy and windy, this time it was sunny and racing conditions were perfect. Fans waving the black-and-white striped flag of the region swarmed the climbs.
Sylvain Chavanel, the 39-year-old riding in a record 18th Tour, launched a solo attack from the breakaway with 100 kilometers remaining but didn't get far.
French rider Elie Gesbert, another member of the breakaway, crashed on a short descent, ending up in a ditch and appearing to scrape his face along a stone wall. But he got back on his bike and continued.
Lilian Calmejane and Tom Skujins, the last remaining members of the breakaway, were caught by the main pack with 10 kilometers to go.
The Tour remains in Brittany for Stage 6 on Thursday, another hilly leg of 181 kilometers (112 miles) from Brest to Mur-De-Bretagne Guerledan, including two climbs of the short but steep Mur-de-Bretagne.
"We always see a bit of action there," Froome said. "I sometimes prefer those stages to the fast, flat stages."
This story has been corrected to show it was the 5th stage, not the 4th.
Associated Press writer Joseph Wilson contributed.
More Tour de France coverage: https://apnews.com/tag/TourdeFrance
Copyright 2018 The Associated Press. All rights reserved. This material may not be published, broadcast, rewritten or redistributed.