Back row forward Ollivon spent the best part of a year on the sidelines after tearing his anterior cruciate ligament at the end of last season.
The 29-year-old returned to club action during the Six Nations and earlier this month led the Barbarians in their 52-21 rout of England.
Antoine Dupont led France to a Six Nations Grand Slam in March but Ollivon will resume his role with the scrumhalf among a host of first-choice players rested by coach Fabien Galthie.
Damian Penaud, Melvyn Jaminet and Yoram Moefana retain their places from the Six Nations squad, while Bordeaux pair Matthieu Jalibert and Maxime Lucu will team up for Les Bleus for the first time.
France have never lost to Japan but drew 23-23 with the Brave Blossoms when the teams last met in 2017.
Team: 15-Melvyn Jaminet, 14-Damian Penaud, 13-Virimi Vakatawa, 12-Yoram Moefana, 11-Matthis Lebel, 10-Matthieu Jalibert, 9-Maxime Lucu, 8-Yoan Tanga, 7-Dylan Cretin, 6-Charles Ollivon, 5-Thomas Jolmes, 4-Thibaud Flament, 3-Demba Bamba, 2-Peato Mauvaka, 1-Jean-Baptiste Gros. Replacements: 16-Pierre Bourgarit, 17-Dany Priso, 18-Sipili Falatea, 19-Thomas Lavault, 20-Selevasio Tolofua, 21-Sekou Macalou, 22-Baptiste Couilloud, 23-Antoine Hastoy.
