"We have taken the power," Colonel Michael Randrianirina, who led a mutiny of soldiers joining anti-government Gen Z demonstrators, said on state radio.
Randrianirina added that the military was dissolving all institutions except the lower house of parliament or National Assembly, which voted to impeach Rajoelina minutes earlier.
In a day of turmoil for the country off east Africa, the 51-year-old leader had sought to dissolve the assembly by decree.
Despite flying out on a French military jet, Rajoelina is refusing to step down in defiance of weeks of Gen Z protests demanding his resignation and widespread defections in the army.
The presidency did not immediately react to Randrianirina's comments but earlier said the assembly meeting was unconstitutional and thus any resolution "null and void".
Rajoelina has said he has moved to a safe place because of threats to his life.
An opposition official, a military source and a foreign diplomat told Reuters he had fled the country on Sunday aboard a French military plane.
Demonstrations erupted in the country on September 25 over water and power shortages and quickly escalated into an uprising over broader grievances, including corruption, bad governance and a lack of basic services.
The anger mirrored recent protests against ruling elites elsewhere, including Nepal and Morocco.
On Tuesday, at Antananarivo's 13 May Square, along the main drag lined with palm trees and French colonial buildings, thousands of protesters danced, marched, sang songs and waved banners denouncing Rajoelina as a French stooge because of his dual citizenship and support from Madagascar's former coloniser.
Many were waving Malagasy flags and the signature Gen Z protest banner of a skull and crossbones from the Japanese One Piece anime series.
At one point, Randrianirina took to the stage and asked: "Are you ready to accept a military takeover?," drawing cheers of approval from the crowd.
Earlier, French President Emmanuel Macron had said that constitutional order must be preserved and that while France understood the grievances of the youth, they should not be exploited by military factions.
Rajoelina had appeared isolated after losing the support of CAPSAT, an elite unit which had helped him seize power in a 2009 coup.
Randrianirina is a well-known commander in CAPSAT.
The unit had joined the protesters over the weekend, saying it would refuse to fire on them and escorting thousands of demonstrators in the main square of the capital Antananarivo.
The paramilitary gendarmerie and the police also broke ranks with Rajoelina.
Madagascar, where the average age is less than 20, has a population of about 30 million, three-quarters of whom live in poverty.
GDP per capita plunged 45 per cent from the time of independence in 1960 to 2020, according to the World Bank.
