Preliminary results show Merz's Christian Democrats (CDU) crashed out of the parliament in the north-eastern state with just 4.9 per cent of the vote, down from 13.3 per cent in the last election in 2021 and just short of the five per cent threshold required to win seats.
The AfD earned around 38.2 per cent of the vote, ahead of the centre-left Social Democrats (SPD) on 35.5 per cent, according to the election commission.
The populist Sahra Wagenknecht Alliance (BSW) also missed parliament, with 4.8 per cent, while the Greens garnered 5.7 per cent of the vote.
The Left - which also defeated the CDU to win the state election in Berlin on Sunday - was on 6.5 per cent.
The result is a nightmare scenario for Merz, whose political future has been the source of intense speculation as his approval ratings have plummeted in recent months.
Long the largest party in post-war Germany, the CDU has never previously failed to hit the five per cent threshold in any state, European or federal election. Its previous record low of nine per cent was set in Bremen in 1951.
Merz admitted the result was a "disaster" but insisted he would push forward with his government's unpopular economic and social reform package, signalling he does not intend to resign.
He conceded that it was not possible to "sugar-coat" Sunday's result, but vowed "to prepare our country for the future".
"What needs to be done now requires backbone, steadfastness and patience. I will muster these qualities - as chairman of the CDU and, above all, as chancellor of our country."
Leif-Erik Holm, the AfD's candidate for state premier, hailed a "wonderful evening" on election night, saying his party would offer to hold talks with all other factions.
with AP and Reuters
