Known for his behind-the-scenes manoeuvring during a career lasting more than three decades, Mandelson was forced from Britain's most desirable diplomatic post after some of his letters and emails to Epstein were revealed this week.
Starmer, struggling in opinion polls after numerous setbacks, had strongly backed his ambassador on Wednesday as a state visit by US President Donald Trump, with whom Mandelson had developed strong ties, looms next week.
Trump has also faced questions about his links with Epstein, with the White House denying that an alleged birthday letter from him to the late financier is authentic.
Mandelson, who was key to the Labour Party's success when Tony Blair was prime minister, had come under scrutiny after US lawmakers released documents including a letter calling Epstein "my best pal".
Emails were published in the media showing that Mandelson had advised Epstein to fight for early release in 2008 when he was about to be sentenced to 18 months in jail for soliciting a minor.
"The emails show that the depth and extent of Peter Mandelson's relationship with Jeffrey Epstein is materially different from that known at the time of his appointment," Britain's foreign ministry said on Thursday.
The reporting of Mandelson's suggestion that Epstein's first conviction was wrongful and should be challenged was "new information", it said, adding that Starmer had asked for his removal.
On Wednesday, Mandelson had said he deeply regretted ever meeting Epstein, and that he had carried on the association "for far longer than I should have done" - a statement that seemed to satisfy the prime minister.
Starmer had heaped praise on Mandelson, the first political appointee to the position in almost half a century, for his efforts in securing a trade deal with the United States.
But after twice being forced out of cabinet under Blair, Mandelson was no stranger to controversy.
Last week the departure of Starmer's deputy, Angela Rayner, over her tax affairs forced Starmer into a wide cabinet reshuffle and overshadowed what had been intended as a relaunch for his Labour government.
As economic growth fails to ignite and potential tax rises loom, Nigel Farage's populist Reform UK has overtaken Labour in opinion polls, and Starmer's party is growing restless.
1800 RESPECT (1800 737 732)
National Sexual Abuse and Redress Support Service 1800 211 028
