In a dramatic last-32 match on Wednesday during which the four-time major winner required treatment on her taped-up right knee, Li came from 4-2 down in the second set to shock the Belarusian 3-6 6-4 7-6 (7-5).
The American eventually clinched victory off her 10th match point — but the first on her own serve.
That made it two-straight losses for top seed Sabalenka, who was beaten by Nikola Bartunkova in the round of 32 at the China Open last week.
Her previous event had been the US Open, where she lost in the final to Elena Rybakina.
Sabalenka was trailing 4-1 in the deciding set when she had treatment on her right knee that was already taped, and she was then 5-2 down before recovering for 5-5 and taking it to a tiebreaker.
Li delivered a big forehand winner on her last match point for one of the biggest wins of her career.
Earlier, second-seeded Jessica Pegula squandered three match points and lost to Jelena Ostapenko in three sets in the American's first match in more than a month.
Ostapenko won 2-6 7-6 (9-7) 6-3 in an unlikely last-32 comeback that included saving match points on three occasions in the second set against Pegula, who hadn't played since losing to Sabalenka in the US Open semis.
Iga Swiatek had a much easier time of it, advancing to the round of 16 with a 6-2 6-1 win over Magdalena Frech.
The six-time major winner started slowly against her Polish compatriot, dropping an early service game and falling behind 2-0 before winning six-straight games to take the first set.
Swiatek continued momentum by winning four-straight games to start the second set.