The mission was scheduled to last 10 hours but was reduced to three hours after the engine problem.
Launching from the United States, it was the rocket's 14th flight since 2023 as Elon Musk's company plans to put the vehicle into routine use starting later this year.
Unlike past Starship flights, the mission on Monday was both a high-stakes test of the next-generation rocket and an operational endeavour carrying millions of dollars' worth of Starlink satellites to orbit, representing Starship's first revenue-generating mission, even though its final design is still under development.
The uncrewed 40-storey-tall vehicle, consisting of a Super Heavy booster rocket topped by the main upper-stage Starship vessel, launched from SpaceX's Starbase complex near Brownsville in Texas.
The Super Heavy booster lofted the Starship upper stage into space before returning itself to Gulf waters for an engine-controlled splashdown, following hard landings for the booster on two previous missions.
Some engines on that booster stopped working during its descent, SpaceX said.
Upon reaching space, one of Starship's three main engines shut down prematurely, a development that led SpaceX spokesman Dan Huot to announce on the company's live stream of the mission that the craft would be unable to reach orbit.
"The team has made the call not to commit to orbit," Huot said.
But Huot reversed course minutes later after an update from engineers confirmed that the craft could still attempt to fire other onboard engines that would boost it into orbit, which Starship successfully did minutes later.
Once in orbit, Starship deployed 26 next-generation Starlink satellites one by one at an altitude of roughly 269km, marking the ship's first orbital deployments.
"All 26 Starlink V3 operational satellites deployed and operating nominally," Musk wrote on X.
"Deploying all 26 Starlink V3 satellites into target orbit is a huge win, especially since this marks Starship's very first revenue-generating commercial flight," said Dean Sladen, quality manager and aerospace engineer at Accu Components.
"But from an engineering perspective, the real story was how the team handled that engine failure during ascent."
Engine issues have popped up on some of Starship's previous suborbital flights in space, a lingering challenge for the program in an area crucial to the rocket's ability to manoeuvre in space.
While that function is less critical for low-orbiting Starlink deployments, it is crucial for Starship's future assignments of taking NASA astronauts to the moon.
Starship's controlled descent into the Pacific Ocean - seven hours earlier than planned - culminated with a thruster manoeuvre that flipped the vehicle vertically so that it hit the sea in a nose-up position.
Fire engulfed the base of the rocket as it touched down on the ocean surface before tipping on its side, followed by a large fireball.
The company had not planned to recover and reuse the spacecraft from the sea following the mission.
Starship was built to launch SpaceX's Starlink satellites more frequently and carry future NASA missions to the moon.
Reaching orbit, a primary benchmark for any new rocket designed to deliver payloads to space, was a long-delayed feat for Starship, which Musk once predicted would be achieved in 2022, to be followed by satellite deployments in 2023.
All 13 of Starship's previous test flights since 2023 were limited to suborbital space in a nearly decade-long development campaign that has cost SpaceX more than $US15 billion ($A21 billion) to date.
Musk has said he envisions ultimately flying hundreds or thousands of Starship missions a year, a launch pace unprecedented in the space industry.
"Starship is 2 to 3 years away from hourly flights," Musk wrote on X on Sunday.
