Frontman Liam Og O hAnnaidh, whose stage name is Mo Chara, was charged with a terrorism offence last month for allegedly displaying a flag in support of Lebanese militant group Hezbollah at a concert in November while saying "up Hamas, up Hezbollah".
He denied the charge.
Starmer told the Sun newspaper this month it was "not appropriate" for Kneecap, who rap in Irish and English, to appear at Glastonbury.
With the outdoor arena at full capacity and scores of Palestinian flags waving in the blazing sun, the Northern Irish trio took to the West Holts stage at the festival in southwest England on Saturday.
"Glastonbury, I'm a free man," O hAnnaidh, wearing a keffiyeh, told the crowd.
Member Naoise O Caireallain, who performs under the name Móglaí Bap, said: "The prime minister of your country - not mine - said he didn't want us to play so f*** Keir Starmer."
Access to the area around the West Holts Stage was closed about 45 minutes before their performance after groups of fans arrived to form a sea of Irish and Palestinian flags.
Rap punk duo Bob Vylan performed on the stage before Kneecap and led the crowd in chants of "Free, free Palestine" and "Death, death to the IDF".
Speaking before the performance, festival-goer Greg Robertson, 30, said: "I don't think politicians should really have too much of an impact on a weekend where everyone's trying to have fun and trying to maybe create a more optimistic future."
Sara Majid, 29, said she liked what Kneecap stood for.
"I'm intrigued by them," she said.
Opposition leader Kemi Badenoch had said the BBC public broadcaster, which is showing the festival, should not feature Kneecap, and 30 music industry bosses asked organisers to pull the band from the line-up, according to a letter leaked by DJ Toddla T, cited by the Guardian newspaper.
In response, more than 100 musicians have signed a public letter in support of the group.
Kneecap's manager Dan Lambert said the group had expected calls for the performance to be cancelled.
"We knew that the biggest pressure would come on Glastonbury because Glastonbury's an institution," he told Reuters, adding that it had not been an issue for organisers.
"We didn't bring it up and they didn't bring it up," he said.
"They treated us professionally."
Organiser Emily Eavis said on Wednesday the festival was a platform for artists from all over the world, adding that "everyone is welcome here".
The BBC said on Saturday it would not be live-streaming the set but said the performance is likely to be made available on-demand later.
The footage of O'Hanna displaying the flag came to light after the trio projected pro-Palestinian messages on stage at the Coachella Valley Music and Arts Festival in California in April, ending with: "F*** Israel. Free Palestine."
Kneecap, which also includes DJ Próvaí, have said they do not support Hamas or Hezbollah.
O hAnnaidh said on Friday the group were "playing characters" on stage, and it was up to the audience to interpret their messages.
Also playing on Saturday afternoon were Leeds rock band Kaiser Chiefs and US star Brandi Carlile, who released an album with Sir Elton John earlier in the year.
with PA
