Chief financial officer Colette Kress will take over as interim chief executive as Nvidia looks for a permanent successor, the US company said on Thursday.
It named former Synopysys chief executive and board member Harvey Jones as its first-ever board chairman.
Huang, who co-founded Nvidia and is its fourth-largest investor with a stake of about 3.61 per cent, will exit the board as well.
Nvidia shares in the US fell more than 14 per cent at the opening bell on Thursday, before recovering to a decline of about five per cent.
Pressure had been mounting on Huang in recent days after reports said the 63-year-old, who is married, had an affair with businesswoman Alice Huang who US authorities suspect of spying for China.
Nvidia's board met on Wednesday to consider his ouster, a person familiar with the matter said, after reports earlier in the day that Huang also had an extramarital affair with a senior company executive who reported to him.
Nvidia's chips dominate the data centres that train and run AI systems, placing it at the centre of both the US-China technology race and a massive wave of investment reshaping the US economy.
The company also invests some of its billions in profits in other AI firms that in turn buy its chips, raising the stakes for the wider tech sector.
The departure could also have far-reaching consequences for US chip export policy, given Huang's push to preserve access to the Chinese market and his close ties with US President Donald Trump.
Revelations that US intelligence was probing Alice Huang for facilitating shipments of Nvidia chips to Chinese firms set off a series of events.
Wall Street began raising questions about Nvidia's governance and Jensen Huang's personal conduct while the company's board launched an internal investigation and hired outside counsel.
US politicians also began to push for tighter curbs on chip exports to China.
Chinese authorities have denied the allegations.
Alice Huang is currently living in Beijing with the knowledge of security officials, Bloomberg News reported, citing people familiar with the matter.
Chinese foreign ministry spokeswoman Mao Ning said in Beijing on Thursday that the US should stop making up "so-called spy stories" to justify technology blockades
The leadership upheaval comes just days before investors are set to hear from Nvidia on its financial performance and outlook, with its quarterly results due on November 17.
Earlier on Wednesday, the New York Times reported that Huang had also had an extramarital affair with a company executive who reported to him.
The relationship was with Kimberly Powell, Nvidia's vice president of healthcare, the New York Times said, citing five people who worked at the company.
with DPA and AP
