Parliament Speaker Baleka Mbete announced that 198 MPs voted against the motion, while 177 voted in favour and nine abstained, Xinhua reported.
MPs of the ruling African National Congress (ANC) burst into applause when the result was announced.
Opposition MPs walked out of the chamber, looking crestfallen.
Democratic Alliance (DA) leader Mmusi Maimane vowed to continue the struggle to remove Zuma from office.
Hundreds of anti-Zuma protesters who gathered outside Parliament, anxiously waiting for the result, expressed disappointment over the outcome.
However, crowds of ANC supporters waiting outside Parliament were jubilant when hearing the news.
“This was a victory not only for Zuma but also for the ANC and the country,” Swowan Laura, a middle school teacher, told Xinhua.
Zuma has survived eight no confidence motion in the past seven years. But this was the first time that a no confidence motion was conducted through secret ballot.
Opposition parties had been pushing for a secret ballot so as to encourage ANC MPs to support the motion.
After Mbete gave the green light to the secret ballot on Monday, the ANC said it has full confidence in the loyalty of the ANC MPs when voting in the no confidence motion.
“We do not nor have we ever doubted their loyalty and discipline in relation to the decisions of the movement,” the ANC said.
The result of Tuesday’s voting once again showed the unity of the ANC and the loyalty of ANC MPs to the organisation, an ANC MP said on condition of anonymity.
He said that by defeating the motion, South Africa has avoided anarchy.
Under the Constitution, if Zuma was removed, the whole cabinet would resign, and if no candidate was chosen from MPs within 30 days to succeed Zuma, general elections would be held in 90 days.
The ANC has warned that any of its MPs who support the anti-Zuma no confidence motion risk being disciplined by the party, insisting that removing Zuma from office would not be in the best interest of the country.
After the vote, the ANC congratulated the Parliament for its robust engagement in the National Assembly that resulted in the “resounding defeat” of the motion of no confidence in Zuma and the cabinet.
The ANC regards it as paramount that not just the elected representatives, but all South Africans are able to engage on the most critical issues facing the country without fear of favor, ANC national spokesperson Zizi Kodwa said.
“The biggest victor of today’s event is our constitutional dispensation. It once again reaffirms the ANC’s position as the leader of society in that the country’s MP’s are able to exercise this critical constitutional provision aimed at safeguarding our democracy,” said Kodwa