Kumar Sangakkara was 28 not out but Sri Lanka lost opener Tharanga Paranavitana for 9 and Mahela Jayawardene for 14 early in their second innings.
Sri Lanka were still pushing towards a commanding lead that would give them a chance of a series-levelling victory.
Sri Lanka has never won a Test in South Africa and fell to an innings defeat in the first match before bowling the Proteas out for 168 on day two in a dramatic turnaround.
Morne Morkel removed Paranavitana and Jayawardene was leg-before-wicket to Marchant de Lange (1-16) for the debutant fast bowler's eighth wicket of the match.