Sri Lanka hit back with two quick wickets after West Indies had dominated the morning session but rain intervened to spoil the momentum.
Only nine overs of play was possible after lunch. Shivnarine Chanderpaul posted his 55th test half-century, his first in the series, in the first over after lunch when he swept left-arm spinner Rangana Herath for four.
But he was out for 54, caught by Mahela Jayawardene off spinner Ajantha Mendis, ending a 99-run fourth-wicket stand with Brendan Nash, who remained unbeaten on 62.
Dwayne Bravo was out without scoring, stumped by Prasanna Jayawardene. Nash who faced 110 balls and hit five boundaries earlier went past 1,000 career runs in tests.