The judge at Aarhus district court ruled the cartoons were neither offensive nor were they intended to denigrate Muslims, according to court papers.
"Even if the text accompanying the pictures could be read as being derogatory and mocking, the cartoons are not offensive," the court said.
Seven Danish Muslim groups launched legal proceedings in March against Jyllands-Posten's editor, Carsten Juste, and cultural affairs editor Flemming Rose.
They accused the paper of publishing text and cartoons which were "offensive and insulting" to the prophet (PBUH).
The plaintifs said the satirical caricatures "attacked believers' honour because they portrayed the prophet as war-like and criminal and made a clear link between the prophet (PBUH), war and terrorism."
The groups brought their case citing Denmark's blasphamy and racism laws.
Before Thursday's ruling the case against Jyllands-Posten had already been rejected by two regional prosecutors and Denmark's prosecutor-general.
A spokesman for the plaintifs, Kasem Said Ahmad, said the verdict was disappointing.
"The court gave Jyllands-Posten the right to attack Muslims and their feelings and linked us to terrorism," Said Ahmad said.
The groups planned to appeal the decision, he added.
One of the cartoons features the prophet (PBUH) wearing a turban that holds a bomb with a lighted fuse.
"We will explore all options that we have under the law to overturn this judgement," Said Ahmad told Danish news agency Ritzau.
He expected the ruling to "trouble" Muslims in Denmark and abroad, in that it suggested it was now permitted to insult the Muslim faith.
Said Ahmad stressed that the aim of the case was "to condemn Jyllands-Posten and to gain an understanding of our religion in society."
"Our feelings and our religious icons must be taken into account," he added.
The case was heard in Aarhus, the town where Jyllands-Posten is based.

Copyright AFP (Agence France-Presse), 2006