The bomb squad was tasked following the discovery of a suspicious object at St John’s Catholic Church in Coleraine.
It was found by a gravedigger as he dug one foot down into the grave ahead of a burial today.
Police say the object was an “old style grenade”.
Also uncovered were two handguns and a quantity of ammunition.
The PSNI believes the cache had been in the grave quite a long period of time.
The items have been taken away for forensic examination and an investigation has been launched to track down those behind the hide.
Police have now removed the cordon around the scene on the Laurelhill Road which was first erected on Tuesday when the alarm was first raised.
No roads were closed and no homes were evacuated during the ongoing alert.
Mayor of Coleraine George Duddy – who himself been the target for sectarian graffiti attacks in recent months – said the gravedigger found something as he was starting to open the grave on Tuesday.
“He wasn’t any great depth down, approximately a foot, when he came across something which he believed suspicious enough to contact the police,” he said.
“There is obviously going to be somebody buried there tomorrow and it may well have some form of delay in regards to that burial.”
There have been heightened tensions in Coleraine in recent weeks following a loyalist band parade in the town.
A number of people have been arrested by detectives involved in the probe which has been codenamed Operation Chimera.