The Fulton Hogan quarry would be on a 170 hectare block of land bordered by Curraghs, Dawsons, Jones and Maddisons Rd.
It was first proposed for the area last November and had faced major community opposition ever since.
Garry Kilday of the Templeton Residents Association said the decision making process had been slow and meetings with Ministers had not progressed either.
"We may has well have gone out and talked to a brick wall," Kilday said.
"They are about as useless as the proverbial."
Now, in addition to the slow progress, Kilday said residents were concerned about the impact of the quarry on people's health.
Silica rock, which could be found in quarries was dangerous when exposed, and several people in Yaldhurst had already been affected, Kilday said.
"It takes away your ability to breathe," he said.
"The crystalised silica fills the capillaries in your lungs and it doesn't allow you to take oxygen into your body."
Kilday said that residents weren't backing down and would continue to fight against the quarry.