The Court of Civil Appeals refused James Davis' appeal in an unsigned decision released Friday.
Davis could now ask the Alabama Supreme Court to intervene, but he didn't immediately return a phone call seeking comment on his next move.
Davis buried his wife Patsy in their front yard in Stevenson after her death in 2009. The city contends the grave is illegal and wants him move it to a cemetery.
A Jackson County judge sided with the city in saying Davis should move his wife's body, and the appeals court declined to overturn that decision.
Davis has said he won't move the grave.