Have you tried backing off on the
ram completely, to stock speed, then overclocking the core alone as high as you can, then after that overclocking the
ram?
Also, what are your voltages like?
Nvidia cards like a little overvolt on the AGP voltage, and also a higher than spec +12V rail.
__________________
Notebook: Dell XPS M1330 Running Fedora 11 Linux (Leonidas)
Desktop: ASUS Rampage Formula X48
Intel Core 2 Quad Q9450 (Yorkfield) @ 3.51Ghz
4GB DDR2-800 PNY
Albatron 8800GT 512MB
Corsair HX-620 PSU
Running Fedora 11 Linux (Leonidas)
Dual Dell 2407WFP 
Drivers, Games, Demos, Mods and Overclocking Tools At AOAFiles