Generally it's always been hit or miss on these cards. There's 2 things that happen with the unlock. One is unlocking more cores in the GPU, and the second is it changes the clocks, both the GPU core and memory clocks, as well as memory timing (supposedly).
This means that first off, the disabled cores might be unstable. So there's no way of knowing if unlocking them will work at all. Secondly, the VRAM might not like the faster speeds at all, so even if the core unlock is successful it might still crash and burn.
There are ways to unlock only the cores, I believe I read about it at Techpowerup.com.
As for overclocking, it's kind of in a similar boat. I'd suggest you read some overclocking guides, you can find many here at Tom's or do a Google search.
Please note that you can seriously screw your card up by flashing the BIOS and/or overclocking, so it's definitely a risk.