Well the electrical properties of the materials that make up a CPU or GPU change when the heat increases or decreases, usually resulting in a much higher resistence and much lower reliability. When you have more resistence in a carefully designed processor there is lag on longer "lines" and so data may not reach its final destination at the right time which causes certain pixels to be what they should have been a nanosecond earlier. The problem becomes worse because today's GPUs have extra processing that is done to the image such as AA or AF, which could make one messed up pixel turn into several.
The VRAM is also affected by temperature in the same way, it is trying to run at whatever speed it is set to run at with whatever timings it is set to run at and the GPU tries to access the memory at the speed it is set to run at and with the timings it is set to run at, but when it is overheated or overclocker too far and the GPU tries to acces it and read (or write) whatever data is stored on it (usually just textures and vertices data) the memory may not be able to send the data to the GPU as quickly as the GPU would like to have it and so you get tearing and "point-miscalculations" that look like spikes or triangles all over the screen.
That's about as basic as I can put it while still having it accurate enough to be said...