x-windows starts sucking cpu

Every once in a while, my xwindows stops functioning and starts using most of the cpu power. I know this because I can ssh in from another computer (slowly) and top shows me. Killing x terminates the session and xdm starts again and everything is good again.

Does anyone know how to narrow down what is sucking the life out of my computer? X works fine normally, just once in a while it goes south.