True, pinning the end-entity cert would require the attacker to factor the end-entity cert. But pinning the end-entity cert is like pointing a loaded gun at your foot (if you lose the key, and all your backups, you are screwed), and EFF isn't even doing it. And it would still boil down to a big performance sacrifice (which gives SSL a bad reputation) for a flawed premise.
>Note: The current specification requires including a second pin for a backup key which isn't yet used in production. This allows for changing the server's public key without breaking accessibility for clients that have already noted the pins. This is important for example when the former key gets compromised.
http://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning