A CLA doesn't give them your copyright; you still own it, and you're licensing it (hence the "L" in "CLA") to them. And, at least under US law, non-remunerative license agreements can be revoked under certain situations (usually having to do with that free work being turned into profit for the licensor IIRC although I admittedly haven't thought about this aspect of copyright law in twenty years)