We can probably agree that Feynman's cancer wasn't directly due to his work on the Manhattan Project, but not for the reasons you cite. He died >40 years after the Manhattan Project. Feynman tells several anecdotes which indicate that he and others may have had significant exposure, and that he certainly had many opportunities for exposure. The Manhattan Project was unlike any other science project to follow in that not only was worker safety much less of a concern but the danger was also less well understood than in any successive project. Additionally, all involved persons were encamped together at Los Alamos, Oak Ridge, Hanford, and Argonne, working closely together on a daily basis, milling about each others' labs, sharing other facilities. It is lucky, with the safety regime in place that more people were not killed or harmed.
I'm not sure what you mean; that was exactly my point. His exposure was likely minimal (I can't recall anything suggesting he was close to any criticality assemblage tests), but his exposure level is still more or less irrelevant. It's basically impossible to separate anyone involved in the Manhattan project that didn't die immediately from exposure from expected background population cancer rates.
That's stated pretty explicitly--even for those few most directly exposed in accidents--in the discussion section of the paper you link (which my first linked paper was a followup study to, 20 years later, btw), and it was found to be true in the larger population of nuclear material workers in the second paper (who were, indeed, working in an environment where much more was known about how to control the dangers of what they were working with).
"worker safety much less of a concern but the danger was also less well understood"
Another important point is to people who don't understand the physics, something not understood "must" be feared, therefore the most dangerous part must have been the physics. However I understand at least some of the chemistry and industrial processes using bulk hydrofluoric scare me more than the physics. Aside from the mechanical / explosive hazards, of course (machine plastic explosives to precise shapes on a milling machine for the implosion weapons, are you insane? You're hooking up a zillion blasting caps to .. what?)
http://www.orau.org/ptp/pdf/accidentsurvivorslanl.pdf