Yes, I am working on using numpy to do the projection using matrices so we dont have to loop over each point and project it individually. That should be a big boost.

The way I handle the different camera images is to simply see which one provides a lower depth and use - with the idea that if the camera is closer, it would provide better information. But what you are suggesting is pretty interestint. I'm going to try that as well.