They absolutely did not say that non-residents should have "more say over the nature of that place." Simply and narrowly that residents should not be able to block housing development, a much different and more straightforward position.
Residents do have the most influence and the most concentrated control. That's not quite the same as being the only authority, or having absolute influence however, and this is a correct allocation of power.
Residents don't and shouldn't expect to have the final say on who can live in their area. This is explicitly acknowledged by, for example, racial housing covenants being illegal these days.
