Everyone does, that's totally legit. However, even without editing anything, just "moving around the file" is much snappier with vim. Even when I'm reading code I'll move to certain parts of a line to better understand what's going on. Plus, you do eventually have to edit code.

If you really never did anything except read code, then I think vim wouldn't give much of an advantage. Then again, neither would any of the shortcuts that every IDE provides.