[up-arrow] 13 points [down-arrow] by isleyaardvark 1 hour ago
This way, the up and down are unambiguously tied to the total number of points (nobody will wonder why there's a down arrow way over next to the word "link"), but are separated by enough of a margin that it's difficult to click the wrong one.
grinich's HN app for the iPhone & iPod Touch has the arrows far apart from each other (it's shown in the image next to the App Store logo and the screencast):
I think the narrow width of the iPhone makes it work, though. Your solution (also thought of by others) is the best solution I can think of for the website.
If it's a good comment it'll probably get more than one upvote, so I wouldn't worry about it too much. Stuff happens.