Like kfridge says, maybe it's people looking for your server. If you were listed in All Seeing Eye or one of those type of programs, then maybe it's people who had you in their list and are connecting to get pings.
Try running a netstat and see if they're all connecting to the same port, and if so, which port it is.