While its not great, its not quite as bad as it may sound.
Just having someones bank account number isnt really much help to anyone.
About the only thing they can do is pay money
into your account.
As noted, if you ever gave anyone a cheque, they have your account details.
Plus any company or person you pay by bank transfer has given you their details, and they dont worry about it.