Sites linking to you, shouldn’t hurt your rankings. There is a new phenonomen called ‘Google Bowling’ – well worth reading. However in my experience SEO wise other than page hijacking there is nothing anyone can do offsite to harm your rankings. Google also now seems to have finally got a hand on page hijacking as well