What can hurt your site the most is linking out to banned sites and having duplicate content, both big no-no’s to Google.
Now this is quite wrong – linking to banned websites will not ban your website – look at CAP for example – half of the people’s sigs are banned websites, and CAP is doing just fine. And there are quite a few websites in Google’s top 20 which have 10+ HOME PAGE links to banned websites – and they have been doing well for a long time.