” thought it may affect the relevance of my sites if a bunch of totally unrelated and unprofessional looking sites were linking to mine.”
Yes, this is not good if the degree is screwed up. If you had one link from a gambling site and 100 links from a skateboard site, that would hurt you in terms of a search engine understanding what you site is about. It is basically true that sites can’t hurt you, but this is a case where it can. For example, if no one else linked to your site, but 100 skateboard domains linked to you with the text “red fuzzy skateboards” you would definitely rank better for fuzzy skateboards than blackjack counting strategies.