They shouldn’t care. If they do care, it means they’re using your link to boost their PR, for SEO reasons, which can ultimately allow them to compete against you in the search engines.
I would add rel=”nofollow” to all of your affiliate links. If the operator objects, I’d drop them, because it would be clear at that point what their intentions were.