Of course google can penalize you for the links going to your website. Look at webmasterworld or any other seo forum, if your backlink count is growing too fast compared to historical levels google assumes you are spamming. But in this case I am not so sure it would hurt.