generally if the site wants to have a decent reputation in the industry, they wouldn’t take down your links unless it happens accidentally…
i once had all my built links deleted by google bot who crawled through a page that was previously password protected but my sysadmin fucked up, so it went through all the “delete” links and everything vanished…poof….
in any case, “the internets” is kind of a living organism so you do have to check your links once in a while. in regards to removing them…well, it just depends on who you are and what site you link to. if the site got completely trashy or turned to a banner farm, then i would remove the link. also if the site’s page has 400 other links on it, i would remove it too…
hope it helps