We are know bounce rate and time on site is a factor the search engines are looking at. I have my suspicions that they are utilizing those factors in a scalable form to factor into the algo. It is simply not as black and white as s measurement of clicks and traffic to a site from the SERPs. Otherwise it would simply be a self-fulfilling prophecy and only the sites that rank would continue to rank.
You should really be more concerned with the user experience on your site and how well that traffic converts from the various referral sources. This factor in itself, if present is very minor and easily exploited with botNets and click fraud. If it is easily to game, then it will more than likely not have much if any impact
~scott polk