Are you using the direct download links?
Most people do not realize they are clicking on an actual download and freak out when they see someting happening on their computer and either pull the plug or do some other stupid thing to stop the “invasion” of their computer and never return to your site and tell all their friends not to.
Sorry if that is not your problem, but it sure soundslike it from here.
Do you have a link to where these downloads are originating?