I don’t use direct download links except where I give a well marked choice – people need to know they are starting a download.
I think the bad conversions from direct downloads come from the fact that players are often startled by them and figure their browser has been hijacked and bail out asap.