Problem is you’re all looking at this from the point of view of what you think is best for you, not what’s best for your customers.
The answer to this depends on many things, but the major thing is how savvy are your visitors? If they are internet freaks then they know they can right-click / middle-click, open in a new window/tab if they want, and doing it for them automatically is likely to annoy them and they won’t come back. They are also likely to appreciate a little icon/indicator that a link will take them out of the site. This type of user also has little difficulty finding your site again after navigating away.
On the other end of the scale if they are relatively tech-illiterate then they are just happy to click around and likely have popups all the time – and opening in a new window does as you say, keep your site visible so they can easily find it again.
So the answer is “it depends”. Get to know your customers and you’ll have a better idea of the right answer to this question for your site.