site.com and http://www.site.com are different (even if they happen to be the same). www is technically a subdomain. You could put different content on each. The fact that hardly anyone does doesn’t really matter.
You should make sure all your links go to one or the other. If you have them going to both, you risk your pagerank being split in two, your rankings being divided between the two, and even duplicate content problems.
Bottom line, it can matter a lot (though it might only matter a little to most sites) and most basically, there is zero advantage to it so you should do what you can to be consistent.