To be honest I never noticed Panda, Caffeine, Panda 2, or 2.5 and now Fresh. My site keeps growing, results keep growing as well and most of my traffic comes from Google.
And when I come to think about it, why should I be affected? I update daily, I write quality long posts all the time, and I have had it optimized for speed since the very beginning.
I believe these updates affect mostly people who decided once to take the shortcut way, say short content, or update once a month, stuff like this. In terms of searches, who knows, maybe it has knocked out a competitor or two, but I always stay focused on me.