Hi Joe,
Maybe I’m butting in where I shouldn’t but I figured I would give it a shot.
Here’s Google’s terms on the use of Robots.txt files.
http://www.google.ca/support/webmasters/bin/answer.py?answer=40364
It’s pretty interesting stuff. I hope it helps.
As for the dance.. It’s always incredibly unpredictable and hurts so many. Unfortunatly, that’s what happens when most people only use one or two search engines. It makes or kills your business is an instant. 