Wondering why your website isn't being spidered? Your firewall may now be part of the issue. We just ran into the issue of a few servers housing a few hundred websites get de-listed from the search engines shortly after these firewalls were installed. At first thinking it was a server wide ban then realizing it wasn't we had to figure out what had changed and the only server changes were the addition of the firewall.
SonicWALL Firewall Blocks Spiders
