sooner or later, Gross stated, he was capable of stem the deluge of traffic by updating the location's robots.txt code. Robots.txt is protocol, in use For the reason that late nineteen nineties, that allows bot crawlers know wherever they could and will't go. it really is greatly acknowledged as among the list of unofficial guidelines of the net.