For the past 90 days, I have noticed a huge increase in crawl activity on my site, from the usual 30k-50k requests a day to between 5 million and 15 million requests a day. Does anyone know why this is happening? There haven’t been any major changes to the site, just some minor content updates. I did see an error in the robots.txt file saying it wasn’t fetched for a short time, but I don’t think that’s the cause. Any ideas?
1 Like
You could determine your next course of action by examining the crawls displayed in the search console.
1 Like