Hi guys, I know this might sound like a naive question. I recently learned that the data in Ahrefs and similar tools aren’t 100% accurate—they’re just estimates. After digging deeper, I found out that these tools use a bot crawler that respects robots.txt and crawls the internet to collect data, and that these bots are often whitelisted in Cloudflare.
So, I’m wondering, can I do the same thing just to collect random data about traffic and gain a better understanding of things? Is that technically doable? What kind of data could I collect this way?
First-time poster—just curious, so please excuse my lack of knowledge.
I don’t believe it is feasible. Indeed, you can build a bot or crawler to scrape the web, but it won’t provide you with traffic information. The pages that your crawler finds must be examined to determine which keyword they are ranking for, where they are ranking, and how much traffic they are generating based on some CTR and SV. which, compared to ahrefs, will be even less precise. When you say “collect random data,” what do you mean?
I don’t believe it is feasible. Indeed, you can build a bot or crawler to scrape the web, but it won’t provide you with traffic information. The pages that your crawler finds must be examined to determine which keyword they are ranking for, where they are ranking, and how much traffic they are generating based on some CTR and SV. which, compared to ahrefs, will be even less precise. When you say “collect random data,” what do you mean?
[quote=“Jafferson, post:1, topic:937”]
Hi guys, I know this might sound like a naive question. I recently learned that the data in Ahrefs and similar tools aren’t 100% accurate—they’re just estimates. After digging deeper, I found out that these tools use a bot crawler that respects robots.txt and crawls the internet to collect data, and that these bots are often whitelisted in Cloudflare.
Hi Jefferson, Building an SEO bot to gather data from websites is technically possible but involves considerable complexity and challenges.
Key Challenges:
Technical Skills: Requires strong programming knowledge and web technology expertise.
Legal and Ethical Concerns: Adherence to website terms, robots.txt, and privacy laws is crucial to avoid legal issues.
Data Accuracy: Websites frequently change, making data collection and accuracy difficult.
Scalability: Efficiently handling and processing large data sets demands robust infrastructure.
Resource Intensive: Web crawling requires significant computational resources.
Potential Data:
On-page SEO metrics, link analysis, website structure, and content analysis are examples of data a compliant bot could collect.