Spammers often spoof Googlebot’s user-agent to disguise themselves. So, verifying the crawler’s identity by IP is important to avoid fake bots accessing your site.
3 Types of Google Crawlers & Their IP Patterns #
| Type | Description | Reverse DNS Mask Examples | IP List Reference Files |
| Common Crawlers | Googlebot and other main Google crawlers. They obey robots.txt rules. | crawl-***-***-***-***.googlebot.comgeo-crawl-***-***-***-***.geo.googlebot.com | googlebot.json |
| Special-Case Crawlers | Used for specific Google products like AdsBot, may or may not obey robots.txt. | rate-limited-proxy-***-***-***-***.google.com | special-crawlers.json |
| User-Triggered Fetchers | Initiated by user actions, e.g., Google Site Verifier, Google Cloud Platform fetches. | ***-***-***-***.gae.googleusercontent.comgoogle-proxy-***-***-***-***.google.com | user-triggered-fetchers.jsonuser-triggered-fetchers-google.json |
How to Verify Googlebot Manually (Command Line) #
- Find the IP address of the crawler from your server logs.
- Reverse DNS lookup
Run this command (replace IP with crawler IP):
host <IP-address>
- The output domain should end with one of these:
- googlebot.com
- google.com
- googleusercontent.com
- googlebot.com
- Forward DNS lookup
Use the domain you got from the reverse lookup:
host <domain-name>
- The result should return the same IP address you started with.
Example Walkthrough #
Say crawler IP is 66.249.66.1
- Reverse DNS:
host 66.249.66.1
Output:
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
- Forward DNS:
host crawl-66-249-66-1.googlebot.com
Output:
crawl-66-249-66-1.googlebot.com has address 66.249.66.1
If both checks match, it’s a verified Googlebot.
Why Verify Googlebot? #
- Protect your server from fake crawlers or scrapers.
- Ensure that crawling rules you set (robots.txt, rate limits) apply to genuine Google crawlers only.
- Helps with security and server resource management.
Pro Tip for FSIDM Students #
Automate this check for bigger sites with tools or scripts that compare incoming crawler IPs with official Google IP ranges (from googlebot.json and other published IP lists). For most small sites, manual or occasional checks suffice.