Audit & Technical SEO · Glossary term

Crawl explained without jargon.

The process by which a robot (called a "crawler") goes through a site's pages, following links, to index them. Googlebot does this daily; our app does it too when you run an audit.

Extended definition

The process by which a robot (called a "crawler") goes through a site's pages, following links, to index them. Googlebot does this daily; our app does it too when you run an audit.

Context and application

A crawl starts from a seed page (usually homepage), reads all links, visits the linked pages, repeats. Your local crawler respects robots.txt, has identifiable user-agent, and self-rate-limits (max 10 req/sec) to avoid overloading the server. Major difference from Googlebot: you run crawl on known sites (auditing them); Googlebot discovers unknown sites through backlinks from already-indexed sites.

In practice

if your site has 500 pages, a full crawl takes 2-4 minutes.

How SEO Master PRO MAX helps

For Crawl, the relevant module in the app is Audit module. See the dedicated page for how it works concretely on your site.

See Audit module →

Related terms

Crawler, Crawl budget, Sitemap.

Now that you know what crawl is, apply it on your site.

SEO Master PRO MAX runs 100% locally on your computer. Full audit in Free plan, no account, no card, no page limit. Multi-AI integration with your own keys.