Indexability

The canonical tag (<link rel='canonical' href='...'>) tells search engines which version of a URL is the 'master' copy. If missing, Google must guess which duplicate to index. If pointing to a different URL, the current page may be deindexed in favour of the canonical target.

The canonical URL must be absolute (include scheme and domain). A relative canonical is treated as invalid by most crawlers.

The <meta name='robots'> tag controls crawler behaviour per page. Common directives:

• noindex — prevents the page from appearing in search results

• nofollow — tells crawlers not to follow links on this page

• noarchive — prevents Google from showing a cached copy

• nosnippet — prevents snippet display in SERPs

A separate <meta name='googlebot'> tag can override directives specifically for Google.

X-Robots-Tag is an HTTP response header that works the same as meta robots but applies to non-HTML resources (PDFs, images) as well. If both meta robots and X-Robots-Tag are present, the most restrictive directive wins.

A noindex in the X-Robots-Tag header is a critical issue because it blocks the page from search results even if the HTML meta tag says 'index'.

Hreflang tags (<link rel='alternate' hreflang='xx' href='...'>) tell Google which language/region versions of a page exist. They help Google serve the correct version in local search results.

Common issues: missing x-default tag, missing self-referencing hreflang, and inconsistent reciprocal links between language versions.

IndexNow is a protocol supported by Bing, Yandex, and other search engines that allows sites to instantly notify crawlers when content is created, updated, or deleted. Instead of waiting for a crawl, the search engine fetches the page immediately.

Setup requires placing a key in a <meta> tag and a verification file at the site root. The audit checks for the meta tag and verifies the key file is accessible.