Robots Meta Tag

The robots meta tag is an HTML element in a page's head — <meta name="robots" content="..."> — that controls how search engines crawl, index, and display the specific page. Unlike Robots.txt, which controls crawler access at the URL level before any fetching, the robots meta tag operates per-page after the crawler has already loaded the document.

Common directive values: index (allow indexing — the default), noindex (do not include this URL in search results), follow (follow outbound links — the default), nofollow (do not pass link signals through outbound links), noarchive (do not show a cached version), nosnippet (do not display a snippet excerpt), and max-snippet/max-image-preview/max-video-preview (control snippet length and richness).

The two highest-leverage directives are noindex and nofollow. noindex is the correct way to keep low-value pages out of the index — internal search results pages, thin tag archives, staging environments, login-walled content, thank-you pages after form submissions. Blocking these in Robots.txt is a common mistake because it prevents the crawler from ever seeing the noindex directive, leaving the URLs indexed via their inbound links.

Google requires that any URL you want to noindex must be crawlable. The robots.txt must allow the path; only then will Google fetch the page, see the noindex meta, and drop the URL. This is one of the most common configuration mistakes in technical SEO audits.

The nofollow directive at the page level is rarely useful — it tells search engines to ignore all outbound links on the page, which is heavy-handed. Per-link nofollow attributes (rel="nofollow" on specific anchors) are usually the right tool when you need granularity.

Robots directives can also be set via the X-Robots-Tag HTTP header, which is essential for non-HTML resources — PDFs, images, and other binary files that cannot host a meta tag. The header accepts the same directive syntax.

Audit through Search Console's URL Inspection tool, which surfaces which directives Google is honouring and whether the page is currently indexed. Misconfigured noindex tags on pages you wanted to rank are a common cause of sudden traffic drops.