Reliable Link Validation with Playwright

Playwright’s request context (playwright.request.newContext) is fast and doesn’t require page navigation for each URL. It also supports headers, cookies, timeouts, and redirect control.

Quick start: Validate links on a single page (TypeScript/JavaScript)

The following example uses Playwright Test. It:

• Extracts all links on a page
• Normalizes and filters them
• Validates each URL with HEAD, falling back to GET
• Records broken links and excessive redirect chains

// tests/link-check.spec.ts
import { test, expect, request } from '@playwright/test';

type LinkResult = {
  sourcePage: string;
  url: string;
  finalURL: string;
  status: number;
  chain: { from: string; to: string; status: number }[];
  error?: string;
};

const isHttpLike = (href: string) =>
  href.startsWith('http://') || href.startsWith('https://') || href.startsWith('/');

const isSkippableScheme = (href: string) =>
  href.startsWith('mailto:') ||
  href.startsWith('tel:') ||
  href.startsWith('javascript:') ||
  href.startsWith('data:') ||
  href.trim() === '' ||
  href.startsWith('#');

function normalizeUrl(href: string, baseURL: string) {
  try {
    // Remove fragment for HTTP checks, but keep it separately if you want to validate anchors later.
    const url = new URL(href, baseURL);
    url.hash = '';
    return url.toString();
  } catch {
    return null;
  }
}

async function followRedirectChain(api: any, url: string, maxHops = 10) {
  let current = url;
  const chain: { from: string; to: string; status: number }[] = [];

  for (let hop = 0; hop < maxHops; hop++) {
    // Use HEAD first for speed; some servers don’t support it.
    let res = await api.fetch(current, { method: 'HEAD', maxRedirects: 0 });
    let status = res.status();

    if (status === 405 || status === 501) {
      res = await api.fetch(current, { method: 'GET', maxRedirects: 0 });
      status = res.status();
    }

    if (status >= 300 && status < 400) {
      const headersObj = res.headers();
      const location = headersObj['location'] || headersObj['Location'];
      if (!location) {
        // Redirect with no location is broken
        return { finalURL: current, finalStatus: status, chain, error: 'Redirect without Location header' };
      }
      const next = new URL(location, current).toString();
      chain.push({ from: current, to: next, status });
      current = next;
      continue;
    }

    // Non-redirect status: we’re done
    return { finalURL: current, finalStatus: status, chain };
  }

  return { finalURL: current, finalStatus: 0, chain, error: 'Redirect chain too long (possible loop)' };
}

test('Validate links on homepage', async ({ page }) => {
  const baseURL = process.env.BASE_URL || 'https://example.com';

  await page.goto(baseURL, { waitUntil: 'domcontentloaded' });
  // Optionally wait for app hydration if SPA:
  // await page.waitForLoadState('networkidle');

  const hrefs = await page.$$eval('a[href]', (as) => as.map((a) => a.getAttribute('href') || ''));

  const urls = Array.from(
    new Set(
      hrefs
        .map((h) => h.trim())
        .filter((h) => !isSkippableScheme(h) && isHttpLike(h))
        .map((h) => normalizeUrl(h, baseURL))
        .filter(Boolean) as string[]
    )
  );

  const api = await request.newContext({
    ignoreHTTPSErrors: true,
    timeout: 15000,
    extraHTTPHeaders: {
      'User-Agent': 'PlaywrightLinkChecker/1.0 (+https://yourdomain.com)',
      'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    },
  });

  const results: LinkResult[] = [];
  for (const url of urls) {
    try {
      const { finalURL, finalStatus, chain, error } = await followRedirectChain(api, url, 10);
      results.push({
        sourcePage: baseURL,
        url,
        finalURL,
        status: finalStatus,
        chain,
        error: error ?? undefined,
      });
    } catch (err: any) {
      results.push({
        sourcePage: baseURL,
        url,
        finalURL: url,
        status: 0,
        chain: [],
        error: err?.message || String(err),
      });
    }
  }

  const broken = results.filter((r) => r.status === 0 || r.status >= 400 || r.error);
  const longChains = results.filter((r) => r.chain.length > 1);

  // Log a concise report
  console.log(`Checked ${results.length} links from ${baseURL}`);
  if (broken.length) {
    console.log('Broken links:');
    for (const b of broken) {
      console.log(`- ${b.url} -> status ${b.status} ${b.error ? `(${b.error})` : ''}`);
    }
  }
  if (longChains.length) {
    console.log('Links with long redirect chains:');
    for (const r of longChains) {
      console.log(`- ${r.url} -> ${r.finalURL} via ${r.chain.length} hops`);
    }
  }

  // Make the test fail if there are any real broken links
  expect.soft(broken, 'No broken links expected').toEqual([]);
});

Notes:

• We use HEAD requests by default, fallback to GET when servers don’t support HEAD.
• We stop at 10 redirect hops to avoid loops.
• We set a custom User-Agent and Accept headers to reduce server filters and false positives.

Scaling up: Crawl an entire site (same-origin) with concurrency and caching

For most teams, validating a single page is just the start. You’ll want to crawl internal pages and validate their links too. The pattern:

• Restrict scope to the same origin to avoid crawling the whole internet.
• Respect robots.txt or allowlist/denylist patterns.
• Use a queue and a visited set to avoid revisits.
• Keep a cache of external URL checks to avoid duplicate requests across pages.
• Limit concurrency to be polite and to reduce flakiness.

Here’s a minimalist site crawler plus link validator using Playwright Test. It navigates pages to collect links (so you also validate dynamic navigation paths), then uses the API request context to validate target URLs.

// tests/site-link-check.spec.ts
import { test, expect, request } from '@playwright/test';

type CrawlConfig = {
  startURL: string;
  maxPages: number;
  sameOriginOnly: boolean;
  concurrency: number;
};

type ValidationResult = {
  sourcePage: string;
  url: string;
  finalURL: string;
  status: number;
  chain: { from: string; to: string; status: number }[];
  error?: string;
};

const config: CrawlConfig = {
  startURL: process.env.BASE_URL || 'https://example.com',
  maxPages: Number(process.env.MAX_PAGES || 50),
  sameOriginOnly: true,
  concurrency: Number(process.env.CONCURRENCY || 5),
};

const shouldSkipHref = (h: string) =>
  !h ||
  h.startsWith('mailto:') ||
  h.startsWith('tel:') ||
  h.startsWith('javascript:') ||
  h.startsWith('data:');

function toAbsolute(href: string, base: string) {
  try {
    const u = new URL(href, base);
    // remove fragment
    u.hash = '';
    return u.toString();
  } catch {
    return null;
  }
}

async function followChain(api: any, url: string, maxHops = 10) {
  let current = url;
  const chain: { from: string; to: string; status: number }[] = [];

  for (let i = 0; i < maxHops; i++) {
    let res = await api.fetch(current, { method: 'HEAD', maxRedirects: 0 });
    let status = res.status();
    if (status === 405 || status === 501) {
      res = await api.fetch(current, { method: 'GET', maxRedirects: 0 });
      status = res.status();
    }
    if (status >= 300 && status < 400) {
      const headers = res.headers();
      const loc = headers['location'] || headers['Location'];
      if (!loc) return { finalURL: current, finalStatus: status, chain, error: 'Redirect without Location' };
      const next = new URL(loc, current).toString();
      chain.push({ from: current, to: next, status });
      current = next;
      continue;
    }
    return { finalURL: current, finalStatus: status, chain };
  }
  return { finalURL: current, finalStatus: 0, chain, error: 'Redirect chain too long' };
}

test('Crawl and validate site links', async ({ browser, request: testRequest }) => {
  const api = await testRequest.newContext({
    ignoreHTTPSErrors: true,
    timeout: 15000,
    extraHTTPHeaders: {
      'User-Agent': 'PlaywrightLinkChecker/1.0 (+https://yourdomain.com)',
    },
  });

  const start = new URL(config.startURL);
  const origin = start.origin;

  const visitedPages = new Set<string>();
  const pageQueue: string[] = [config.startURL];
  const linkCache = new Map<string, ValidationResult>(); // cache per target URL

  const results: ValidationResult[] = [];

  while (pageQueue.length && visitedPages.size < config.maxPages) {
    const batch = pageQueue.splice(0, config.concurrency);

    // Crawl pages concurrently
    const pageResults = await Promise.all(
      batch.map(async (pageURL) => {
        if (visitedPages.has(pageURL)) return { pageURL, links: [] as string[] };
        visitedPages.add(pageURL);

        const context = await browser.newContext({ ignoreHTTPSErrors: true });
        const page = await context.newPage();
        try {
          await page.goto(pageURL, { waitUntil: 'domcontentloaded', timeout: 30000 });
          // Optional: wait for network idle for SPAs, but be careful with long-polling
          // await page.waitForLoadState('networkidle', { timeout: 10000 });

          // Extract links and normalize
          const hrefs = await page.$$eval('a[href]', (as) => as.map((a) => a.getAttribute('href') || ''));
          const absLinks = Array.from(
            new Set(
              hrefs
                .map((h) => h.trim())
                .filter((h) => !shouldSkipHref(h))
                .map((h) => toAbsolute(h, pageURL))
                .filter(Boolean) as string[]
            )
          );

          // Enqueue internal pages for crawling
          for (const u of absLinks) {
            const uOrigin = new URL(u).origin;
            if (config.sameOriginOnly && uOrigin === origin && !visitedPages.has(u) && pageQueue.length < 5000) {
              pageQueue.push(u);
            }
          }

          return { pageURL, links: absLinks };
        } catch (e) {
          console.warn(`Failed to load ${pageURL}: ${(e as Error).message}`);
          return { pageURL, links: [] as string[] };
        } finally {
          await context.close();
        }
      })
    );

    // Validate extracted links with caching and limited concurrency
    const toValidate: { source: string; url: string }[] = [];
    for (const { pageURL, links } of pageResults) {
      for (const url of links) toValidate.push({ source: pageURL, url });
    }

    const validatorQueue = [...toValidate];
    const validating: Promise<void>[] = [];
    const maxConcurrentChecks = 15;

    const runNext = async () => {
      const item = validatorQueue.shift();
      if (!item) return;

      const { source, url } = item;
      if (!linkCache.has(url)) {
        try {
          const { finalURL, finalStatus, chain, error } = await followChain(api, url, 10);
          const entry: ValidationResult = {
            sourcePage: source,
            url,
            finalURL,
            status: finalStatus,
            chain,
            error: error ?? undefined,
          };
          linkCache.set(url, entry);
          results.push(entry);
        } catch (err: any) {
          const entry: ValidationResult = {
            sourcePage: source,
            url,
            finalURL: url,
            status: 0,
            chain: [],
            error: err?.message || String(err),
          };
          linkCache.set(url, entry);
          results.push(entry);
        }
      } else {
        // Use cached result but stamp source page for traceability
        const cached = linkCache.get(url)!;
        results.push({ ...cached, sourcePage: source });
      }

      await runNext(); // pull next item
    };

    for (let i = 0; i < Math.min(maxConcurrentChecks, validatorQueue.length); i++) {
      validating.push(runNext());
    }
    await Promise.all(validating);
  }

  // Analyze
  const broken = results.filter((r) => r.status === 0 || r.status >= 400 || r.error);
  const chains2plus = results.filter((r) => r.chain.length > 1);

  console.log(`Crawled ${visitedPages.size} pages and validated ${results.length} links`);
  console.log(`Broken: ${broken.length}, Long redirect chains: ${chains2plus.length}`);

  // Fail the test if there are broken links
  expect.soft(broken, 'Broken links found').toEqual([]);

  // Optionally write a JSON report artifact
  // require('fs').writeFileSync('link-report.json', JSON.stringify(results, null, 2));
});

Key improvements over the single-page script:

• Caching avoids double-checking the same URL across pages.
• Same-origin-only crawling prevents accidental internet-wide scans.
• Concurrency limits make it polite and CI-friendly.

Python alternative (sync API)

Prefer Python? The pattern is the same. Here’s a compact snippet to validate links from a single page, including redirect handling:

# tests/test_links.py
from playwright.sync_api import Playwright, sync_playwright, expect
from typing import Dict, List, Tuple, Optional

def is_skippable(href: str) -> bool:
    return (not href) or href.startswith(("mailto:", "tel:", "javascript:", "data:", "#"))

def normalize_url(href: str, base: str) -> Optional[str]:
    try:
        from urllib.parse import urljoin, urlparse, urlunparse
        absolute = urljoin(base, href)
        parsed = list(urlparse(absolute))
        parsed[5] = ""  # drop fragment
        return urlunparse(parsed)
    except Exception:
        return None

def follow_redirect_chain(request_context, url: str, max_hops: int = 10) -> Tuple[str, int, List[Dict], Optional[str]]:
    current = url
    chain: List[Dict] = []

    for _ in range(max_hops):
        res = request_context.fetch(current, method="HEAD", max_redirects=0)
        status = res.status
        if status in (405, 501):
            res = request_context.fetch(current, method="GET", max_redirects=0)
            status = res.status

        if 300 <= status < 400:
            location = res.headers.get("location") or res.headers.get("Location")
            if not location:
                return current, status, chain, "Redirect without Location header"
            from urllib.parse import urljoin
            nxt = urljoin(current, location)
            chain.append({"from": current, "to": nxt, "status": status})
            current = nxt
            continue

        return current, status, chain, None

    return current, 0, chain, "Redirect chain too long"

def test_validate_links_on_page(playwright: Playwright):
    base_url = "https://example.com"

    browser = playwright.chromium.launch()
    context = browser.new_context(ignore_https_errors=True)
    page = context.new_page()
    page.goto(base_url, wait_until="domcontentloaded")

    hrefs = [a.get_attribute("href") or "" for a in page.query_selector_all("a[href]")]
    urls = list(
        {u for u in (normalize_url(h, base_url) for h in hrefs if not is_skippable(h)) if u}
    )

    req = playwright.request.new_context(
        ignore_https_errors=True,
        timeout=15000,
        extra_http_headers={
            "User-Agent": "PlaywrightLinkChecker/1.0 (+https://yourdomain.com)"
        },
    )

    broken = []
    for url in urls:
        final_url, status, chain, error = follow_redirect_chain(req, url, 10)
        if error or status == 0 or status >= 400:
            broken.append((url, status, error))

    print(f"Checked {len(urls)} links on {base_url}")
    if broken:
        print("Broken links:")
        for u, s, e in broken:
            print(f"- {u} -> {s} {f'({e})' if e else ''}")

    assert not broken, f"Found {len(broken)} broken links"

Run with pytest after installing Playwright and browsers. Extend similarly to navigate multiple pages or read URLs from a sitemap.

Handling modern web realities: SPAs, anchors, iFrames, and more

Dynamic UIs and complex architectures introduce nuances. Here’s how to keep your validator accurate:

• Hydration and client-side routing (SPAs):

  • Wait for hydration when needed. Consider page.waitForLoadState('networkidle') but watch for websockets/long-polling that never go idle.
  • If menus, accordions, or infinite-loading content reveal additional links, script the interactions before extracting hrefs.
  • For router-based links that never trigger full page loads, you still validate the href targets via API, not via page navigation.

• Anchor validation (#fragment):

  • For same-page anchors, validate that the target element exists:
    • Extract fragment from the original href.
    • Query for an element with a matching id or name on the page.
  • This check catches “jump to section” failures that don’t involve HTTP.

• iFrames:

  • Iterate over frames and extract links within iframes as well: const frames = page.frames(); for (const f of frames) { await f.$$eval('a[href]', ...); }
  • Respect same-origin policies and scope; you may choose to only validate top-origin frames.

• Mixed content:

  • If your page is HTTPS, flag HTTP links to images/scripts/css or HTTP navigation targets. Browsers may block them or warn users.

• Authenticated areas:

  • Use Playwright to log in once (e.g., via form submission or auth tokens) and reuse context cookies for API requests by creating the request context from the same browser context storage state.
  • In Playwright Test, you can create a storage state file after login and load it for both page and request contexts. That way, protected URLs return 2xx instead of 401.

• Robots and rate limits:

  • For external domains, be conservative. Limit concurrency, respect timeouts, and set a descriptive User-Agent.
  • Optionally fetch and parse robots.txt to avoid disallowed paths.

Beyond pass/fail: smarter analysis of redirects and quality

Redirects aren’t inherently bad. They can be part of canonicalization, language routing, or campaign tracking. But you should track:

• Redirect chain length: Aim for zero or one hop; multiple hops slow users and leak signals.
• Permanent (301/308) vs temporary (302/307): Prefer permanent when appropriate to preserve SEO equity.
• Protocol and host changes: HTTP->HTTPS is good; unexpected domain switches may be risky.
• Query stripping or UTM rewriting: Flag if critical parameters are lost between hops.
• Redirect loops: Terminate after N hops and report loops explicitly.

Consider adding policies to your validator:

• Fail if chain length > 1
• Warn if temporary redirects persist after 30 days
• Disallow redirects to external domains except for an allowlist

Robustness techniques: reduce flakiness and false positives

• Retries with backoff:
  • Retry transient failures (timeouts, 429, some 5xx) a couple of times with exponential backoff.
• Timeouts and budgets:
  • Use modest timeouts (10–15s) and cap total runtime via max pages or max links.
• Headers and identity:
  • Send Accept and User-Agent headers. Some servers return different responses to unknown clients.
• TLS quirks:
  • ignoreHTTPSErrors: true prevents TLS failures from blocking checks on staging sites with self-signed certs.
• De-duplication:
  • Normalize URLs (remove trailing slashes consistently, unify case for hosts, strip fragments) to avoid redundant checks.

Here’s a small retry wrapper you can add to your validator:

async function withRetries<T>(fn: () => Promise<T>, retries = 2, baseDelayMs = 500): Promise<T> {
  let attempt = 0;
  let lastErr: any;
  while (attempt <= retries) {
    try {
      return await fn();
    } catch (err) {
      lastErr = err;
      const delay = baseDelayMs * Math.pow(2, attempt);
      await new Promise((r) => setTimeout(r, delay));
      attempt++;
    }
  }
  throw lastErr;
}

Wrap fetch calls with withRetries to handle transient network hiccups gracefully.

Actionable checks you can add today

• Validate external link targets but ignore specific domains prone to rate limits (add a denylist).
• Ensure target="_blank" links include rel="noopener noreferrer" to mitigate reverse tabnabbing.
• Flag mixed content (HTTP URLs on HTTPS pages).
• Verify anchor fragments exist on the source page.
• Track top N broken links and include the source pages where each was found.
• Enforce redirect chain budget: e.g., at most one hop, warn at two, fail at three.

Reporting: from console logs to CI artifacts

How you present results determines whether teams act on them. Consider:

• JSON artifact: Include fields like sourcePage, url, finalURL, status, chain, error. Store as CI artifact for triage.
• CSV summary: Useful for spreadsheets and non-technical stakeholders.
• HTML report: Group by status code, domain, or source page; link to the pages and targets for quick reproduction.
• Trends: Track broken link counts over time to measure improvements.

A simple JSON writer:

import fs from 'fs';

function writeJsonReport(file: string, data: any) {
  fs.writeFileSync(file, JSON.stringify(data, null, 2), 'utf-8');
}

In Playwright Test, you can call writeJsonReport('link-report.json', results) at the end of the test.

CI/CD integration

Add the link check to your pipeline, ideally after deployment to a preview or staging environment.

• GitHub Actions example steps:

  • Install dependencies
  • Install Playwright browsers
  • Run tests with a BASE_URL
  • Upload link-report.json as an artifact

• Failure strategy:

  • Break the build on any 4xx/5xx for internal links.
  • Allow a small number of flaky external failures with a threshold, e.g., fail if > 5 broken externals.
  • Treat long redirect chains as warnings at first; later enforce as failures.

• Scheduling:

  • Run on every PR for touched areas (e.g., docs) and nightly across the whole site.

Common pitfalls and how to avoid them

• Validated the same URL dozens of times:
  • Use a cache keyed by normalized URL.
• Endless SPA loading prevents networkidle:
  • Avoid waiting for networkidle universally; use specific selectors or timeouts.
• HEAD returns 405 causing false negatives:
  • Implement GET fallback.
• Redirects to locale selectors or consent pages:
  • Set headers or cookies to stabilize responses (e.g., Accept-Language, consent cookies).
• Auth-protected links appear broken:
  • Share auth state between the browsing context and request context.
• External sites block you (403, 429):
  • Throttle concurrency, add polite headers, and add an allowlist/denylist.

Putting it all together: a pragmatic checklist

• Scope and policy
  • Define which pages to crawl (start URLs, same-origin, max pages).
  • Decide how to treat externals, redirects, and auth-protected resources.
• Extraction
  • Wait for the right readiness state (domcontentloaded or networkidle).
  • Interact with the page if needed to reveal links.
  • Include iframe links if in scope.
• Normalization and filtering
  • Resolve relative URLs.
  • Remove fragments for HTTP checks; validate anchors separately.
  • Skip mailto, tel, javascript, data.
• Validation
  • Use HEAD with GET fallback.
  • Follow redirect chains up to a sane limit; record hops.
  • Retry transient errors with backoff.
• Performance
  • Cache results.
  • Limit concurrency.
  • Respect timeouts.
• Reporting
  • Emit JSON/CSV.
  • Group by severity.
  • Integrate into CI and set pass/fail thresholds.
• Continuous improvement
  • Reduce redirect hops over time.
  • Fix or redirect 404s systematically.
  • Monitor trends.

Final thoughts

Link validation isn’t just a hygiene task; it’s quality assurance for navigation, SEO, and trust. With Playwright, you get the speed of API-level checks, the realism of a browser, and the reliability you need for automation and CI. Start with a single-page validator, expand to a scoped crawler, and evolve your policies as your site grows.

Most importantly, make the output actionable. Developers should know exactly which links are broken, where they came from, and how severe the issue is. Do that, and your site will feel faster, cleaner, and more professional—with far fewer “404s from nowhere” to sabotage the experience.