Skip to content

Instantly share code, notes, and snippets.

@thomascrepain

thomascrepain/broken-links-test

Last active August 29, 2015 14:16
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save thomascrepain/c8eefcbce291876e3f60 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save thomascrepain/c8eefcbce291876e3f60 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. thomascrepain revised this gist Mar 6, 2015. 1 changed file with 4 additions and 4 deletions.
    8 changes: 4 additions & 4 deletions broken-links-test
    View file
    Original file line number Diff line number Diff line change
    @@ -1,4 +1,4 @@
    wget -r -S http://www.test.be 2>&1 | tee /tmp/crawlTest;
    tail -f /tmp/crawlTest | grep "HTTP/1.*500";
    tail -f /tmp/crawlTest | grep "HTTP/1.*404";
    wget --spider -w 1 -r -p http://www.example.com 2>&1 | grep -B 2 -E '\.\.\. [45]\d{2}'

    Use access log on the webserver to check for the referrer on the broken links:
    tail -f access.log | grep -E 'HTTP/1.1" [45]\d{2}'
  2. thomascrepain created this gist Mar 2, 2015.
    4 changes: 4 additions & 0 deletions broken-links-test
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,4 @@
    wget -r -S http://www.test.be 2>&1 | tee /tmp/crawlTest;

    tail -f /tmp/crawlTest | grep "HTTP/1.*500";
    tail -f /tmp/crawlTest | grep "HTTP/1.*404";