CODE EXAMPLE FOR SHELL

wget --spidedr

wget --spider --debug -e robots=off -r -p http://jcode.me 2>&1 
        | egrep -A 1 '(^HEAD|^Referer:|^Remote file does not)' > ~/wget.log