Skip to content

Instantly share code, notes, and snippets.

@defcon79
defcon79 / wget--crawl.sh
Created September 2, 2019 20:23 — forked from steveosoule/wget--crawl.sh
Wget - Options & Sample Crawler
#!/bin/sh
# wget --mirror --adjust-extension --page-requisites --execute robots=off --wait=30 --rand om-wait --convert-links --user-agent=Mozilla http://www.example.com
### V1
# wget \
# --recursive \
# --no-clobber \
# --page-requisites \
# --html-extension \
// ES6 flag --harmony_default_parameters needed when run in Node 5.0.0
function mergesort(list, compare = (x, y) => {return x < y} ) {
// breaking recursive call
if(list.length <= 1) return list;
// ES6 flag --harmony_destructuring needed when run in Node 5.0.0
var {leftHalf, rigthHalf } = splitList(list);
// Recursive call.