Skip to content

Instantly share code, notes, and snippets.

@chuckhoupt
Last active February 20, 2023 18:48
Show Gist options
  • Select an option

  • Save chuckhoupt/6265268 to your computer and use it in GitHub Desktop.

Select an option

Save chuckhoupt/6265268 to your computer and use it in GitHub Desktop.

Revisions

  1. chuckhoupt revised this gist Aug 19, 2013. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions Read Me.md
    Original file line number Diff line number Diff line change
    @@ -24,8 +24,8 @@ truly duplicated. This is done by combining find's modification-time
    filter (-mtime) and cpio's pass-thru hard-link mode (-p and -l). The
    basic core of the script are the commands:

    find $HOME -mtime -30d -print | cpio -dpl $SHADOW_HOME
    rsync -avz --delete --omit-dir-times $SHODOW_HOME/ $BACKUP_ACCOUNT:recent-changes
    find $HOME -mtime -30d -print | cpio -dpl $SHADOW_HOME
    rsync -avz --delete --omit-dir-times $SHODOW_HOME/ $BACKUP_ACCOUNT:recent-changes

    Much of the script bulk is added filtering to eliminate temporary or
    derivative files and directories from the process.
  2. chuckhoupt revised this gist Aug 19, 2013. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions Read Me.md
    Original file line number Diff line number Diff line change
    @@ -3,8 +3,8 @@ Backup-Recent Shell Script

    Offsite backups can be very useful for disaster recovery. However,
    backing up all data to the cloud can be slow and expensive. A compromise
    is to backup only recently modified data to the cloud. Many online
    services provide small amounts of online storage, which can be useful
    is to backup only recently modified data to the cloud. Many ISPs/WebHosts
    provide small amounts of online storage, which can be useful
    for recent-file backup. For example, DreamHost accounts come with 50GB
    of free backup storage -- not enough for a complete backup, but plenty
    of space to hold many months of recent work.
  3. chuckhoupt revised this gist Aug 19, 2013. 1 changed file with 30 additions and 2 deletions.
    32 changes: 30 additions & 2 deletions Read Me.md
    Original file line number Diff line number Diff line change
    @@ -1,3 +1,31 @@
    Backup-Recent.command
    =====================
    Backup-Recent Shell Script
    ==========================

    Offsite backups can be very useful for disaster recovery. However,
    backing up all data to the cloud can be slow and expensive. A compromise
    is to backup only recently modified data to the cloud. Many online
    services provide small amounts of online storage, which can be useful
    for recent-file backup. For example, DreamHost accounts come with 50GB
    of free backup storage -- not enough for a complete backup, but plenty
    of space to hold many months of recent work.

    How it Works
    ------------

    Although rsync is very powerful, it doesn't have an option to filter
    files based on age. To work around this, the script creates a scratch
    shadow copy of the user's home directory containing only recently
    modified files.

    Of course, a naive shadow copy of recent files would consume large
    amounts of disk and time. For efficiency, the shadow copy is built with
    hard-linked copies, which means only a partial directory structure is
    truly duplicated. This is done by combining find's modification-time
    filter (-mtime) and cpio's pass-thru hard-link mode (-p and -l). The
    basic core of the script are the commands:

    find $HOME -mtime -30d -print | cpio -dpl $SHADOW_HOME
    rsync -avz --delete --omit-dir-times $SHODOW_HOME/ $BACKUP_ACCOUNT:recent-changes

    Much of the script bulk is added filtering to eliminate temporary or
    derivative files and directories from the process.
  4. chuckhoupt renamed this gist Aug 19, 2013. 1 changed file with 0 additions and 0 deletions.
    File renamed without changes.
  5. chuckhoupt renamed this gist Aug 19, 2013. 1 changed file with 0 additions and 0 deletions.
    File renamed without changes.
  6. chuckhoupt renamed this gist Aug 19, 2013. 1 changed file with 0 additions and 0 deletions.
    File renamed without changes.
  7. chuckhoupt created this gist Aug 19, 2013.
    56 changes: 56 additions & 0 deletions Backup-Recent.command
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,56 @@
    #!/bin/bash

    set -eux

    BACKUP_ACCOUNT=backupuser@backup.example.com

    MAXSIZE=45000000000

    # Wait for the net to come up
    while [ ! $(ifconfig | grep 'status: active') ]; do
    sleep 1
    done

    HOST=$(hostname -s)

    DAYS=180

    SHADOW_DIR=/tmp/$HOST-$USER-recent

    if true
    then
    rm -rf $SHADOW_DIR
    mkdir $SHADOW_DIR

    date

    find "$HOME" \
    -path "$HOME/Library" -prune -or \
    -path "$HOME/Pictures/iPhoto Library/Thumbnails" -prune -or \
    -path "$HOME/Music/iTunes" -prune -or \
    -path "$HOME/.Trash" -prune -or \
    -path "$HOME/.m2" -prune -or \
    -path "$HOME/Downloads" -prune -or \
    -name .svn -prune -or \
    -name CVS -prune -or \
    -name 'iPod Photo Cache' -prune -or \
    -not -type d \
    -not -name .DS_Store \
    -not -name '*.o' \
    -not -name '*.class' \
    -not -name '*.jar' \
    -mtime -$DAYS \
    -print \
    | cpio -pdl $SHADOW_DIR

    date

    du -sk $SHADOW_DIR

    fi

    # omit dir times, because cpio doesn't preserve dir times, causing re-sync every time
    rsync -avz --delete --stats --omit-dir-times \
    "$SHADOW_DIR"/ $BACKUP_ACCOUNT:"$HOST-$USER-recent"

    rm -rf $SHADOW_DIR
    3 changes: 3 additions & 0 deletions Read Me.md
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,3 @@
    Backup-Recent.command
    =====================