Skip to content

Instantly share code, notes, and snippets.

@shzdb

shzdb/Incremental Dump.sql.md

Created June 9, 2025 10:28
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save shzdb/380cdb73b81ea7ae0102e63259eb2883 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save shzdb/380cdb73b81ea7ae0102e63259eb2883 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. shzdb created this gist Jun 9, 2025.
    20 changes: 20 additions & 0 deletions Incremental Dump.sql.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,20 @@
    # First go:

    Query: `Sort: Last updated - From Jan 2024 To Nov 2024` -> Row 200K e (~5 sec)
    > Store the last_export timestamp & last_output as csv
    # Second go:
    Query : `Sort: Last updated - From last_export To now` -> Row 20K (~40 millisec)
    > Store last_export timestampt & last_output + current_output(append)
    > output : last_output + current_output as csv
    ---
    To avoid db read for last update. Intelligently use filenames.

    `csv_latest_{last_output_timestamp}.csv`

    Save previous one as :
    `csv_archive_{output_timstamp}.csv`
    `csv_archive_{output_timstamp}.csv`

    Keep maybe last 10 files.