shzdb/Incremental Dump.sql.md

Created June 9, 2025 10:28

Star (0) You must be signed in to star a gist
Fork (0) You must be signed in to fork a gist

Select an option

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/shzdb/380cdb73b81ea7ae0102e63259eb2883.js"></script>
Save shzdb/380cdb73b81ea7ae0102e63259eb2883 to your computer and use it in GitHub Desktop.

Download ZIP

Raw

Incremental Dump.sql.md

First go:

Query: Sort: Last updated - From Jan 2024 To Nov 2024 -> Row 200K e (~5 sec)

Store the last_export timestamp & last_output as csv

Second go:

Query : Sort: Last updated - From last_export To now -> Row 20K (~40 millisec)

Store last_export timestampt & last_output + current_output(append) output : last_output + current_output as csv

To avoid db read for last update. Intelligently use filenames.

csv_latest_{last_output_timestamp}.csv

Save previous one as : csv_archive_{output_timstamp}.csv csv_archive_{output_timstamp}.csv

Keep maybe last 10 files.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment