Skip to content

Instantly share code, notes, and snippets.

@liiight

liiight/awsaio_poc.py

Created September 22, 2020 21:46
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save liiight/09e3eb0049517bda24b33bd35f1bc40b to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save liiight/09e3eb0049517bda24b33bd35f1bc40b to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. liiight created this gist Sep 22, 2020.
    23 changes: 23 additions & 0 deletions awsaio_poc.py
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,23 @@
    def download_data(s3_keys: List[Tuple[str, int]], total_size: int):
    size_progress = Progress(
    "{task.description}",
    TextColumn("[bold blue]{task.fields[key]}", justify="right"),
    BarColumn(),
    DownloadColumn(),
    TransferSpeedColumn(),
    TimeRemainingColumn(),
    )
    keys_progress = Progress("{task.description}", BarColumn(), "{task.completed} of {task.total}")
    with size_progress, keys_progress:
    size_task = size_progress.add_task("[red]Downloading", total=total_size, key='?')
    keys_task = keys_progress.add_task("[blue]S3 Keys", total=len(s3_keys))
    for key, file_size in s3_keys:
    size_progress.update(size_task, advance=file_size, key=key)
    keys_progress.advance(keys_task)
    output = base_path.joinpath(*key.split(prefix)[-1].split('/'))
    if output.exists() and output.read_bytes():
    continue
    output.parent.mkdir(parents=True, exist_ok=True)
    output.touch()
    s3.download_file(Bucket=bucket_name, Key=key, Filename=str(output))
    console.log(f'Successfully downloaded [bold cyan]{len(s3_keys)}[/bold cyan] keys')