Skip to content

Instantly share code, notes, and snippets.

@qqpann

qqpann/bq_hash.py

Last active February 6, 2021 10:22
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save qqpann/e3199f9420767c72e45be3c02922d1dd to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save qqpann/e3199f9420767c72e45be3c02922d1dd to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. qqpann revised this gist Feb 6, 2021. 1 changed file with 5 additions and 0 deletions.
    5 changes: 5 additions & 0 deletions bq_hash.py
    View file
    Original file line number Diff line number Diff line change
    @@ -1,3 +1,8 @@
    from pathlib import Path
    from hashlib import sha256

    datadir = Path('data')

    def get_df_from_bq(sql: str, **opt):
    hashed = sha256(sql.encode("utf-8")).hexdigest()
    cache = datadir / "cache"
  2. qqpann created this gist Feb 6, 2021.
    12 changes: 12 additions & 0 deletions bq_hash.py
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,12 @@
    def get_df_from_bq(sql: str, **opt):
    hashed = sha256(sql.encode("utf-8")).hexdigest()
    cache = datadir / "cache"
    cache.mkdir(exist_ok=True)
    fname = cache / f"{hashed}.csv"
    if fname.exists():
    return pd.read_csv(fname, **opt)
    else:
    client = bigquery.Client()
    df = client.query(sql).to_dataframe()
    df.to_csv(fname, index=False)
    return df