Zdenko Hrček zdenulo
zdenulo
/ covid_stats.ipynb
Created
April 26, 2021 21:10
A Jupyter Notebook to download Covid stats data and save for the whole world into BigQuery table
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/bin/bash | |
| DUMP_FOLDER="/tmp" | |
| GCS_BUCKET="my-gcs-bucket" | |
| DT=$(date "+%Y%m%d") | |
| function backup() { | |
| database=$1 | |
| filename="${database}_${DT}.sql" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import logging | |
| import apache_beam as beam | |
| from apache_beam.options.pipeline_options import PipelineOptions | |
| from apache_beam.io.gcp import bigquery | |
| GCP_PROJECT = '' # set | |
| TEMP_LOCATION = '' # set | |
| STAGING_LOCATION = '' |
zdenulo
/ bq_public_datasets
Created
December 15, 2018 19:13
Goal is to find out which tables are regularly updated.
Script fetches info about public datasets / tables in BigQuery and writes into file sorted by most recent modified
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import csv | |
| from google.cloud import bigquery | |
| client = bigquery.Client(project='bigquery-public-data') | |
| datasets = client.list_datasets() | |
| data = [] | |
| for dataset in datasets: |