I hereby claim:
- I am leozilla on github.
- I am dleonhar (https://keybase.io/dleonhar) on keybase.
- I have a public key ASDtRQnpcf96DUg7wT3gWfV73k0oO3LizhDZDuWbLd79-Qo
To claim this, I am signing this object:
leozilla
/ gist:2fb36fa3ec98face4fff5d1161c50a3a
Created
October 7, 2021 16:37
Load and Export Avro from BigQuery
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 1. writeObjectContainerFile | |
| 2. loadAvroFromLocal | |
| 3. exportToGcs | |
| def writeObjectContainerFile[T <: GenericRecord](data: Iterable[T], schema: Schema): File = { | |
| import org.apache.avro.file.DataFileWriter | |
| import org.apache.avro.generic.{GenericDatumWriter, GenericRecord} | |
| val file = File.createTempFile("avro-container", "") | |
| val datumWriter = new GenericDatumWriter[GenericRecord](schema) |