Skip to content

Instantly share code, notes, and snippets.

@QuadTriangle

QuadTriangle/csv_to_arrow.py

Created December 4, 2018 07:14
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save QuadTriangle/2c4b346898071e28e8912ca1afa583d5 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save QuadTriangle/2c4b346898071e28e8912ca1afa583d5 to your computer and use it in GitHub Desktop.
Download ZIP
csv to apache arrow file converter
Raw
csv_to_arrow.py
import pandas as pd
import pyarrow as pa
df = pd.read_csv("data.csv")
table = pa.Table.from_pandas(df)
with open('out.arrow', 'bw') as f:
writer = pa.RecordBatchFileWriter(f, table.schema)
writer.write(table)
writer.close()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment