Skip to content

Instantly share code, notes, and snippets.

@Hugheym

Hugheym/saving_nyc_data.py

Created April 27, 2020 11:09
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save Hugheym/9708c007f7111e3061e35d0d8da64438 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save Hugheym/9708c007f7111e3061e35d0d8da64438 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
saving_nyc_data.py
%python
for colName in results_df.columns:
if not colName.endswith("datetime") and colName not in ("payment_type", "vendor_id", "store_and_fwd_flag"):
results_df[colName] = results_df[colName].astype(float)
sparkDf = spark.createDataFrame(results_df.infer_objects())
sparkDf.write.mode("overwrite").saveAsTable("ny_taxi_sample")
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment