Hugheym/spark_filtering_nan_and_infinity.scala

Created April 28, 2020 02:35

Star (0) You must be signed in to star a gist
Fork (0) You must be signed in to fork a gist

Select an option

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/Hugheym/0d57d942f27b2bdef000fa64ecbf78b6.js"></script>
Save Hugheym/0d57d942f27b2bdef000fa64ecbf78b6 to your computer and use it in GitHub Desktop.

Download ZIP

To filter Nan and Infinity values out of DataFrame.

Raw

spark_filtering_nan_and_infinity.scala

df.filter(!$"colName".isin(Double.NaN, Double.PositiveInfinity, Double.NegativeInfinity))

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment