George Gemelos ggemelos
ggemelos
/ AddThisExample
Created
June 26, 2015 14:58
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #start a screen | |
| screen -S spark | |
| # Start Spark | |
| /opt/spark-1.3.1-bin-hadoop2.4/bin/pyspark --master yarn-client --num-executors 34 --spark.yarn.executor.memoryOverhead 2000 --spark.executor.memory 4g --spark.shuffle.spill true --spark.shuffle.memoryFraction .6 --spark.storage.memoryFraction .6 --spark.driver.memory 4g | |
| ### | |
| # INITIALISE PYSPARK CONSOLE (copy and paste into console) | |
| from pyspark.sql import SQLContext | |
| sqlContext = SQLContext(sc) |
ggemelos
/ PySparkSQLExamples
Last active
August 29, 2015 14:23
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # GUIDE | |
| # https://spark.apache.org/docs/latest/sql-programming-guide.html#overview | |
| ### | |
| ## HOW TO START PYSPARK CONSOLE (copy and paste into terminal) | |
| ### | |
| /opt/spark-1.3.1-bin-hadoop2.4/bin/pyspark --master yarn-client --num-executors 34 --spark.yarn.executor.memoryOverhead 2000 --spark.executor.memory 4g --spark.shuffle.spill true --spark.shuffle.memoryFraction .6 --spark.storage.memoryFraction .6 --spark.driver.memory 4g | |
| ### | |
| ### |