OG mslitao

😀

Welcome! I'm a machine learning scientist,and a full-stack data engineer.

MACHINE LEARNING SCIENTIST & DATA ENGINEER

mslitao / StreamingHLL.scala

Last active August 29, 2015 14:19 — forked from MLnick/StreamingHLL.scala

	import spark.streaming.StreamingContext._
	import spark.streaming.{Seconds, StreamingContext}
	import spark.SparkContext._
	import spark.storage.StorageLevel
	import spark.streaming.examples.twitter.TwitterInputDStream
	import com.twitter.algebird.HyperLogLog._
	import com.twitter.algebird._

	/**
	* Example of using HyperLogLog monoid from Twitter's Algebird together with Spark Streaming's

maxtomassi / StopSparkStreaming.scala

Last active August 29, 2015 14:08

Gracefully stopping a Spark Streaming application

	def main(args: Array[String]) {

	// Prepare your environment

	val ssc = new StreamingContext(conf, Seconds(batchDurationInSec))

	// Do your processing

	sys.ShutdownHookThread {
	log.info("Gracefully stopping Spark Streaming Application")

MLnick / StreamingCMS.scala

Created February 13, 2013 15:00

Spark Streaming with CountMinSketch from Twitter Algebird

	import spark.streaming.{Seconds, StreamingContext}
	import spark.storage.StorageLevel
	import spark.streaming.examples.twitter.TwitterInputDStream
	import com.twitter.algebird._
	import spark.streaming.StreamingContext._
	import spark.SparkContext._

	/**
	* Example of using CountMinSketch monoid from Twitter's Algebird together with Spark Streaming's
	* TwitterInputDStream

MLnick / StreamingHLL.scala

Last active January 24, 2024 19:39

Spark Streaming meets Algebird's HyperLogLog Monoid

	import spark.streaming.StreamingContext._
	import spark.streaming.{Seconds, StreamingContext}
	import spark.SparkContext._
	import spark.storage.StorageLevel
	import spark.streaming.examples.twitter.TwitterInputDStream
	import com.twitter.algebird.HyperLogLog._
	import com.twitter.algebird._

	/**
	* Example of using HyperLogLog monoid from Twitter's Algebird together with Spark Streaming's