Navneet Kumar navneetk

Channels Are Not Enough

... or Why Pipelining Is Not That Easy

Golang Concurrency Patterns for brave and smart.

Intro

General Background and Overview

Probabilistic Data Structures for Web Analytics and Data Mining : A great overview of the space of probabilistic data structures and how they are used in approximation algorithm implementation.
Models and Issues in Data Stream Systems
Philippe Flajolet’s contribution to streaming algorithms : A presentation by Jérémie Lumbroso that visits some of the hostorical perspectives and how it all began with Flajolet
Approximate Frequency Counts over Data Streams by Gurmeet Singh Manku & Rajeev Motwani : One of the early papers on the subject.
[Methods for Finding Frequent Items in Data Streams](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.187.9800&rep=rep1&t

Latency numbers every programmer should know

L1 cache reference ......................... 0.5 ns
Branch mispredict ............................ 5 ns
L2 cache reference ........................... 7 ns
Mutex lock/unlock ........................... 25 ns
Main memory reference ...................... 100 ns             
Compress 1K bytes with Zippy ............. 3,000 ns  =   3 µs
Send 2K bytes over 1 Gbps network ....... 20,000 ns  =  20 µs
SSD random read ........................ 150,000 ns  = 150 µs

Read 1 MB sequentially from memory ..... 250,000 ns = 250 µs

	# /tmp/test = EBS-SSD
	# /mnt/test = instance-store

	root@ip-10-0-2-6:~# dd bs=1M count=256 if=/dev/zero of=/tmp/test
	256+0 records in
	256+0 records out
	268435456 bytes (268 MB) copied, 3.26957 s, 82.1 MB/s
	root@ip-10-0-2-6:~# dd bs=1M count=256 if=/dev/zero of=/tmp/test
	256+0 records in
	256+0 records out

	-- Two dashes start a one-line comment.

	--[[
	Adding two ['s and ]'s makes it a
	multi-line comment.
	--]]

	----------------------------------------------------
	-- 1. Variables and flow control.
	----------------------------------------------------

	// A simple sub command parser based on the flag package
	package subcommand

	import (
	"flag"
	"fmt"
	"os"
	)

	type subCommand interface {

	Latency Comparison Numbers (~2012)
	----------------------------------
	L1 cache reference 0.5 ns
	Branch mispredict 5 ns
	L2 cache reference 7 ns 14x L1 cache
	Mutex lock/unlock 25 ns
	Main memory reference 100 ns 20x L2 cache, 200x L1 cache
	Compress 1K bytes with Zippy 3,000 ns 3 us
	Send 1K bytes over 1 Gbps network 10,000 ns 10 us
	Read 4K randomly from SSD* 150,000 ns 150 us ~1GB/sec SSD

	# /etc/security/limits.conf
	* soft nofile 999999
	* hard nofile 999999
	root soft nofile 999999
	root hard nofile 999999

	===========================================================
	# /etc/sysctl.conf
	# sysctl for maximum tuning

	// $ 6g echo.go && 6l -o echo echo.6
	// $ ./echo
	//
	// ~ in another terminal ~
	//
	// $ nc localhost 3540

	package main

	import (