This gist is a collection of my rough notes from Strange Loop 2012.

Follow me on GitHub or Twitter to get updates as they're posted.

I'm posting these notes immediately after each talk. Expect typos, formatting glitches, incomplete thoughts, and ...

Michael Stonebraker

Abstract

--- rough notes ---

• "not your father's transaction processing"
• how does this fit into big data?
  • big volume - I have too much data
  • big velocity - data is coming at me too fast
  • big variety
• focusing on high velocity today
• in 1985, 1,000 transactions/second seemed like an amazing stretch goal
• today, you can do that on your iPhone
• TP (Transaction Processing) is now a much broader problem (New TP)
  • massively multiplayer games
  • social networking
  • real time ad placement (i.e., you have 1 millisecond to decide which ad to show me)
  • real time couponing
  • etc.
  • sensor tagging generates new TP applications
    • marathon runners
    • taxicab
    • dynamic traffic routing
    • car insurance "by the drink"
    • mobile social networking
    • ...
    • and TP volumes are ginormous!!
    • serious need for speed and scalability
  • wall street electronic trading
  • real-time fraud detection
  • micro transactions (e.g., buying a soda with your iPhone)
• In all cases
  • workload is a mix of updates and queries
  • coming at you like a firehose
  • still and ACID problem
    • don't lose my data
    • make sure it's correct
  • tends to break traditional solutions
• put differently
  • you need to ingest a firehose in real-time
  • you need to process, validate, enrich and respond in real-time (i.e., update)
  • you often need real-time analytics
• if your data doesn't fit in main memory now, then wait a couple of years and it will
  • yes, Facebook is an exception; you are not Facebook
  • main memory is going down in price faster than the size of TP data is growing
• 2007 paper "Through the OLTP Looking Glass" found that traditional databases (e.g., Oracle, DB2, etc.) spend less than 10% of their time doing "useful work." The rest is the overhead of record-level locking, latching, recovery, and buffer pool management.
• How do we go faster?
  • Main memory deployment gets rid of buffer pool (which eliminates 25% of the overhead); leaves other 75% of overhead intact
• Solution Choices
  • OldSQL - legacy RDBMS vendors
    • Code lines date from the 1980s
    • 30 years worth of "bloatware"
    • Mediocre performance on New TP
    • Slow because they spend all of their time on overhead
    • Would have to re-architect their legacy code to do better
    • They all face the "The Innovators Dilemma"
    • They'll ultimately drift off into the sunset
  • NoSQL - Give up SQL and ACID for performance
    • Give up SQL? That's throwing away 30 years of RDBMS experience.
    • "Stored procedures are good! One round trip from app to DBMS rather than one round trip per record. Move the code to the data, not the other way around." (Editor's note: Hmm. Datomic anyone?)
    • Give up ACID? Can you guarantee that you won't need it tomorrow?
    • Eventual consistency does not mean "eventual consistency"; eventual consistency means "creates garbage"
    • Eventual consistency only works if the changes are allowed to happen in any order (i.e., commutative)
    • Appropriate for:
      • non-transactional systems
      • single record transactions that are commutative
  • NewSQL
    • Preserve SQL and ACID
    • Get performance from a new architecture
    • Scale by running on a cluster of nodes
    • Automatic sharding (parallelism)
    • Focus on OLTP workload
      • a few high volume transaction signatures; implement as stored procedures
      • occasional ad-hoc transactions
    • Issue #1: Buffer pool; Solution: Run in memory
    • Issue #2: Write ahead log; Solution: replication and tandem-style failover (and fail back); You need HA anyway for New TP
    • Issue #3: Multithreading; Solution: Don't do it, or get rid of all shared data structures
    • Issue #4: Record-level locking; Solution: run to completion in timestamp order => no locking
• VoltDB is NewSQL
  • Open source
  • 70x faster than OldSQL, running on the same hardware
  • 5x faster than Cassandra on VoltDB key-value layer
  • Runs a subset of SQL
  • Scales linearly to 384 cores

Stuart Sierra

Abstract

--- rough notes ---

• The concept of design patterns have been largely overlooked in dynamic languages.
  • Norvig (1998) - 16 of the 23 patterns in the GoF book have qualitatively simpler implementation in Lip or Dylan than in C++ for at least some uses of each pattern
• "Are there any Haskell programmers in the audience? Yeah: you're gonna be pissed." (i.e., not gonna talk about monads)
• Pattern-Oriented Software Architecture
  • categories
    • architectural patterns
    • design patterns
    • idioms (specific to a single programming language)
  • we're going to focus somewhere in the middle between design patterns and idioms

• State/event pattern
  • about
    • state is derived from previous state + input
    • many divers inputs
    • need to recover past states
    • need to visualize intermediate states
  • implementation
    • update-state function takes the current state and an event and returns the new state
  • can recreate any past state by reducing over events
  • great for logging (i.e., you store every event that happened in the system)
  • offers great flexibility for how you will store state in your system
    • one extreme: never store state; you can always rebuild it from events
• Consequences pattern
  • about
    • each event can trigger multiple events
    • generated events cause state changes
    • need to visualize intermediate states
  • implementation
    • function takes the current state and an event, and returns a sequence of consequences
    • the consequences might be more events
  • consequence functions do not compose naturally; you have to update state in between

• Accumulator pattern
  • large collection of inputs; maybe larger than memory
  • small or scalar result
  • lazy sequences (i.e., map, mapcat, filter, etc.)
  • reduce is the universal accumulator
• Map-reduce pattern
  • input is linear; maybe larger than one disk
  • motivated by things that were historically true, but are becoming less so:
    • disks are (were?) slow and local to one machine
    • networks are (were?) slow
  • prediction that map-reduce will become less important as the constraints above become less and less true
  • now quite map and reduce
• Reduce/combine Pattern
  • example:
    • Clojure reducers
      • Reducers - A Library and Model for Collection Processing)
      • Anatomy of a Reducer
  • operates in parallel
• Recursive expansion pattern
  • build up result out of primitives
  • build abstractions in layers
  • recuse until no more work left to do
  • e.g., macroexpansion, Dataomic transaction functions

• Pipeline pattern

  • process with many discrete steps

  • similar "shape" of data at each step; usually a map or record

  • only one execution path

  • example:

      (defn large-process [input]
        (-> input
            subprocess-a
            subprocess-ab
            subprocess-c))
    
      (defn subprocess-a [data]
        ; ...
        )
    

  • useful to be able to easily the exact order of the steps; show it to business users to vet that you've ordered it correctly?

• Wrapper pattern
  • about
    • process with many discrete steps
    • one main execution path
    • possible branch at each step
  • Clojure's Ring library is an example
  • implementation
    • input: a function
    • output: a function that does something before and/or after the given function; might not call the given function at all
• Token pattern
  • about
    • may need to cancel an operation, but ...
    • the operation itself is not an identity
  • implementation
    • create a fn that performs the operation
    • returns a fn that allows you to cancel (i.e., undo) the operation
  • examples:
    • the scheduled thread pool in Java
    • Clojure watches
• Observer pattern
  • yes, it's a GoF pattern
  • register an observer fn with stateful container
  • examples:
    • Clojure watches
• Strategy pattern
  • yes, another GoF pattern
  • many processes with a similar structure
  • need extension points for future variations
  • examples:
    • Clojure protocols; very similar to the original OO approach
    • Clojure multimethods; dispatch on input

Gary Bernhardt

Abstract

--- rough notes ---

Gary introduces a side project that Gary has been working on part-time for a little over a year, including ...

• modal (like Vim)
• terminal only
• "much more powerful than Vim"
• not an IDE
• layers
  • overlay an orthogonal "layer" on top of the source code
  • examples:
    • diff layer
    • crash layer => overlay backtrace onto the source code
    • performance layer => overlay time profiling information onto the source code
• interactions
  • the user experience:
    • one keystroke shows you all the code that the current line interacts with
    • renders a graphical tree-like display in the terminal
    • can navigate through the graph
    • uses GraphViz
  • Useful questions you can answer
    • what code does this test hit?
    • what code does this request hit?
    • what code might hit this crash point?

• raster graphics
• 24-bit color
• italics, bold, underline
• momentary keypresses

And, we've all been punk'd. All lies. None of the things above actually exist.

• We were surprised (right?) when Gary said that he wrote his own terminal. Why? Why is it surprising that someone would write a terminal?
  1. shipping culture => our "shipping culture" is poisonous to infrastructure replacement, even when the infrastructure sucks
  2. legacy and paralysis
     • we overlook the things that have existed our entire programming careers (e.g., terminals)
     • we overlook their limitations and whether those limitations make sense in today's world
• Advocating for thinking, hammock time, prototyping
• Our "shipping culture", incremental changes don't allow for fundamental improvements (i.e., rethinkings) like the ones Gary described above

Editor's note: With Light Table, Catnip, aneditor/anterminal, and Bret Victor's upcoming talk, it seems like "Re-imagining Your Development Environment" will be one of the themes of Strange Loop 2012

Michael Feathers

Abstract

--- rough notes ---

• Revised talk title? "Using Haskell Type Signatures as a Functional Design Notation"

• Haskell type signatures useful for thinking about design

• Even in Ruby code, Michael finds himself adding Haskell-esque type signatures as comments (i.e., documentation for methods)

     String -> [String] -> [[String]] -> [String] -> String
  

• Look at the shape of your data and how each function transforms the data; doing so will influence how you want to name your functions

  • Example 1
  • Example 2

• Focuses on data

• Think of problems as transformations from one thing to another thing

Robert Pitts

Abstract

--- rough notes ---

• Ruby gives us many of the tools we need to write code in a more functional style
  • Enumerable includes most (all?) of your favorite HOFs (e.g., #map, #reduce, #filter, etc.)
  • First(ish) class functions
    • blocks, procs, lambdas, methods
    • Currying and partial application (via Proc.curry added in 1.9)
  • Continuations (via callcc)
  • Tail call optimization (as of Ruby 1.9)
    • not enabled by default
    • enabled by passing args when launching Ruby
  • Facets (added in 1.9)
• hamster - persistent, efficient, immutable, lazy data structures in Ruby
• stunted - not yet ready for production use
• celluloid - actor-based concurrent object framework for Ruby
• monads (e..g, ick)

Editor's note: If this topic interests you, you might also enjoy Alan Dipert's "Functional Programming in Ruby" slides from Ruby Hoedown 2011.

David Nolen

Abstract

--- rough notes ---

• Reiterating one of the key motivations for ClojureScript: JavaScript has reach.
• Arms race among Google, Apple, Mozilla to have the fastest JS runtime. As a result, we (developers) win.
• Argues that no browser (other than Chrome) will ever adopt Dart.
• "Lisp programmers know the value of everything and the cost of nothing." — Alan Perlis
  • And that's unfortunate, because efficiency is important (because performance is important)
• ClojureScript is expression oriented (i.e., not a mix of statements and expressions)
• ClojureScript's advantages over vanilla JavaScript
  • persistent data structures
  • functions are values
  • protocols
  • namespacing
  • browser-connected REPL
  • macros
• Translations from JavaScript to ClojureScript
• Admittedly, adopting ClojureScript does have some costs
  • debugging [A]
  • a lot of code generated for trivial programs (e.g., "Hello World")
  • some components of Clojure not yet present in ClojureScript [B]
  • not bootstrapped; requires Clojure
  • uses JavaScript arithmetic underneath (which is inferior); does not use Clojure numerics
  • performance enhancements are not evenly distributed (e.g., keywords, multimethods)
• Live demo of spectralnorm benchmark
  • Java: 9 seconds
  • JavaScript (on V8): 12 seconds
  • Dart-optimized JavaScript (on V8): 13 seconds
  • ClojureScript-generated JavaScript (on V8): 13 seconds
• Live demo of using core.logic in ClojureScript
• Q & A
  • Is ClojureScript in use at the New York Times? No.
  • Source maps are coming along. Stay tuned.
  • What helps your dev workflow? lein-cljsbuild

[A] Editor's note: Debugging is probably better than you expect. It's better than I expected.

[B] Editor's note: Check out the "Differences from Clojure" wiki page for a good breakdown.

Amanda Laucher and Paul Snively

Abstract

--- rough notes ---

• Logic programming is in the air
  • Datalog
  • MiniKanren
  • core.logic
• Assumptions for this talk
  • you've used a REPL
  • logic programming is upon us; see above
  • some languages are better suited for correct code; admittedly a potentially controversial assumption
  • we spend way more time reading code than writing it
  • we are lazy bastards!
• Quotes
  • "When it doubt, create a type." — Martin Fowler
  • "Make illegal states unrepresentable." — Yaron Minsky
  • Michael Feathers defines legacy code as "code without automated tests"
  • "In 5 years, we'll view compilation as the weakest form of unit testing." — Stuart Halloway
  • "Given a good test suite, the return on investment simply does not justify the use of static typing." — Jay Fields
• Stereotypes
  • It's easier to refactor with tests that types
  • No tests = no trust
  • Tests take a long time to run and types to compile
  • Property-based testing can replace unit testing
  • Modular design only occurs with TDD
  • I don't get errors than can be prevented by types
  • Ivory tower vs. hippies
  • 100% test coverage or bust [A]
  • Typed code is verbose
  • Types take too long
  • Testing is for QA
• They individually worked through code katas. In an effort to see the other side of the "tests vs types" debate, they each approached the kata in ways that were different than their usual approach.
• Amanda coming from an F# background (strong emphasis on testing than types); Paul coming from a Scala background (stronger emphasis on types)
• Amanda's approach:
  • F#
  • Signatures first
  • Types first (as opposed to tests first)
  • All algorithmic development via REPL experimentation
  • Tests for validation
• Amanda's lessons learned:
  • types saved me from even having to think about certain categories of tests
  • tests help me out when I get stuck, but I mostly run them in the REPL and delete
  • structured her code differently to make it more friendly to work with in the REPL
  • most modern languages don't have a strong enough type system to make illegal state unrepresentable
  • it's easy to get lost in a space where you never deliver
  • tests verify when types can't prove
• Paul's approach:
  • Scala
  • Tests without types (as opposed to preferring types first)
  • Property-based testing
  • Types
  • Delete some tests
• Paul's lessons learned
  • even for such a small problem, spelling out unit tests made me want to gouge out my eyeballs with a rusty spoon; 32 unit tests for just 4 small user stories
  • when developing property-based tests, every forAll made me think, "could/should that be a type?"
  • for some use cases, having examples of correct input/output gave no real guidance whatsoever
  • test code is still code, with its own correctness, maintenance, etc. burden
• Amanda's and Paul's discussion (i.e., things they seemed to agree upon) as they worked through these katas:
  • small codebase = little value for the type system
  • types scale better than tests (i.e., scaling as the code base grows)
  • types have little value when talking with non-technical users
  • the hardest part is understanding the requirements
  • we are rarely handed sample input/output as part of the requirements (which makes it harder to write tests) [B]
  • tests can be good for forming ideas, but then can be deleted
• Paul's and Amanda's kata solutions will be up on GitHub soon

[A] Editor's note: Testing Anti-Patterns — How to Fail With 100% Test Coverage

[B] Editors note: This might be a problem if you're coding in a cave. But you're not. So, go talk to your customer. Just sayin'.