Skip to content

Instantly share code, notes, and snippets.

@guyboertje

guyboertje/logstash-pipe-bench.txt

Last active August 29, 2015 14:25
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save guyboertje/5e3b66134b3e19298b55 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save guyboertje/5e3b66134b3e19298b55 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. guyboertje revised this gist Jul 27, 2015. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion logstash-pipe-bench.txt
    View file
    Original file line number Diff line number Diff line change
    @@ -33,5 +33,5 @@ yes '{"test": 1}' | bin/logstash -e 'input{stdin{}} filter{clone{}} output{stdo
    I see the same relative numbers for old JrJackson and new JrJackson when removing the JSON lines codec from either end.

    Question:
    What is the explaination for this?
    What is the explanation for this?

  2. guyboertje revised this gist Jul 27, 2015. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion logstash-pipe-bench.txt
    View file
    Original file line number Diff line number Diff line change
    @@ -2,7 +2,7 @@ For all below, this applies...
    ruby -v
    jruby 1.7.20 (1.9.3p551) 2015-05-04 3086e6a on Java HotSpot(TM) 64-Bit Server VM 1.8.0_45-b14 +jit [darwin-x86_64]

    1) As a basline for experimentation I substitute the JSON parser and generator for 'perfect' simulations.
    1) As a baseline for experimentation I substitute the JSON parser and generator for 'perfect' simulations.
    So for parse I simply return a new Hash and for generate I return a new String with 1 interpolation.

    yes '{"test": 1}' | bin/logstash -e 'input{stdin{codec => json_lines}} filter{clone{}} output{stdout{codec => json_lines}}' | pv -Wlart > /dev/null
  3. guyboertje created this gist Jul 27, 2015.
    37 changes: 37 additions & 0 deletions logstash-pipe-bench.txt
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,37 @@
    For all below, this applies...
    ruby -v
    jruby 1.7.20 (1.9.3p551) 2015-05-04 3086e6a on Java HotSpot(TM) 64-Bit Server VM 1.8.0_45-b14 +jit [darwin-x86_64]

    1) As a basline for experimentation I substitute the JSON parser and generator for 'perfect' simulations.
    So for parse I simply return a new Hash and for generate I return a new String with 1 interpolation.

    yes '{"test": 1}' | bin/logstash -e 'input{stdin{codec => json_lines}} filter{clone{}} output{stdout{codec => json_lines}}' | pv -Wlart > /dev/null
    01:00 [89.3k/s] [ 89k/s]

    This give me a 'high water' mark. Even with a perfect JSON parser/generator we can't go faster than this.

    2) With JrJackson at v0.2.9 (current).
    yes '{"test": 1}' | bin/logstash -e 'input{stdin{codec => json_lines}} filter{clone{}} output{stdout{codec => json_lines}}' | pv -Wlart > /dev/null
    01:00 [54.2k/s] [50.9k/s]

    3) With JrJackson at v0.3.0 (new unreleased)
    yes '{"test": 1}' | bin/logstash -e 'input{stdin{codec => json_lines}} filter{clone{}} output{stdout{codec => json_lines}}' | pv -Wlart > /dev/null
    01:00 [73.1k/s] [71.4k/s]

    BUT Here is the strangeness!

    If I remove the JSOM lines codec from either end, I see a reduced thoughput.

    For perfect JSON...
    yes '{"test": 1}' | bin/logstash -e 'input{stdin{codec => json_lines}} filter{clone{}} output{stdout{}}' | pv -Wlart > /dev/null
    01:00 [56.1k/s] [54.6k/s]

    and...
    yes '{"test": 1}' | bin/logstash -e 'input{stdin{}} filter{clone{}} output{stdout{codec => json_lines}}' | pv -Wlart > /dev/null
    01:00 [84.9k/s] [ 85k/s]

    I see the same relative numbers for old JrJackson and new JrJackson when removing the JSON lines codec from either end.

    Question:
    What is the explaination for this?