Skip to content

Instantly share code, notes, and snippets.

@colinsurprenant

colinsurprenant/logstash_filtersr _performance.md

Last active April 22, 2021 01:27
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save colinsurprenant/50ba340cac2367710e21 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save colinsurprenant/50ba340cac2367710e21 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. colinsurprenant revised this gist Dec 6, 2015. 1 changed file with 3 additions and 1 deletion.
    4 changes: 3 additions & 1 deletion logstash_filtersr _performance.md
    View file
    Original file line number Diff line number Diff line change
    @@ -27,4 +27,6 @@ $ while true; do cat t.t; done | bin/logstash -f myconfig.conf | pv -Wbart > /d
    10.6KiB 0:01:15 [ 631 B/s] [ 732 B/s]
    ```

    Here `[ 631 B/s]` is the current throughput in events per second and `[ 732 B/s]` is the average throughput for the test duration
    Here `[ 631 B/s]` is the current throughput in events per second and `[ 732 B/s]` is the average throughput for the test duration

    It is important to let logstash run for a while, to factor out the statup time and the JVM warmup. Typically the average throughput settles after 30 secs to 1 minute.
  2. colinsurprenant revised this gist Dec 6, 2015. 1 changed file with 4 additions and 1 deletion.
    5 changes: 4 additions & 1 deletion logstash_filtersr _performance.md
    View file
    Original file line number Diff line number Diff line change
    @@ -9,17 +9,20 @@ input {
    }
    }
    ```

    - modify your config to use the **stdout output** with the **dots codec**

    ```
    output {
    stdout {
    codec => dots
    }
    }
    ```

    - continously send your sample data to logtash and check pv numbers

    ```
    ```sh
    $ while true; do cat t.t; done | bin/logstash -f myconfig.conf | pv -Wbart > /dev/null
    10.6KiB 0:01:15 [ 631 B/s] [ 732 B/s]
    ```
  3. colinsurprenant revised this gist Dec 6, 2015. No changes.
  4. colinsurprenant revised this gist Dec 6, 2015. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion logstash_filtersr _performance.md
    View file
    Original file line number Diff line number Diff line change
    @@ -19,7 +19,7 @@ output {
    - continously send your sample data to logtash and check pv numbers
    ```sh
    ```
    $ while true; do cat t.t; done | bin/logstash -f myconfig.conf | pv -Wbart > /dev/null
    10.6KiB 0:01:15 [ 631 B/s] [ 732 B/s]
    ```
  5. colinsurprenant revised this gist Dec 6, 2015. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions logstash_filtersr _performance.md
    View file
    Original file line number Diff line number Diff line change
    @@ -18,6 +18,7 @@ output {
    }
    - continously send your sample data to logtash and check pv numbers
    ```sh
    $ while true; do cat t.t; done | bin/logstash -f myconfig.conf | pv -Wbart > /dev/null
    10.6KiB 0:01:15 [ 631 B/s] [ 732 B/s]
  6. colinsurprenant revised this gist Dec 6, 2015. 1 changed file with 5 additions and 3 deletions.
    8 changes: 5 additions & 3 deletions logstash_filtersr _performance.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,18 +1,20 @@
    ### Measuring logstash filters performance

    - given a data file sample named `data.log`
    - modify your config to use the stdin input so that you can pipe you sample log file to logstash
    - modify your config to use the **stdin input** so that you can pipe you sample log file to logstash
    ```
    input {
    stdin {
    codec => line
    }
    }
    ```
    - modify your config to use the dots output
    - modify your config to use the **stdout output** with the **dots codec**
    ```
    output {
    dots {}
    stdout {
    codec => dots
    }
    }
    - continously send your sample data to logtash and check pv numbers
  7. colinsurprenant revised this gist Dec 6, 2015. 1 changed file with 2 additions and 0 deletions.
    2 changes: 2 additions & 0 deletions logstash_filtersr _performance.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,3 +1,5 @@
    ### Measuring logstash filters performance

    - given a data file sample named `data.log`
    - modify your config to use the stdin input so that you can pipe you sample log file to logstash
    ```
  8. colinsurprenant revised this gist Dec 6, 2015. 1 changed file with 5 additions and 2 deletions.
    7 changes: 5 additions & 2 deletions logstash_filtersr _performance.md
    View file
    Original file line number Diff line number Diff line change
    @@ -15,5 +15,8 @@ output {
    - continously send your sample data to logtash and check pv numbers
    ```sh
    $ while true; do cat data.log; done | logstash -f myconfig.conf | pv -Bart
    ```
    $ while true; do cat t.t; done | bin/logstash -f myconfig.conf | pv -Wbart > /dev/null
    10.6KiB 0:01:15 [ 631 B/s] [ 732 B/s]
    ```

    Here `[ 631 B/s]` is the current throughput in events per second and `[ 732 B/s]` is the average throughput for the test duration
  9. colinsurprenant created this gist Dec 6, 2015.
    19 changes: 19 additions & 0 deletions logstash_filtersr _performance.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,19 @@
    - given a data file sample named `data.log`
    - modify your config to use the stdin input so that you can pipe you sample log file to logstash
    ```
    input {
    stdin {
    codec => line
    }
    }
    ```
    - modify your config to use the dots output
    ```
    output {
    dots {}
    }
    - continously send your sample data to logtash and check pv numbers
    ```sh
    $ while true; do cat data.log; done | logstash -f myconfig.conf | pv -Bart
    ```