Dev-Dipesh/ELK with Nginx.md

ELK

Our ELK stack setup has four main components:

Logstash: The server component of Logstash that processes incoming logs
Elasticsearch: Stores all of the logs
Kibana: Web interface for searching and visualizing logs, which will be proxied through Nginx
Filebeat: Installed on client servers that will send their logs to Logstash, Filebeat serves as a log shipping agent that utilizes the lumberjack networking protocol to communicate with Logstash

These are the versions we are currently setting up in this installment, please make note accordingly if you have specific version requirements.

JDK Version - 8
Elasticsearch - 2.x
Logstash - 2.2
Kibana - 4.5

Installation Steps

For production go with three separate instances for each, elasticsearch, logstash and kibana. Currently we are setting it up on a single machine. You can opt for similar configuration on any cloud provider.

OS  - Ubuntu 14.04 LTS
RAM - 4Gb
CPU - 2

####1 - Java 8 Installation

Add Oracle Java PPA to apt:
- $ sudo add-apt-repository -y ppa:webupd8team/java
Update your apt package database:
- $ sudo apt-get update
Install the latest version of Oracle Java 8
- $ sudo apt-get -y install oracle-java8-installer

####2 - Elasticsearch Installation

Import Elasticsearch public GPG key into apt
- $ wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
Create the Elasticsearch source list
- $ echo "deb http://packages.elastic.co/elasticsearch/2.x/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-2.x.list
Update your package database
- $ sudo apt-get update
Install Elasticsearch
- $ sudo apt-get -y install elasticsearch
Start Elasticsearch service
- $ sudo service elasticsearch restart
Test
- $ curl localhost:9200
If the output is similar to this, then you will know that Elasticsearch is running properly:

{
  "status" : 200,
  "name" : "Jigsaw",
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "1.7.1",
    "build_hash" : "b88f43fc40b0bcd7f173a1f9ee2e97816de80b19",
    "build_timestamp" : "2015-07-29T09:54:16Z",
    "build_snapshot" : false,
    "lucene_version" : "4.10.4"
  },
  "tagline" : "You Know, for Search"
}

Enable Elasticsearch to start on boot $ sudo update-rc.d elasticsearch defaults 95 10

Production tip: DO NOT open any other ports, like 9200, to the world! There are many bots that search for 9200 and execute groovy scripts to overtake machines.

####3 - Logstash Installation

The Logstash package is available from the same repository as Elasticsearch, and public key is already installed, so let's create the Logstash source list:
- $ echo 'deb http://packages.elastic.co/logstash/2.2/debian stable main' | sudo tee /etc/apt/sources.list.d/logstash-2.2.x.list
Update apt package database:
- $ sudo apt-get update
Install Logstash
- $ sudo apt-get install logstash
Enable start on boot
- $ sudo update-rc.d logstash defaults 97 8
Run service logstash
- $ sudo service logstash start
We have yet to configure Logstash, but let leave it for later.

####4 - Kibana Installation

Download and install the Public Signing Key
- $ wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
Add the repository definition to your /etc/apt/sources.list.d/kibana.list file
- $ echo "deb http://packages.elastic.co/kibana/4.5/debian stable main" | sudo tee -a /etc/apt/sources.list

WARNING

Use the echo method described above to add the Kibana repository. Do not use add-apt-repository, as that command adds a deb-src entry with no corresponding source package. When the deb-src entry, is present, the commands in this procedure generate an error similar to the following:

Unable to find expected entry 'main/source/Sources' in Release file (Wrong sources.list entry or malformed file)
Delete the deb-src entry from the /etc/apt/sources.list.d/kibana.list file to clear the error.

Update apt and install Kibana
- $ sudo apt-get update && sudo apt-get install kibana
Enable start on boot
- $ sudo update-rc.d kibana defaults 95 10
Start service Kibana
- $ sudo service kibana4 start

danpem · 2017-10-05T01:22:34Z

Cool works

mvharii · 2018-03-05T10:17:13Z

Thanks Sir it works. However in above log stash section : We have yet to configure Logstash, but let leave it for later? what else to be configured and ES/Kibana/LS .. where they have stored in my local box? am unable to find the folders.. (as am new still learning could you please help me)

duypv98 · 2018-12-17T02:38:48Z

Cool sir

edsonharantes · 2019-05-07T18:06:40Z

love the half life reference :D

ismailyenigul · 2020-07-14T20:50:14Z

Hi @Dev-Dipesh
The following lines must be added to nginx conf to make basic auth work

   auth_basic “Kibana”;
   auth_basic_user_file  /etc/nginx/htpasswd.users;

Here is the complete conf

server {
  listen 80;
    server_name _;
    auth_basic “Kibana”;
   auth_basic_user_file  /etc/nginx/htpasswd.users;

  error_log   /var/log/nginx/kibana.error.log;
  access_log  /var/log/nginx/kibana.access.log;



  location / {
    rewrite ^/(.*) /$1 break;
    proxy_ignore_client_abort on;
    proxy_pass http://127.0.0.1:5601;
    proxy_set_header  X-Real-IP  $remote_addr;
    proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header  Host $http_host;
   
  }
}

petsoukos · 2021-07-27T07:07:30Z

How about putting Logstash behind a proxy? It wont work using the same logic with Kibana. Something about being a TCP protocol or something.

Dev-Dipesh/ELK with Nginx.md

Select an option

No results found

Select an option

No results found

ELK

Installation Steps

danpem commented Oct 5, 2017

Uh oh!

mvharii commented Mar 5, 2018

Uh oh!

duypv98 commented Dec 17, 2018

Uh oh!

edsonharantes commented May 7, 2019

Uh oh!

ismailyenigul commented Jul 14, 2020

Uh oh!

petsoukos commented Jul 27, 2021

Uh oh!