Streaming Spring boot logs to ELK stack

In my previous blog, we have done ELK installation on windows 10 and we have even tried to push messages from input console to Elastic Search and finally viewed on Kibana Server.

I will write a separate blog on why do we need ELK?

In this blog, I’ll show you how can we push spring boot application log directly to Elastic search using Logstash which we can analyze on Kibana and If you don’t know how to install ELK on windows 10 then you can refer my previous blog and start Elastic Search and Kibana server.

Prerequisite

• Elastic Search and Kibana running on your machine
• Basic knowledge of Spring boot application

If you don’t want to start your application from scratch then you can download one spring boot application from my GitHub repository as well.

I am assuming that the Elastic Search and Kibana server are running on your machine and you have a fair idea of how to start the Logstash server and what is Logstash conf file.

So, to push spring boot logs continuously to Elastic Server, We have to open one TCP port in Logstash server and for that we have to create one Logstash config file (say elklogstash.conf) under ${LOGSTASH_HOME}/conf directory mentioning on which port TCP port should be listening under input filter and where to push the data once we received under Output filter.

For simplicity, I am skipping the filter tag as it is optional.

elklogstash.conf

Now start the Logstash server bypassing newly created conf file.

   bin\logstash -f .\config\elklogstash.conf

Cool! Now Logstash server is also up and running and if you observe the log, you will realize that it is also listening on port 4560 as mentioned in the conf file. Configure the newly created index (elkbootlogs) on Kibana as we have done during the ELK setup.

Now let's do some changes to spring boot application so that it can push all the logs to 4056 TCP port.

For this tutorial, I am using spring-logger project from my Github repository.

Add below dependency to the pom.xml file. We need Logstash encoder to encode messages.

<!-- Added for logstash Encoder-->

<dependency>

<groupId>net.logstash.logback</groupId>

<artifactId>logstash-logback-encoder</artifactId>

<version>6.2</version>

</dependency>

Open logback-spring.xml file which is under the resource folder and create new appender (say elk). The task of this appender is to push logs to the destination TCP socket and under this appender, compulsory use LogstashEncoder.

<appender name="elk" class="net.logstash.logback.appender.LogstashTcpSocketAppender">

    <destination>localhost:4560</destination>

    <!-- encoder is required -->

    <encoder class="net.logstash.logback.encoder.LogstashEncoder" />

</appender>

Add new appender to root level

<!-- LOGGING everything at INFO level -->

<root level="info">

<appender-ref ref="RollingFile" />

<appender-ref ref="Console" />

<appender-ref ref="elk" />

</root>

Save all files and start your application. So, we are done with all the setup. Its time to check whether all the changes are done properly or not.

Open Kibana on your browser (http://localhost:5601) and select your index under the Discover tab. You will see all logs are populating on Kibana as well.

Congratulations! Our configuration is working absolutely fine and it is pushing logs to Elastic Search. 

You can download the source code from here, ELK code chnages are under elkstack branch.

ELK (ElasticSearch Logstash and Kibana ) Installation on Windows 10

In this blog, I’ll show you how can we install ELK on our windows 10 machine - that is ElasticSearch, Logstash, and Kibana.

These three different products are most commonly used together for log analysis. Using ELK stack, we can achieve centralized logging which helps in identifying the problems. 

ELK is heavily used in microservices architecture where your docker images are running on 1000's of POD and you can't go to each pod to trace the logs.

Logstash:

It is the data collection pipeline tool. It is the first component of ELK Stack which collects data inputs and feeds it to the Elasticsearch. It collects various types of data from different sources, all at once and makes it available immediately for further use.

Elasticsearch:

It is a NoSQL database which is based on Lucene search engine and is built with RESTful APIs. It is a highly flexible and distributed search and analytics engine. Also, it provides simple deployment, maximum reliability, and easy management through horizontal scalability. It provides advanced queries to perform detailed analysis and stores all the data centrally for quick search of the documents.

Kibana:

Kibana is a data visualization tool. It is used for visualizing the Elasticsearch documents and helps the developers to have an immediate insight into it. Kibana dashboard provides various interactive diagrams, geospatial data, timelines, and graphs to visualize the complex queries done using Elasticsearch. Using Kibana you can create and save custom graphs according to your specific needs.

Download

You can download all the zip from here

• https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.4.2-windows-x86_64.zip
• https://artifacts.elastic.co/downloads/logstash/logstash-7.4.2.zip
• https://artifacts.elastic.co/downloads/kibana/kibana-7.4.2-windows-x86_64.zip

Unzip all the three folders to get their folder files.

Install ElasticSearch

• Open ElasticSearch folder and go to its bin folder.
• Run ElasticSearch to start the ElasticSearch server.

• Once started, go to browser and type localhost:9200

Install Kibana

• Open Kibana folder and go to its bin folder.
• Run kibana.bat file to start the Kibana server.

• Once started, go to browser and type localhost:5601

Install Logstash

• Logstash is a pipeline that pushes data to elasticSearch. So, before starting the Logstash. We have to create one config file.
• Logstash config file will be having three parts i.e. input, filter (Optional) & output.
• Create logstash.conf file under ${logstash}/conf folder. it will simply take the input from the console and push it to ElasticSearch.

• Run below command to start Logstash server and it will wait for the input to push it to elasticsearch.

bin\logstash -f .\config\logstash.conf

• Once started, go to browser and type localhost:9600

• To push the data, I have copied the log file of one project to the console.

• Once done, go to Kibana portal, Management-> Index Patterns -> Create Index pattern. You will observe that logstashdemo which we have set in logstash.conf file is already present here. Now define Kibana index by setting the same name and click on next step button.

• Add @timestamp to set Default time and click on Create Index Pattern.

• Index has been created. Now, to view the data, go to discover tab and click on message (All the logs will be pushed under message index)

Congratulation! We are done with the ELK Setup on Windows 10