How to Centralize SpringBoot logs to ELK Elasticsearch using Filebeat and Logstash

[@davidwilson4804]

Hi Without log stash also we can send the log file for springboot application

[@simplifyingtech]

@@davidwilson4804 yes we can using other technologies like Splunk etc

[@buksvdl]

Why take the easy path? No xml/yaml/json - application.properties only. Why? How many cfg files do you maintain? How many places did you define your elastic config? management.elastic.metrics.export.host etc

[@simplifyingtech]

It's springboot app no other configs required like xml yml etc,
All answers are already in video if you watch video without skipping... elastic config only need to define in logstash file that's it..

[@RahulSaini-pp5kz]

you also have tomcat in the folder but you never told about it? Will everything work well without that? Pls tell

[@simplifyingtech]

Hey this is microservice , tomcat is in built in springboot . There is no use of external tomcat here, if its web application other than springboot then its required.

[@RahulSaini-pp5kz]

@@simplifyingtech thank you it is working now, just want to ask how can we also see live logs in observability->logs->stream, as if now it is not showing

[@anon_geek]

what if we don't want to use logstash? Can we directly ship logs from the java application console without writing to a file with slf4j to elasticsearch using filebeat?
Example: Upon starting up my application I start getting logs which are printed in the IDE console. Is it possible to ship those logs to Elasticsearch directly?

[@simplifyingtech]

you need shipper to ship your logs to elasticsearch, and logstash is shipper. alternative splunk/fluentd

[@anon_geek]

@@simplifyingtech So I used your example and a different example using which I was able to directly connect filebeat with elasticsearch as per the documentation and visualize them using Kibana.
So, was concerned on how logstash comes in handy? like is it an optional use case?

[@simplifyingtech]

Good qus
To answer this we can communicate to ES from filebeat locally but if we talk on multi cluster environment and client server model we need logstash to aggregate logs from many servers and apply common transformations and filtering to your events...

[@gokulakrishnanr2675]

[ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
how to resolve this ?

[@simplifyingtech]

Check where you placed config file .. place where its mentioned in video

[@akashKVlogs-l9s]

I wanted to parse data to json format?

[@simplifyingtech]

Yes elastic output data in json format only.

[@fahadsiddiqui4359]

What to do when an error is coming as "Failed to execute action{:action=>LogStash}"

[@simplifyingtech]

There is your error, logstash could not find java . You will need to install java or set JAVA_HOME for the logstash processs
run below equivalent in windows and check
sudo -u logstash which java

[@rahulsoni-lx5rb]

🤩🤩🤩