Spring Boot With Splunk Integration | Realtime logs analysis using Splunk | HEC | JavaTechie
ฝัง
- เผยแพร่เมื่อ 8 ก.ย. 2022
- In this tutorial I will walk you through step by step how to utilise Splunk's Http Event Collector (HEC) feature to index customised log events via a Spring boot application and a log4j2 .
Splunk provides the common Universal Forwarder utility to push log automatically from servers to index but this technique will make your application capable of logging into Splunk directly without having a properly configured server infrastructure in place
Spring boot microservice Live course Just started (Recordings available)
Hurry-up & Register today itself!
COURSE LINK : javatechie5246.ongraphy.com/
PROMO CODE : Java40
Splunk download link :
www.splunk.com/en_us/download
GitHub:
github.com/Java-Techie-jt/spl...
Blogs:
/ javatechie
Facebook:
/ javatechie
guys if you like this video please do subscribe now and press the bell icon to not miss any update from Java Techie
Disclaimer/Policy:
--------------------------------
Note : All uploaded content in this channel is mine and its not copied from any community ,
you are free to use source code from above mentioned GitHub account - วิทยาศาสตร์และเทคโนโลยี
Never seen a splunk integration to application in any channel. We use it in our application, but donot know the backend configuration. Thank you so much for your efforts and keep going.
Excellent Sir ❤
As always most useful video !! To the point, clearcut explanation !!!
As usual crystal clear explanation on this topic! Would love to see more detailed video/ crash course on splunk for java developers!thank you
Ohh my God, I am waiting since long time spring boot with splunk integration, Thanks Basant for the great concept
A really very detailed walkthrough, very helping and prominent 👋👍
Thanks for providing an excellent session
Yes bro more video we want. bcoz these is very easy for microservices also
Thank you so much for providing valuable session. I really appreciate you for giving valuable time.
Thanks Basant for giving such nice explanation.
Another superb video. The topic covered is useful irl
Thanks Basant bhai for this wonderful session👏
This was really helpful. Thanks!
you are a best Teacher Bro 👨🏫, keep up the good work
Eagerly whaitin for this.
Thank you for the splunk explanation
Excellent tutorial👏👏
Nice intro. Keep up the good work.
For anyone running on docker. You need to expose 2 ports. 8000 is for the UI & 8088 which is the HTTP event collector.
I have same doubt
For docker we changed aport 8080:5000(or any other ports we want sooo) it will not contradict..docker and splunk we will integrate it together
Great Video!. Please explain about distrubuted tracing using Splunk in your future video.
Awesome.. please create more content on splunk..
Thank you so much Sir!!
Excellent bro
Would like to see more videos on splunk search. Keep up the good work Basant
Will do
excellent
Great resources
Nice video
Hi basanth, please make a tutorial on how to implement logger for multithreaded application in java, and CONGRACTS FOR 100K SOON🥳🥳🥳
Good Video.. Please make one video on splunk with ECS AWS
Thank you
Thanks Sir
hi Basant ,thanks for the video . Could you please make a video on how metrics and tracing can be achieved using splunk in microservices
Perfect bhai .. in upcoming videos if you could please please add some tutorials about the ELK stack, that will be very very helpful. Projects are largely moving to ELK and Graphana with Microservices observability.
Hi Ashish , please take a look at the tutorial 👇 . Already i covered ELK
th-cam.com/video/5s9pR9UUtAU/w-d-xo.html
nice !
nice
Bro...thanks awesome . Have a query difference b/w Splunk & ELK(filebeat) which one is good and what is the scenario will these. Any Pro's & Cons . Kindly share the same .
Thanks for providing an excellent session. I have been following many TH-cam channels to learn the concepts. I feel you are able to explain very simply and it's really useful for me.
In my project we were creating a user provided service (splunk) in PCF and binding that service with the application.
Any idea how the index was created internally in PCF?
Can you help me out, I am unable to see the logs on the splunk, although those logs appear on my terminal
❤❤❤
thanks a lot to upload the video , I need one video on how to handle exception in production environment using logs.
Irrespective of any environment you will find logs directly in splunk dashboard just you need to filter and analyse further buddy
Thanks for update me.
What is the difference between Spunk, ELK, Appdynamics and Promethius? Can you please make a video on this topic? Please
Hi bro, believe logging not the only facility provided by splunk. Can you please provide how it helps to analyze systems behavior by analyzing logs. Also clear by Environment (is there option beside index ?). Also reporting using gui possible?
Thank you so much Sir 🙏for the Splunk explanation this video is very useful great job....just one request please make video on end to end full stack application with angular Springboot....
Already i uploaded a spring boot crud application with angular please check
Refer this th-cam.com/video/QV7ke4a7Lvc/w-d-xo.html
Recently I encounter the question how we can search logs using 2 indexes same time for single search. Could please make a video in this use case for the Transaction
Such an amazing explanation. But sir I'm facing too many problems while implementing. Errors are regarding jakarta, splunk dependancy added there and many more things which I think happened because of spring boot 3.0. Can you please tell how to resolve all those errors?
Hi everyone,
Can i use splunk along with sleuth and zipkin in springboot project for logging as splunk and zipkin for distributed tracing.
Excellent session.. Starting from setup to adding and searching logs.... Can you please add more videos on splunk for more searching tips... Setting alerts... And preparing dashboard in splunk(like no of request on a Api etc)
Yes i will do that 🙂
@@Javatechie Yes, that we be helpful as in most of the real time applications, alerts will trigger for specific conditions
@@Javatechie can you please make a video on what are the tools or frameworks used in the real time. Like now you expained for splunk which is used for logging in realtime. For service discovery in your sessions you have explained it using Eureka but in the production level they might be using different tool or framework to achieve this serice discovery right. simillarly loadbalncing,resiliance,API Gateway and declarative client they might using some realtime tools.can you please please make a vidoe on this.
@@Javatechie thank you so much
Azure App Insight Log Analytics vs Splunk ? Which is preferred and efficient plus simple way bhai?
what level type logs are configured while during the production? please clarify my query?
Can u make video how we send pipeline logs on AWS Cloudwatch
Is it possible to show spring boot application db connection pool Matrix in the splunk dashboard ?
Hi, Please can you cover ELK stack with spring-boot microservice application in details including logging agent, It will be very vert helpful.
Thanks in Advance
Why the controller and service class logs are combined under single entry. In our app we have the different entry for each log entry we do in code. Which setting is responsible for that?
More videos on splunk sir... In my current project it is getting used and I am new to this tool
Thank you so much for this video
Hi sir
Your videos are truly next level. And helped me alot. I too want to create content like this for python and aws. Can you please tell me what all tools do you use to create videos and how do you create the display page please?
Please connect with me on javatechie4u@gmail.com
How to configure log2.spring.xml file for different environment. As of now we are configuring for dev index what about other environments like qa, prod etc..
Logs are printed correctly in the console without any Error, but not getting pushed to the Splunk server.
Please help
splunk enterprise system requriements please tell us that also
How can we make each log in new line for readability instead of having whole flow logs in one record in splunk? Can any one help me with this.
Hi Sir, If we want Splunk Logs back in ourJava Application.
Is it possible?
i did the same, evrything is fine , but in dashboard it showing ( No results found. Try expanding the time range.) , i dont know why, anyone help me
How do we remove the initial unnecessary lines e.g. {“host”:”localhost………. I want to see only app logs
The other texts makes hard to read.
bro i am not getting logs in Splunk
i fallowed the same step but not getting
Please validate your index
Nice explanation! But this demonstration is for monolithic architecture. Could you please also explain the log aggregation for microservices architecture using splunk?
Sure i will give it a try
@@Javatechie Thankyou!
Hi sir , Any specific advantages does it can provide as compared to kibana ...please suggest ... I am looking for a scope more of a distributed request tracing .
Do we need to name the logger file in the same format as specified in the video?
And we we have multiple logger files (for dev and other for QA and the logging level of each environment is different ?)
No you can customise it as per your needs. It's just a demo
@@Javatechie can we have multiple files also? With different indexes?
Yes we can have
The logs are not getting reflected in splunk . Please help
Please copy paste the logback.xml correctly
I have copy pasted it correctly but still it is not reflected in splunk
I have been working on POC of it , how did you get to know the respository URL
Which repository url ?
Need video on how to use new elasticsearch java client
Already uploaded please search with keyword elastic search in my channel
CAN YOU CREATE video for generating alert?
Yes working on it
Hi Basant How to post sleuth traceId to splunk?
I have not done any similar integration will check and update you
Can someone help, I am unable to see any logs in the splunk enterprise after following the tutorial
Facing same problem. Were you able to find a solution?
Hi, I tried to follow your tutorial but when I searched the index in splunk is not displaying anything :(
Did you configured index , source and source type correctly in log4j.xml . Make sure to define splunk http port 8088
stuck on the same problem. were you able to solve it?
Yes...same problem...no logs displaying.
feels like it's not needed to put the classname into the log message as the logging framework already knows that and passes it for you.
How to generate separate trace I'd for logs
Traceid or index ?
TraceId
Can I format logs for better readability ?
Yes you can do that just specify your log pattern in log4j2.xml which i created in resources folder
@@Javatechie okay thanks for your support.
Could you please zoom while showing code or configuration it is difficult to see
Sure buddy going forward i will take care this
@@Javatechie Thank you
I do not see logs on Splunk, but I see no error on Spring application log as well
Copy paste my log4j.xml file
@@Javatechie thanks for your reply. I figured out the issue. Basically, the splunk logger library does not work with Splunk 9 which I was trying with, falling back to Splunk 8 solved the issue
Cool , can you share the dependency on what you used
@@Javatechie I used your code. Did not changed anything. I just downloaded the Splunk version to 8.2. FYI, I used the docker for splunk
@@nilanjansarkar100 got it
React +spring webflux + gcp/aws + monitor grafana trending combination. It would be more helpful if we get video with mini project
seems like elasticsearch
Non comparable buddy both are different context
i get this error Sir when i do search the log with index="order_api_dev", "Configuration initialization for C:\Program Files\Splunk\etc took longer than expected (4343ms) when dispatching a search with search ID 1663750394.31. This usually indicates problems with underlying storage performance."
Shutdown and restart your splunk server again please
Great Video I am getting the below error while running the above code ,Please advice me to resolve the below
2022-12-21 01:06:46,480 main ERROR Unable to invoke factory method in class com.splunk.logging.HttpEventCollectorLog4jAppender for element SplunkHttp: java.lang.NoClassDefFoundError: okhttp3/Callback java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:138)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1122)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:1047)
a