Logstash Job with Real-time visualization with Kibana (Lecture 16)

[@vaibhavambasta8053]

windows users if logstash is not working, set sincedb_path='nul' ; if config file is unable to be read, place it in logstash/bin and use "logstash -f logstash_cars.conf" in cmd, in the bin directory

[@fellifel147]

Made the right choice to watch these videos on ELK, weldone man. Great video, simplified and clear explanation. Thanks a million

[@priyankamalviya3927]

still valid 2 years later! I just want to add - sincedb_path => "/dev/null" was causing me error and I removed it to get rid of the error

[@kevin-ru6oe]

A mí me funcionó con eso

[@ander.carrera]

where you in linux? that path is only suitable for linux

[@kevin-ru6oe]

@@ander.carrera I think it is. I was in linux too

[@arslanebahlel3990]

Can you make a Playlist with ELK stack lessons ? because i cant find all your vids..

[@iktiar]

Yes, i had the same issue, please make playlist

[@K0818069199]

Beautifully tutorial .. thanks Imtiaz

[@jerrycalvanese8413]

Very through explanation with clear step-by-step directions. Thank you!

[@yihao7496]

Thank you so much! Your videoes about ELK are exactly what I need!

[@SushilKumarBhaskar]

Excellent Explained EE+ 1000000+ INCREMENTAL LIKES

[@konstantinkaiser2383]

How do you display that the terminal is "working/ calculating " busy"? Mine just give no indication if stuck or still working on smth.

[@michaelbroyles1559]

Once the data is indexed into Kibana, is it persistent after Logstash is closed? Through reboots?

[@aditya23041988]

Hello Imtiaz
I tried creating the config file to load the csv and ran the logstash in cmd. I encountered with the following error:
Please help!!
C:\Users\adi230688\Desktop\ELK\logstash-6.4.0\bin>logstash -f C:\Users\adi230688
\Desktop\ELK\logstash-6.4.0\bin\logstash-chk.conf
Sending Logstash logs to C:/Users/adi230688/Desktop/ELK/logstash-6.4.0/logs whic
h is now configured via log4j2.properties
[2018-08-28T00:13:36,942][WARN ][logstash.config.source.multilocal] Ignoring the
'pipelines.yml' file because modules or command line options are specified
[2018-08-28T00:13:39,035][INFO ][logstash.runner ] Starting Logstash {"
logstash.version"=>"6.4.0"}
[2018-08-28T00:13:45,431][ERROR][logstash.inputs.file ] Unknown setting 'sta
rt' for file
[2018-08-28T00:13:45,459][ERROR][logstash.agent ] Failed to execute ac
tion {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"L
ogStash::ConfigurationError", :message=>"Something is wrong with your configurat
ion.", :backtrace=>["C:/Users/adi230688/Desktop/ELK/logstash-6.4.0/logstash-core
/lib/logstash/config/mixin.rb:86:in `config_init'", "C:/Users/adi230688/Desktop/
ELK/logstash-6.4.0/logstash-core/lib/logstash/inputs/base.rb:60:in `initialize'"
, "org/logstash/plugins/PluginFactoryExt.java:233:in `plugin'", "org/logstash/pl
ugins/PluginFactoryExt.java:166:in `plugin'", "C:/Users/adi230688/Desktop/ELK/lo
gstash-6.4.0/logstash-core/lib/logstash/pipeline.rb:71:in `plugin'", "(eval):8:i
n `'", "org/jruby/RubyKernel.java:994:in `eval'", "C:/Users/adi230688/Desk
top/ELK/logstash-6.4.0/logstash-core/lib/logstash/pipeline.rb:49:in `initialize'
", "C:/Users/adi230688/Desktop/ELK/logstash-6.4.0/logstash-core/lib/logstash/pip
eline.rb:90:in `initialize'", "C:/Users/adi230688/Desktop/ELK/logstash-6.4.0/log
stash-core/lib/logstash/pipeline_action/create.rb:38:in `execute'", "C:/Users/ad
i230688/Desktop/ELK/logstash-6.4.0/logstash-core/lib/logstash/agent.rb:309:in `b
lock in converge_state'"]}
[2018-08-28T00:13:46,582][INFO ][logstash.agent ] Successfully started
Logstash API endpoint {:port=>9600}
C:\Users\adi230688\Desktop\ELK\logstash-6.4.0\bin>

[@ramstein74]

Fantastic. The best i´ve found so far.

[@kevin-ru6oe]

Why there are model.keyword and model? Is there a way to delete one of them?

[@chiranjitsaha5857]

I'm using xpack security in Elastic search and my java logs are not getting configured in kibana through logstash. Can you please let me know the issue also if there any config file changes required then please share with me.

[@georgegatuma7234]

Precise and very informative, niice

[@annamo7311]

Great video! i am confused about how to upload several related csv files use logstash but just use an index， have u ever use the graph function of the kifana?

[@parimalkumarRetailBI-Architect]

How do you load nested json to elaticsearch using logstash ? and also nested data from rdbms? can you please share a example conf

[@emiribrahimbegovic813]

How much resources does kibana use? Also how impacted is your production

[@sayalichincholikar3272]

Hi Imtiaz, While running the logstash command I am getting an error like : java.IllegalStateException. Can you please help me to solve it. It is related to jdk version? I am using jdk 1.8.0_215

[@veereshgs5298]

Thanks, make more about elastic search videos 👍

[@Mehrotradheeraj80]

hi i am getting below error and all data is not indexing
org.elasticsearch.ElasticsearchParseException: failed to parse date field [-764144999999] with format [epoch_millis]: [failed to parse date field [-764144999999] with format [epoch_millis]]

[@weedimples3027]

I was able to get all the data into elastic search and mimic all your steps, but I'm having trouble with elasticsearch not recognizing the date columns. When trying to setup a date range or historgram in a visualization, it doesn't recognize the date columns as date, but instead string as you stated in the beginning.

[@NiteshSinghDE]

very nice tutorial, everything is explained so clearly

[@shezanbaig2733]

Hi Imtiaz,
While running the logstash command with the config file, I am getting the following error:
warning: ignoring JAVA_OPTS= -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -Djruby.regexp.interruptible=true -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -Dlog4j2.isThreadContextMapInheritable=true ; pass JVM parameters via LS_JAVA_OPTS
Any idea how to tackle this.

[@roughnek201]

Where is tutorial #1 of this series?

[@baharudinyusuf6254]

whether to make the visualization automatically using the script. so when having a lot of data do not need to make visualization one by one. if you can how the example script to create the visualization. thank you

[@rajeshj934]

Can Kibana Display Images?

[@AmruAshik]

@imtiaz : Excellant explannation, please do provide the links to previous lectures, couldnt find it.
And pplease do a video on kibana visualization for maps

[@samruttipatil6763]

"pipeline has started successfully " this msg is printed in the command prompt but it doesnt shows the data which is loaded in the csv file....what must be issue in that ......help me with it

[@pranjalgohain8628]

I am stuck with the same issue

[@gurmanbirsingh1209]

if once you were able to load data---try to make modifications in txt file cuz logstash wont load same data in elasticsearch again.

same here

[@amaramarg5893]

could youu tell me this ELK comes under devops or data science

[@karthikksk3319]

my logstash config file is not loading the data

[@wandellamaia6516]

Is it normal to take a while to run after logstash was started sucessfully?

[@anantababa]

how to read more than 3 logs file from same server and create different index for the diff file ?I have tried with type but it is not working with more than 2 files .

[@Nopehandler]

@Imtiaz Ahmad I have a question. All this is happening on your local machine. How do I do this on an AWS ES domain?

[@mikef4040]

Anyone else having issues with loading the data?
Getting an error like pipelines.yml cannot be loaded.
Help me please,
Thanks alot

yes same here

[@LOOKY81]

Great video. It was very useful for me. Thank you

[@anandm801]

Great job Imtiaz

[@ujjawalkumar7047]

I'm getting an error as Error: No such file or directory - C:\dev
ull.
How can I get ride-off it?

[@ujjawalkumar7047]

Solved

[@Grandfathermagic]

the lecture is very old and the logstash -f won't work your configuration

[@javeruribe]

Lo amo man gracias llevaba mucho tiempo tratando de subir mi archivo y no sabia bien la ruta

[@5kyX0]

Why is it that after I successfully create my pipeline, my data is still not in kibana though my index is?

[@kshitijyelpale1713]

Hello imtiaz,
Nice tutorial for beginner level. I have a query, the cars data is not going to elastic search. I have did same configuration as per windows. In the log file, pipeline started and logstash is also properly running, but data is not going
Can you please help or anyone ?
Bye, thanks and regards

[@edwinpoulose8851]

same problem here, have you found any solutions

[@gokulkathirvel2359]

is data drill down possible in kibana ???

[@dominicprior9844]

Great video! Thanks!

[@sonunitjsr223]

I enjoyed watching this. Thanks for the video .

[@badahzineb974]

Hello what about the tile map please ?

[@rupeshrupesh3123]

how to join 2 csvs files

[@jlgavila]

When video with JMS?

[@zaineberraji1931]

Thank you so much, It was very useful for me

[@404india4]

Suppose if i wan't to push sample.log file which contains data in text/JSON, how can we update the filter part in config file. ?

[@hassanassaad4743]

same problem,
if u manage to solve it please tell me :)

[@Bibhaw]

the price_eur column was defined as a float in the logstash configuration file but kibana is showing it as Number, Why so?

[@jacksdjfam]

Did you use the config he went over in the video? If so it was defined as an integer using mutate {convert => ["price_eur", "integer"] }

[@kalyanijoshi6884]

What is the extension of the config file-XMl? I am not able to run this configuration file

[@jacksdjfam]

.config Write the file in noted and name your file with a .config extension and in the save as type box select all files (otherwise it will save as a .txt

[@gauravagarwal1775]

it says something is wrong with your configuration I m using windows for this and have followed your steps
there is an error saying "block in initialize" what does this mean
Also it says unknown setting 'seperator' for csv
failed to execute action

[@pranjalgohain8628]

change it to 'separator'

[@nabilelaslaoui7957]

I enjoyed watching this video, thank you very much

[@FotisMylonas]

Very helpful tutorial!

[@sampathkumar4141]

Any one working in kibana dashboard

[@sampathkumar4141]

Ple response I have Chang to create dashboard in kibana I am new in this tool

[@amazingshorts69]

Where is previous lectures

[@mashuma9797]

th-cam.com/video/rKy4sFbIZ3U/w-d-xo.html

[@josephkingstonleomariamich7220]

Pakka explaination! Thanks a lot

[@ayushmalewar]

Logstash not executing after pipeline started.
It's just displaying the following output:
Settings: Default pipeline workers: 4
Pipeline main started

[@athreyavadayar4790]

putting the wrong path in your config file usually causes this. Otherwise try creating a copy of the file under a different name and using that, or restarting ES and kibana.

[@athreyavadayar4790]

you can also create a new file by heading the data you intend to use and try that

[@rabeeshcharles4207]

Do u used below thing in output section....
Stdout {}

[@MilsonPazienza]

its work for me, many thanks

[@dwivedys]

Excellent video

[@makhus8337]

Hi,
I could not find Lecture 15

[@Cintia93]

th-cam.com/video/rKy4sFbIZ3U/w-d-xo.html

[@makhus8337]

Thanks

[@gabrielcs137]

Yeaaar, thanks, work for me!!!

[@corneliussigei8641]

Amazing video

[@whotIamNot]

Please Where is the older Lecture

[@GurdeepSabarwal]

th-cam.com/video/rKy4sFbIZ3U/w-d-xo.html

[@1982haids]

Great videos.

[@pranjalgohain8628]

Not sure what really happened but the same steps were followed but it just gets stuck.. Any idea..
[2018-10-12T14:22:15,859][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5}
[2018-10-12T14:22:16,647][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[localhost:9200/]}}
[2018-10-12T14:22:16,663][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>localhost:9200/, :path=>"/"}
[2018-10-12T14:22:16,896][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"localhost:9200/"}
[2018-10-12T14:22:17,019][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>5}
[2018-10-12T14:22:17,058][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost"]}
[2018-10-12T14:22:17,251][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2018-10-12T14:22:17,298][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all"=>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2018-10-12T14:22:17,876][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#"}
[2018-10-12T14:22:17,949][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2018-10-12T14:22:18,014][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
[2018-10-12T14:22:18,561][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

[@3dlove100]

i am getting error as can not create pipeline :(

[@nerimene4995]

Me too! Did you overcome this error?

[@karancam1983]

I am trying to run ELK on windows, i was able to run elasticsearch and kibana successfully but whenever i run the config file of logstash, it says "No log4j2 configuration file could be recognized", it's almost been a week now since i am trying to run it. Please help.

[@nerimene4995]

Try to reinstall Logstash from here:
www.elastic.co/downloads/logstash

[@stephanemouako5408]

thanks for this video.

[@luisjaime4758]

thanks amazing videos

[@oridahmed8955]

how the do please install logstash

[@nerimene4995]

here you go
th-cam.com/video/rKy4sFbIZ3U/w-d-xo.html

[@mcjuds1449]

03:53 config file

[@badlekibhawna]

Nothing realtime about dashboards as promises