Love this video. I was able to follow along. Rather than just using your repo, I downloaded it with git then coped the relevant functions and log files from es.py into my script so I could see how it worked. Really good walkthrough - appreciate the knowledge sharing. Thank you.
wow U saved my headache - enrichment after the index is set. I was doing it in logstash - but thats a lot of planing - I do always forget smtg and had rebuild the index from syslog again and again.
Can you clarify what you are trying to do? I believe the answer is yes. You can query various index data into dataframes and then perform join operations against them
hello, I want to ask. I have executed the query and in the index data I entered. but the csv/json file is not uploaded to the file. I checked the index pattern, the health status is yellow and the file size is 208b???
It did not bring in the data then. It probably also shows 0 documents. Most likely what you are experiencing is the changes in the elasticsearch-py library since this repo was put together. If you flip to the opensearch-py library it should still work as in this video. Moving forward, elasticsearch-py has some changes to how it operates. I may try doing an updated video and github repo later to show the differences.
pretty useful. However, is there a way that, when you transform the response to a dataframe, to add a column _score? Seems the response does not include it tho...
Yes you can. For that you simply submit calls to Elasticsearch based on it's API. They have lots of examples in their documentation using curl. If you are using OpenSearch, they have their own python module that is nearly identical
Love this video. I was able to follow along. Rather than just using your repo, I downloaded it with git then coped the relevant functions and log files from es.py into my script so I could see how it worked. Really good walkthrough - appreciate the knowledge sharing. Thank you.
Thanks for this! Been putting off ILM for too long - the video with code really helps demystify certain things - cheers!
Glad it helped!
Fantastic video with extremely clear explanation. Fantastic teacher thank you
Awesome video, thanks for sharing :)
Thank you
wow U saved my headache - enrichment after the index is set. I was doing it in logstash - but thats a lot of planing - I do always forget smtg and had rebuild the index from syslog again and again.
I still love logstash but it definitely requires planning. Knowing how to do things directly with data in Elasticsearch opens a lot of doors
Can I import 2 schemas from elasticsearch to dataframes and perform joins on the jupyter notebook?
Can you clarify what you are trying to do? I believe the answer is yes. You can query various index data into dataframes and then perform join operations against them
@@HASecuritySolutions I want to import two indexes from elasticsearch to the notebook as separate dataframes and join them to cross check information
can you help me i will connect to elastic cloud with file python
hello, I want to ask. I have executed the query and in the index data I entered. but the csv/json file is not uploaded to the file. I checked the index pattern, the health status is yellow and the file size is 208b???
It did not bring in the data then. It probably also shows 0 documents. Most likely what you are experiencing is the changes in the elasticsearch-py library since this repo was put together. If you flip to the opensearch-py library it should still work as in this video. Moving forward, elasticsearch-py has some changes to how it operates. I may try doing an updated video and github repo later to show the differences.
Thank you. How could I access the elastic search records for a given period of time for a given index?
For this, use the same search in the video but at a range query to it
great stuff :)
Thank you
pretty useful. However, is there a way that, when you transform the response to a dataframe, to add a column _score? Seems the response does not include it tho...
Elasticsearch responses won't natively be a dataframe. However, you can use pandas or any other library of choice to convert the responses into one
can we create index without using elastic search python module? by using endpoint url and how to bulk upload data on those created index?
Yes you can. For that you simply submit calls to Elasticsearch based on it's API. They have lots of examples in their documentation using curl. If you are using OpenSearch, they have their own python module that is nearly identical
@@HASecuritySolutions okay, thanks for answering. So, is it possible to push data to nested folders? please can you explain with any dummy example
@@pulkitdikshit6148 do you mean nested objects? I'm uncertain what you mean by nested folders
Thank you so much.
You're welcome!