Great video. Clear and concise. Been searching high and low, for ages, for a tutorial showing how to find my way down the tree to the data I wanted in a JSON object. Learnt a lot from this. Superb!! Deserves way more 'Likes' than it has.
Nice content and amazing video John Bro, I learn multiple things from this video, especially getting JSON data from scriptTag. Thanks for making this type fruitful content
just in case anyone else has the same issue I did. Sometimes when using '.text' to gather the text it returns no text at all. Instead you must use '.string' and you can still use '.strip()' with that if needed. I'm not a professional so I have no idea why that happens, but there you go :)
Thankyou so much sir. I don't know how to thank you.. I've been waiting this video for so long.. A big fan from Philippines❤❤❤. Please keep on creating great content like this
Thanks for a great video! Can you think of another way to delete the useless data by using something other than [58:-1] ? This method definitely works, but it seems like a fragile solution that could fail if a future website code update changes this amount of spacing after the script header. Also, using .text did not work for me but .string did (for anyone having trouble).
Thanks for you comment! Yes you are right it isn’t the best solution. Another way would be to use regex and match the string that way but it is more complicated and not necessarily code change proof either, but more so than the method I used. Thanks for pointing out .string worked for you!
Problem : when i write " script = soup.find_all('script')[4] " it works showing the json data but when i write " script = soup.find_all('script')[5].text.strip() " noting is showing. What is the problem? Thanks for the great tutorial.
Using the index [5] will look for the next script tag in order and if there is not data there you won’t get anything back. In my example all the data was in one set of script tags
@@JohnWatsonRooney sorry i did mistake in the first comment, in both index it is 4. but this one " script = soup.find_all('script')[4] " fetch entire script tag data and this one " script = soup.find_all('script')[4].text.strip() " didn't showing anything.
I have a weird case when trying this line `script = soup.find_all('script')[4].text` if I omitted the `.text`, I can get the script correctly but when adding `.text`, I didn't get any output and the string of script is empty. Any ideas? ** I could solve that by using split like that `script = str(soup.find_all('script')[4]).split('initialReduxState = ')[1][:-15]` and that worked well.
Thats a great work around. Since doing this method more on some of my personal scripts i have noticed it requires a bit more effort to get it to work properly for you!
Hey, I worked with a similar website recently. Do you perhaps know WHY some websites embed the JSON content in instead of dynamically populate the page out of an API call?
Hi. I never actually stopped to look it up - I figured it was a design pattern for one of the front end frameworks, I’m not into front end so much so I don’t actually know why!
Hi John, Is possible to get this JSON script without loading the whole page ? Or maybe even get them from elsewhere ? My requests sometimes come up with 403 errors when scraping.
Hello John, can you share your thoughts about what are the pros and cons in webscraping using tools like parsehub, octoparse ects. and coding manually using beautifulsoap in python
I would probably get each script tag and check to see if the first characters match what i was expecting, or just try to force parsing them using try and except.
Great tutorial. but in a certain web that I tried I got this type of error when print(r.status_code) "SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1125)')))" could you please figure out why this happen.
hi, thanks for the video. Can you tell me how can I extract data from script 1 in this video that is similar to what I am trying to do but I always get ("session: ... , element:.....")
Hi, thanks for such an informative tutorial. I am trying to scrap a website. However, my python code is giving an error of HTTP Error 522: Origin Connection Time-out . It was perfectly fine a couple of days ago but now showing this error.
Hi! how i can do with a site that don't have de with the JSON? you have and email i cant sent you i have problem for scrap with pyhton the h1 because it is in a JSON
while getting the text within the script tag the program result empty column libraries imported beautiful soup , json , requests command line all_script = soup.find_all('script') main_script = all_script[142] print(main_script.text) result enmpty
@@JohnWatsonRooney thansk your reply I m trying to find out your videos. Where you scrape the data from script tag using scrapy. But I m not able to find it. Can you please share the video link here. Thanks once again.
@@JohnWatsonRooney This is done by using Beautiful soup but I want to complete it by using scrapy framework of python. Please share the video link here
![โรงเรียนดัดสันดาน นักเรียนเถื่อน..!! [โรงเรียนไดเม่]](http://i.ytimg.com/vi/vRKPeXn4Fl4/mqdefault.jpg)
![[Gegagedigedagedago] NEW Help Gegagedigedagedago Nugget escape from Nikocado Avocado Challenge](http://i.ytimg.com/vi/hJWnwRC2Fz8/mqdefault.jpg)
Great video. Clear and concise. Been searching high and low, for ages, for a tutorial showing how to find my way down the tree to the data I wanted in a JSON object. Learnt a lot from this. Superb!! Deserves way more 'Likes' than it has.
Nice content and amazing video John Bro, I learn multiple things from this video, especially getting JSON data from scriptTag. Thanks for making this type fruitful content
just in case anyone else has the same issue I did. Sometimes when using '.text' to gather the text it returns no text at all. Instead you must use '.string' and you can still use '.strip()' with that if needed. I'm not a professional so I have no idea why that happens, but there you go :)
Thanks.. had exactly that issue, I was lucky and the data I needed was contained in text, (soup.get_text().split())
woww thanks for the tips! I have been looking for the answer why does using '.text' returns not text at all
you saved me big time
Lifesaver
Saved my life
Thanks alot John spent hours getting error until i saw your video ..you gained a subscriber
Thanks for the sub!
THANKS MAN!! Finally something that is straightforward and works!!
Thankyou so much sir. I don't know how to thank you.. I've been waiting this video for so long.. A big fan from Philippines❤❤❤. Please keep on creating great content like this
I thought you would like this one!
Helped me through a challenging project. Thank you
Glad it helped!
6:00 you can also use the replace method and replace the whole 58 letter string with "". Would have saved you the effort of counting :)
Yes good tip thank you!
It really helped me a lot. Thanks for posting it.
Thanks for a great video! Can you think of another way to delete the useless data by using something other than [58:-1] ? This method definitely works, but it seems like a fragile solution that could fail if a future website code update changes this amount of spacing after the script header.
Also, using .text did not work for me but .string did (for anyone having trouble).
Thanks for you comment! Yes you are right it isn’t the best solution. Another way would be to use regex and match the string that way but it is more complicated and not necessarily code change proof either, but more so than the method I used. Thanks for pointing out .string worked for you!
Thank you so much. I was getting a blank response before I saw this comment. This should be pinned.
@@JohnWatsonRooney thank you for your suggestion ".string" because i have the some problem.
Thanks mate, saved me from messing up my uni project
This video helped me a lot. Thanks John I appreciate it.
You are a gentleman and a scholar.
Thank you so much. You save me. Really helpful video !!🤩
Thanks for the tutorial. Very helpful!
You've just saved me lots of time - and a huge headache. Thank you.
Thank you so much ! this was exactly what I was looking for
Problem : when i write " script = soup.find_all('script')[4] " it works showing the json data but when i write " script = soup.find_all('script')[5].text.strip() " noting is showing. What is the problem?
Thanks for the great tutorial.
Using the index [5] will look for the next script tag in order and if there is not data there you won’t get anything back. In my example all the data was in one set of script tags
@@JohnWatsonRooney sorry i did mistake in the first comment, in both index it is 4.
but this one " script = soup.find_all('script')[4] " fetch entire script tag data
and this one " script = soup.find_all('script')[4].text.strip() " didn't showing anything.
If you're still struggling with this, instead of text, use string. I tried this method because I had the same issue as you
@@stewielol Thanks man ! you saved my project
@@stewielol thank you this helped me sir!
I have a weird case when trying this line `script = soup.find_all('script')[4].text` if I omitted the `.text`, I can get the script correctly but when adding `.text`, I didn't get any output and the string of script is empty. Any ideas?
** I could solve that by using split like that `script = str(soup.find_all('script')[4]).split('initialReduxState = ')[1][:-15]` and that worked well.
Thats a great work around. Since doing this method more on some of my personal scripts i have noticed it requires a bit more effort to get it to work properly for you!
@@JohnWatsonRooney You are the MASTER my bro. Thanks a lot for all the tutorials you offered. Can I have your email? Do you have a website?
Thanks dude. This finally solved my frustration!
Very helpfull and very clear
Hey, I worked with a similar website recently.
Do you perhaps know WHY some websites embed the JSON content in instead of dynamically populate the page out of an API call?
Hi. I never actually stopped to look it up - I figured it was a design pattern for one of the front end frameworks, I’m not into front end so much so I don’t actually know why!
@@JohnWatsonRooney Thank you for your kind answer! ☺️
Hi John,
Is possible to get this JSON script without loading the whole page ? Or maybe even get them from elsewhere ?
My requests sometimes come up with 403 errors when scraping.
Thank you for the content. Quite explanatory as always. Could you mind elaborating further on how we can iterate over the elements in the json object?
one of the best videos, I had the same problem, thanks a lot!
loved it bro...your explaination is so intuitive..thanks a ton .man
Best explained 👍
Hello John, can you share your thoughts about what are the pros and cons in webscraping using tools like parsehub, octoparse ects. and coding manually using beautifulsoap in python
Thank you very much. It was very useful.
Project saver video . Ty
How to get the index of the script tag dynamically because in my case I am getting required text in different script tags on daily basis
I would probably get each script tag and check to see if the first characters match what i was expecting, or just try to force parsing them using try and except.
thankyou so much this is what i am looking for!
Hi John, just wanted to know is there any way to scrap hidden div tags/elements using playwright, beautifulsoup etc?
Thanks
very useful thankyou sir
Thanks! Very useful!
what to do if script has id or other tags? i.e., (script id ='xxxxx" type='yyyy'
You can use css selectors to get the coolest tag - # I’d for ID. I did a video on Chompjs that shows this better!
Great tutorial. but in a certain web that I tried I got this type of error when print(r.status_code) "SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1125)')))"
could you please figure out why this happen.
hi, thanks for the video. Can you tell me how can I extract data from script 1 in this video that is similar to what I am trying to do but I always get ("session: ... , element:.....")
Thank you!
In a time stamp 7:35
I upload my json file
So that i get error
'Type' object is not subscriptable
Tell me the solution of this
Hi, thanks for such an informative tutorial. I am trying to scrap a website. However, my python code is giving an error of HTTP Error 522: Origin Connection Time-out . It was perfectly fine a couple of days ago but now showing this error.
script type='text/javascript' with $(document).ready(function() it's possible?
Hi! how i can do with a site that don't have de with the JSON? you have and email i cant sent you i have problem for scrap with pyhton the h1 because it is in a JSON
while getting the text within the script tag the program result empty column
libraries imported beautiful soup , json , requests
command line
all_script = soup.find_all('script')
main_script = all_script[142]
print(main_script.text)
result
enmpty
Thanks
Thank you so much !
nice. thanks!
Amazing and wonderful as usual. Can you share the code or at least the url?
Nice jobs!!!
Can we done this task using scrapy.
Absolutely, check out my channel for some scrapy videos
@@JohnWatsonRooney thansk your reply I m trying to find out your videos. Where you scrape the data from script tag using scrapy. But I m not able to find it. Can you please share the video link here. Thanks once again.
@@JohnWatsonRooney This is done by using Beautiful soup but I want to complete it by using scrapy framework of python. Please share the video link here
bro i use regular expression to simplify this task
Absolutely, that way works well too
Thanks.
Could you write web site in description under video.
top G, thank you, but i found little bit more better solution:
Example
brand = re.search(r'"THENAMEOFSTRING":\s*"(\w+)"', script).group(1)
Thanks a lot, sir. It helped me a lot
Thanks so much!