Thank you. I'm trying to learn go and i finally had a real world thing i need to code up so started looking here after finishing the basic tutorials. Normally i would just whip something up in python as the pages i'm scraping is maybe 10 pages and trying to imput that date into json format to import into another software. hopefully i can get it working this weekend.
Hello Shaun, hope you're able to pick up some tips from the video. :) The tutorial assumes that you're working with one page, but you have mentioned that your use case would have about 10 pages or so. The Colly docs have an example that scrapes nested pages, so hopefully it will be useful. Here's the link: github.com/gocolly/colly/blob/master/_examples/coursera_courses/coursera_courses.go . Good luck with your weekend project!
Hey I am very new to Golang, just one question:- Why did you put collector. visitor at the end? First, we are making the request to it, then we are parsing it with a collector.OnHTM? So it should be at the top? Or it's just first, we are defining the structure of everything that what we need, what tag we need to parse and then we define the vision URL?
Hello Utkarsh Agrawal, thanks for stopping by. And thanks for the question. The OnHTML method is registering a callback function, but it's not executing it. We are just setting it up. The Visit method starts the collecting job at the URL we have passed to it. It will also go and call the callback function we had registered by using OnHTML. Hope that helps.
Hello SweepStakes, thank you for taking the time to leave a message. :) I believe there are so many great beginner Go courses out there, so I probably may not create one. If you're looking for a free course right here on TH-cam, you may want to check out th-cam.com/video/YS4e4q9oBaU/w-d-xo.html
Hello, the tutorials are especially great,first of all, thank you for sharing; Chinese users can't access TH-cam channel in China due to network proxy problem, I want to share your tutorials to Chinese users, can I do that,? looking forward to your reply, thanks.
mam can you say how to scrape data from a SPA site, which is built on vanillaJS and cant run on js disabled mode??? how to set headers and get the xht links from network tabs??? thank you mam
Hi there. Excel is able to read CSV files, so one way you could do it would be: 1. convert the JSON to CSV (using a package like json2csv) 2. then import the converted CSV file into Excel Hope this helps.
helo mam , i have doubt in making a web scrapper and write it to csv file ..is there any way i can message you privetly..it would be of great help for me
Hello Adithya, Go has an “encoding/csv” package you can check out. pkg.go.dev/encoding/csv I found this example that you can also take a look at here: golangdocs.com/reading-and-writing-csv-files-in-golang
@@adithyarajendran2930 I am not currently able to provide assistance on an individual basis. However, I've added a couple of links above that could be helpful. One tip would be to complete the tutorial and then replace the JSON parts with CSV.
Great Video, except i am running into errors that not too sure how to resolve. 1= .\main.go:25:4: undefined: log collector.OnHTML(".factList li", func(element *colly.HTMLElement){ factID, err := strconv.Atoi(element.Attr("id")) if err != nil { log.Println("Could not get id") 2= .\main.go:31:8: undefined: factId factDesc := element.Text fact := Fact{ ID: factId, Description: factDesc, } Hope i can hear back from you.
For the first error, can you check to see if you imported the "log" package? For the second error, you declared it as "factID" (with a capital ID) but you called it using "factId" (small letter 'd')
Hey there, you can avoid overwriting the file by making sure the filename is different every time. For example, you can consider appending a timestamp to the JSON filename.
Hi there. You can determine the structure of your struct based on the shape of the data you will be scraping. So you can inspect the HTML to see the structure.
Thanks very useful video in very short time. It helps me to create web scraper in few minutes
Thank you for stopping by! Glad the video was helpful 🙌
Thanks a lot, sister. God bless you.
Thank you for your kind comment! Hope you have a good day.
Awesome content
straight to the point explanation
and LOVELY thumbnails lol
thanks!
Thank you for your lovely comment! Happy to have you here :)
you helped me a lot!
Greetings from Brazil!
-Nate
Thanks for your comment, Nate! So glad you found the video helpful! 🙌
Thank you from Vietnam
Thank you for being here! 🙌🇻🇳
Thank you. I'm trying to learn go and i finally had a real world thing i need to code up so started looking here after finishing the basic tutorials. Normally i would just whip something up in python as the pages i'm scraping is maybe 10 pages and trying to imput that date into json format to import into another software. hopefully i can get it working this weekend.
Hello Shaun, hope you're able to pick up some tips from the video. :) The tutorial assumes that you're working with one page, but you have mentioned that your use case would have about 10 pages or so. The Colly docs have an example that scrapes nested pages, so hopefully it will be useful. Here's the link: github.com/gocolly/colly/blob/master/_examples/coursera_courses/coursera_courses.go . Good luck with your weekend project!
This was so informative and understandable. Thank you!
Glad it was helpful!
Thanks for this useful video. I look forward to seeing more Go tutorials from you in future. Best, HB
Thank you for the kind comment! Glad you found the video useful. :)
Очень хорошая подача материала!🔥
Thanks. This is useful!
Useful tutorial 🌹🌹
Thank you for your support 🙏❤️
Wow simply great 👍
Hey I am very new to Golang, just one question:-
Why did you put collector. visitor at the end? First, we are making the request to it, then we are parsing it with a collector.OnHTM? So it should be at the top?
Or it's just first, we are defining the structure of everything that what we need, what tag we need to parse and then we define the vision URL?
Hello Utkarsh Agrawal, thanks for stopping by. And thanks for the question. The OnHTML method is registering a callback function, but it's not executing it. We are just setting it up. The Visit method starts the collecting job at the URL we have passed to it. It will also go and call the callback function we had registered by using OnHTML. Hope that helps.
Bonjour, merci beaucoup, est ce que la façon de faire a changé en 2022 ?
This should still work in 2022. If you run into any problems, please let me know in a comment here 🙏
Thank you, do you think you can create a course for beginners?
Hello SweepStakes, thank you for taking the time to leave a message. :)
I believe there are so many great beginner Go courses out there, so I probably may not create one.
If you're looking for a free course right here on TH-cam, you may want to check out th-cam.com/video/YS4e4q9oBaU/w-d-xo.html
@@DivRhino thank you very much ❤️
@@abiodun6897 You're welcome
Amazing. Thank you. You are awesome.
Thank you for your kind comment :)
Do more videos like this 🔥🔥🔥
Thank you for your comment!
Hello, the tutorials are especially great,first of all, thank you for sharing;
Chinese users can't access TH-cam channel in China due to network proxy problem, I want to share your tutorials to Chinese users, can I do that,?
looking forward to your reply, thanks.
Thank you for your comment. Unfortunately I’m not too sure how to help Chinese viewers access TH-cam. Outside of using a VPN, I do not have any ideas.
mam can you say how to scrape data from a SPA site, which is built on vanillaJS and cant run on js disabled mode??? how to set headers and get the xht links from network tabs???
thank you mam
Hello. Thank you for your question. You might want to look into something like puppeteer to help you scrape an SPA. Hope that helps.
thank you
And thank you!
Thanks Sir
You're welcome, sir
She is a lady, not Sir.
do you have a video tutorial on authorization
Hello there! Thank you for your question. I do not currently have any video tutorials on authorisation.
hello, how would i extract specified data from a website and store in Excel
Hi there. Excel is able to read CSV files, so one way you could do it would be:
1. convert the JSON to CSV (using a package like json2csv)
2. then import the converted CSV file into Excel
Hope this helps.
good
helo mam , i have doubt in making a web scrapper and write it to csv file ..is there any way i can message you privetly..it would be of great help for me
Hello Adithya, Go has an “encoding/csv” package you can check out.
pkg.go.dev/encoding/csv
I found this example that you can also take a look at here:
golangdocs.com/reading-and-writing-csv-files-in-golang
@@DivRhino Can I MSG u personally mam it is a simple question but I don't know how to do..it would be great pleasure if u help ❤️
@@adithyarajendran2930 I am not currently able to provide assistance on an individual basis. However, I've added a couple of links above that could be helpful.
One tip would be to complete the tutorial and then replace the JSON parts with CSV.
@@DivRhino ok mam will do it thank you
Great Video, except i am running into errors that not too sure how to resolve.
1= .\main.go:25:4: undefined: log
collector.OnHTML(".factList li", func(element *colly.HTMLElement){
factID, err := strconv.Atoi(element.Attr("id"))
if err != nil {
log.Println("Could not get id")
2= .\main.go:31:8: undefined: factId
factDesc := element.Text
fact := Fact{
ID: factId,
Description: factDesc,
}
Hope i can hear back from you.
Resolved. I needed to :
include the log in the import
correct a typo
and install GiT
Hello there, could you put your code in a gist or git repo? Might be easier to help you out that way :)
For the first error, can you check to see if you imported the "log" package?
For the second error, you declared it as "factID" (with a capital ID) but you called it using "factId" (small letter 'd')
@@DivRhino So glad to hear back from you.
github.com/OceansBlue2017/GoLang/blob/main/main
Much Appreciated.
@@oceansblue692 Hello there, you have accidentally misspelled "factretriever" in your AllowedDomains on line 21. Let me know if that fixes it
How to not overwrite the json file content each time
Hey there, you can avoid overwriting the file by making sure the filename is different every time. For example, you can consider appending a timestamp to the JSON filename.
how to scrape if there is no ID
Hi there. You can determine the structure of your struct based on the shape of the data you will be scraping. So you can inspect the HTML to see the structure.