Puppeteer Tutorial - Puppeteer Full Course for Beginners 2022

แชร์
ฝัง
  • เผยแพร่เมื่อ 13 ธ.ค. 2024

ความคิดเห็น • 85

  • @alexandratravels
    @alexandratravels 2 ปีที่แล้ว +16

    By far the most friendly Puppeteer Tutorial I've ever seen on the internet, thanks a lot Michael🙏! You're quite literally one of the best teachers on youtube.

  • @mohdalim77
    @mohdalim77 21 วันที่ผ่านมา +1

    I got an interview a day after and you are saving me with this

  • @mattwaala
    @mattwaala ปีที่แล้ว +1

    Thanks bro :) This is the best tutorial on youtube about web scrapping in node . Keep it up :)

  • @imposteralchemist4521
    @imposteralchemist4521 ปีที่แล้ว +2

    Very intutive and friendly pupettier tutotial we need :)

  • @amoutsatsos
    @amoutsatsos ปีที่แล้ว +1

    Μπραβο Μιχάλη! Πολύ ωραίο turorial οτι χρειαζόμουν για να φτιάξω scraping script για ένα πελάτη. Ευχαριστώ πολύ

  • @Amar11115
    @Amar11115 ปีที่แล้ว +3

    God Bless You Man!
    Really awesome tutorial.
    Thanks a bunch!😄

  • @sesu6743
    @sesu6743 ปีที่แล้ว +2

    39:52 sudden jump where code has changed. Having a lot of trouble following..

    • @MichaelKitas
      @MichaelKitas  ปีที่แล้ว

      Sorry about that, editing mistake

  • @DSCSManoj
    @DSCSManoj ปีที่แล้ว +1

    Hi Micheal, at 18:25 the program gives an error, saying.querySelector is not a function. Can you please help

    • @MichaelKitas
      @MichaelKitas  ปีที่แล้ว

      Can you share your code or is it the exact same as mine?

    • @DSCSManoj
      @DSCSManoj ปีที่แล้ว

      Sorry there was some mistake from my end. Thanks for the early reply.

  • @trestar7624
    @trestar7624 2 ปีที่แล้ว +3

    Hi Michael, the new puppeteer version gives an error when using puppeteer-extra. I get this error:
    Error: An `executablePath` or `channel` must be specified for `puppeteer-core`
    Any help or update would be appreciated!

    • @MichaelKitas
      @MichaelKitas  2 ปีที่แล้ว

      You need to download chromium locally and specific it's path using "executablePath" option

  • @mykun8737
    @mykun8737 ปีที่แล้ว +1

    hi Michalel, Node-Cron is notwoking on vercel, could you please make a video to help me?

    • @MichaelKitas
      @MichaelKitas  ปีที่แล้ว

      It can’t work, Vercel is running server less

  • @VenomsGT
    @VenomsGT 2 ปีที่แล้ว +1

    Thank you for the tutorials!

  • @Omarwaqar-pt7wf
    @Omarwaqar-pt7wf ปีที่แล้ว +1

    I have one comcern ,what if we are building a production we site that does web scraping using puppetier the I guess it's possible that the website we are scraping can block us so how can we bypass that issue?

    • @MichaelKitas
      @MichaelKitas  ปีที่แล้ว

      Check out my latest video on bypassing browser fingerprinting: th-cam.com/video/pPQl4_3Bv7E/w-d-xo.html&lc=UgxIbun9KNHxiI89RkN4AaABAg

  • @pwchan9748
    @pwchan9748 2 ปีที่แล้ว +1

    How about a proxy with a credential? Besides, bypassing the ReCaptcha can use speech to text, maybe you can look into it.

    • @MichaelKitas
      @MichaelKitas  2 ปีที่แล้ว

      Use proxy chain to validate proxy: www.npmjs.com/package/proxy-chain

    • @MichaelKitas
      @MichaelKitas  ปีที่แล้ว

      Just created a video on proxies, check it out: th-cam.com/video/Q5qkGq_Ustc/w-d-xo.html

  • @manasmishra9019
    @manasmishra9019 ปีที่แล้ว +1

    Hey,great video man. I am facing a bit of problem in pagination as the class of the "next" button is different, any help on that frontier?

    • @MichaelKitas
      @MichaelKitas  ปีที่แล้ว

      Thank you, you can replace old selector of "next" button with new selector on code.

  • @senceryurdatapan111
    @senceryurdatapan111 2 ปีที่แล้ว +1

    Im getting an error if I want to use puppeteer-extra-plugin-stealth: UnhandledPromiseRejectionWarning: Error: An `executablePath` or `channel` must be specified for `puppeteer-core`

    • @MichaelKitas
      @MichaelKitas  2 ปีที่แล้ว

      If you use puppeteer-core, then as the error suggestes, you need to install chromium and add the path of chromium or chrome as "executablePath" in the launch options

  • @jritzeku
    @jritzeku ปีที่แล้ว +1

    WOW there is so much valuable info here! Just curious, are these similar techniques used to create something like a sneaker bot? Was hoping to practice the skills learned here and sneakerbot seems like a good one. I'll prob use my own e-commerce app instead since im sure that websites like nike will tools to prevent bots.

    • @MichaelKitas
      @MichaelKitas  ปีที่แล้ว +1

      Thank you, yes, that’s correct 👍

  • @johngenrichpilarta4089
    @johngenrichpilarta4089 ปีที่แล้ว +1

    I have a problem in pagination & saving data to csv file timestamp,, in pagination my browser is not paginating it didn't go to page 2 automatically and didn't get any item . I need to click the page one by one to get the items in every page

    • @MichaelKitas
      @MichaelKitas  ปีที่แล้ว

      Correct, you need to code it to do so

  • @dzboss5136
    @dzboss5136 2 ปีที่แล้ว +1

    how to rotate proxy autentication using seleniumwire please

    • @MichaelKitas
      @MichaelKitas  2 ปีที่แล้ว +1

      Will make a video on that soon

  • @AnastasisPyriochos
    @AnastasisPyriochos ปีที่แล้ว +1

    Very good Job Mike! I am currently into Cypress any suggestions (like a roadmap) on how to make a smooth transition to puppeteer ?Cheers Thanks

    • @MichaelKitas
      @MichaelKitas  ปีที่แล้ว

      The course should go through all the basics that you need for smooth transition, if you have any other ideas let me know and I can make a video on them

  • @fitter2boss72
    @fitter2boss72 ปีที่แล้ว +1

    By monitor looks like we scrap pages (urls) one by one. Not in parallel.

    • @MichaelKitas
      @MichaelKitas  ปีที่แล้ว +1

      No, that's not the case. Monitor shows the queue of links as we scrape a specific amount of links each time

  • @pwchan9748
    @pwchan9748 2 ปีที่แล้ว +1

    puppeteer vs selenium, which one is better for intensive automation? like speed, performance?

    • @MichaelKitas
      @MichaelKitas  2 ปีที่แล้ว

      Puppeteer is faster, I use it 99% of the time

  • @SACHINDHOTE10
    @SACHINDHOTE10 ปีที่แล้ว +1

    This video clears lot of my errors, currently working on rough project, where we can pass the search value from frontend and gets the results in frontend, using puppeteer at back end. But I am unable to integrate frontend(HTML page) with puppeteer code. I almost watched every TH-cam tutorial for this requirement but no one explains how to handle puppeteer ode from front end, would appreciate if you can create a video on this. Thanks.

    • @MichaelKitas
      @MichaelKitas  ปีที่แล้ว

      Nice idea, will do that soon

  • @korkut31
    @korkut31 ปีที่แล้ว +1

    in a 'loop' using appendFile and WriteFile is not appropriate, this leads memory issues. writeStreams should ne used.

    • @MichaelKitas
      @MichaelKitas  ปีที่แล้ว

      Thanks for pointing that out

  • @NoraJoklim
    @NoraJoklim 9 หลายเดือนก่อน +1

    I have built an app with Puppeteer but need to ship it. I don’t know how to move it from my local to a cloud or something

    • @MichaelKitas
      @MichaelKitas  7 หลายเดือนก่อน

      Making a video on how to host it to AWS Lambda today, otherwise you can use a Window VM or something similar.

  • @RumberPlumber-ob3ll
    @RumberPlumber-ob3ll ปีที่แล้ว +1

    Can we make it to search certain number of like 30 different text in search under a minute?

    • @MichaelKitas
      @MichaelKitas  ปีที่แล้ว

      Not sure exactly of your goal, but yes you can use puppeteer cluster to do multiple searches at the same time to achieve fast results

    • @RumberPlumber-ob3ll
      @RumberPlumber-ob3ll ปีที่แล้ว +1

      @@MichaelKitas thx for the tutorial puppeteer is better than python its easy to understand and its undetectable

  • @matheusmaaia
    @matheusmaaia ปีที่แล้ว +1

    Thank you, sir!

  • @pokeytradetf2582
    @pokeytradetf2582 2 ปีที่แล้ว +2

    Would be dope if you made a selenium full course.

  • @duniamotivasi8342
    @duniamotivasi8342 ปีที่แล้ว +1

    Brother what themes do you use?

  • @japeter89
    @japeter89 ปีที่แล้ว +1

    how can i capture the time it takes for all network requests to finish?

    • @MichaelKitas
      @MichaelKitas  ปีที่แล้ว

      You can log time before visiting a page and then visit a page with networkidle2 option and then log time after that is finished

  • @mod7ex_fx
    @mod7ex_fx ปีที่แล้ว +1

    thank you so much brother)

  • @UnhungSign
    @UnhungSign 9 หลายเดือนก่อน

    18:42 - I have an issue, my code is skipping this step entirely so I can't get it to log the product name, I tried messing with my query selector and I added the console log to find out what the program was getting stuck on and found out it just skips this whole code, I have the same code as you up to this step.
    for(const producthandle of productHandles){
    console.log("2");

    // pass the single handle
    const productTitle = await page.evaluate(el => el.querySelector("div > div > span > div > div > div > div.puisg-col.puisg-col-4-of-12.puisg-col-8-of-16.puisg-col-12-of-20.puisg-col-12-of-24.puis-list-col-right > div > div > div.a-section.a-spacing-none.puis-padding-right-small.s-title-instructions-style > h2 > a.a-link-normal.s-underline-text.s-underline-link-text.s-link-style.a-text-normal > span").textContent, producthandle)
    console.log(productTitle)
    };

  • @increasinginnumber
    @increasinginnumber ปีที่แล้ว +1

    help it says module not found

    • @MichaelKitas
      @MichaelKitas  ปีที่แล้ว

      Did you run 'npm install '?

    • @increasinginnumber
      @increasinginnumber ปีที่แล้ว

      @@MichaelKitas Thank you for your reply michael i just needed to save my file before running

  • @Cognitoman
    @Cognitoman 5 หลายเดือนก่อน +1

    I haven’t watched the whole series. I’m going to though

  • @Cognitoman
    @Cognitoman 5 หลายเดือนก่อน +1

    Thanks bro

  • @danielgillett370
    @danielgillett370 ปีที่แล้ว +1

    I'm really enjoying your tutorial! Thank you! I've used puppeteer once before, when it was still newly released. I've got a site that I'd ilke to interact with but it's got a lot of embedded javascript that I can''t seem to get past. Using the pages developer tools I can see the html elements. However, downloading the page, the elements are in javascript files. Is that something you'd like to help me with? I could pay you if we can get it working. If you're interested, please let me know and I'll send you the site. ...it's nothing bad. Just an e-commerce idea that I'm trying out. cheers!

    • @MichaelKitas
      @MichaelKitas  ปีที่แล้ว

      Sure, you can send me more info here: mixaliskitas@gmail.com

  • @nothaiderali
    @nothaiderali ปีที่แล้ว +1

    Theme name?

  • @ITEngines
    @ITEngines ปีที่แล้ว +1

    can you make videos on python requests.

  • @danielpaes6457
    @danielpaes6457 2 ปีที่แล้ว

    Hello, I'm from Brazil and I had difficulties to apply the content of your video, because the puppeteer version changed and with that it also changed some things that ended up in errors. If you can update the video with puppeteer content from the latest version.

    • @MichaelKitas
      @MichaelKitas  2 ปีที่แล้ว

      Can you share the errors? The core functionalities haven’t changed or not that I noticed

    • @newtonmuthigani
      @newtonmuthigani 2 ปีที่แล้ว +1

      @@MichaelKitas I get a cannot read property null when adding the s-result-item to get all products what would be they way around this , am trying to scrape the amazon website too

    • @newtonmuthigani
      @newtonmuthigani 2 ปีที่แล้ว +1

      works after adding try-catch block

  • @evgenysm
    @evgenysm 2 ปีที่แล้ว

    Thanks a lot for the video! But I've a question regarding the pagination. You do the while loop then the for loop goes over the items, once for loops finishes you check the next button state.
    I find this approach quite confusing.
    I think it would be a more clear if you save the "read items script" as a separate function or file and enable this code once navigation to the next page has been done.
    What do you think about this option?

    • @MichaelKitas
      @MichaelKitas  2 ปีที่แล้ว

      Sure, that’s a more clear approach

    • @evgenysm
      @evgenysm 2 ปีที่แล้ว

      @@MichaelKitas thanks for a quick answer, do you have an example of how this can be done? Not sure how to implement it

  • @zeddscarlxrd4331
    @zeddscarlxrd4331 ปีที่แล้ว

    Damn this awsome

  • @sekmekci
    @sekmekci ปีที่แล้ว +1

    You say "1" all the time in the video, then change it to "5", but you never show how or why you change it. Because of that, it's hard to follow, but I learned a general overview of puppeteer thanks!

  • @apostoloskotsiras9458
    @apostoloskotsiras9458 ปีที่แล้ว +1

    oreos aderfe

  • @phantazzor
    @phantazzor 2 ปีที่แล้ว +1

    I will forever get that error for the heroku version:
    Error: Failed to launch the browser process! spawn /Applications/Google Chrome ENOENT

    • @MichaelKitas
      @MichaelKitas  2 ปีที่แล้ว +1

      You need to use a different executablePath on heroku I show how to do that in the tutorial

    • @MichaelKitas
      @MichaelKitas  2 ปีที่แล้ว

      @@phantazzor Yes, at 2:25:00

    • @MichaelKitas
      @MichaelKitas  2 ปีที่แล้ว

      @@phantazzor I don't understand what your are saying. I though the issue was on heroku

    • @MichaelKitas
      @MichaelKitas  2 ปีที่แล้ว

      @@phantazzor If i didn't do it then it's not required, I am saying it's an option

    • @MichaelKitas
      @MichaelKitas  2 ปีที่แล้ว

      @@phantazzor In the video I show heroku method as well. Thanks :)