Damn, that was awesome! I can only imagine, what goes through your head, when you simply write lines of code like if it was normal sentence and it actually works! Well done, you've deserved your like and subscribe.
Thank you dude! I finally found a real tutorial that helps me on my way getting data out of a pdf that Isn’t easy and well structured. I have another problem that you aren’t facing here. My invoice has items that are split on multiple pages and I need to figure out how to assign that data from the second page to the invoice item of the page before!
That’s a fun challenge - similar to when you have multiple lines you want to assign to one. You have a variable that tracks where you are in the page / document, and use that to determine when to combine / assign info as needed.
@@PythonicAccountant yes, I’m hyped to test it tomorrow, I tried some other libraries too, but none of them worked for me. In the first place I wanted to do it with OCR but quickly found out that there are easier options when the pdf is searchable :D but I also face an other Problem where the PDF isn’t searchable and these pdfs contain handwritten text + copied invoices, I worked with easyOCR at some point in a different project but it‘s not working for me right now
Thank you man soo much earlier i was working with PyPDF2 but it was not providing proper spacing while reading the file but with the help of pdfplumber i fixed this issue. and now code is running completely fine. Thanks man for helping with the name of library
Bro... you saved me AF... I was trying to get data from a PDF and export that data to .csv or .xlsx and I was about to die trying to solve it :p. BTW I'm very new in scripting. Cheer from Argentina. You deserve millons of likes and suscriptors! (now I'm one of them)
Wow fantastic video! Very informative and shows how to parse PDFs in the real world! Well done. This will help me learn about Pythonic parsing from various data sources. Please make equivalent for MS Word docx files. Thank you! 👍
Awesome ! Thanks for such a informative session, can you please continue the nlp lectire on data extractions from diff files like json, webscrapping,and pdf
I do have several on the web scraping, and more PDFs, so check out my other videos! I don’t have too much on JSON but I do have a lot of experience working with that format, so can certainly look at adding some videos on that!
Love it...I'm new to python...learning as I want to do some data analysis on a dataset that can only be generated in a poorly laid out pdf (don't ask why!) So this a great intro / guide to some fantastic functions which gives me hope :-)
Absolutely fantastic video. Quick question I have, what changes would you make to the syntax if the PDF consists of let's say 350 pages or more with the same form of data across the pdf? That would be the biggest help. Thank you
Thanks for the question! The only change id make is to start a list, and iterate through each page in the pdf, appending the resulting extract to the list. That would only hold one pdf page in memory at once, and the list wouldn’t get that big.
This is very helpful, thank you. I was wondering, is there a way to search for properties besides the text itself? For example, say every vendor number is always highlighted in blue, or it's always bold and a larger font size, or it is always directly above a thick horizontal line. Is there a way for PDF plumber to search for the text that matches those particular properties?
Great tutorial! I agree, it’s the best tutorial I found d so far when working with unstructured PDF files. I have a question although my docs are not invoices but forms. Some of these PDFs have graphics, i.e., blue circles/elipses to “select” the best answer to some questions. Is there a way to identify these graphics and then select the text enclosed in them? If needed, I can share one of PDF so you can see what I mean. TIA
Extremely helpful and easy. I'm looking to extract transaction details from bank or credit card statements for analysis say 3yrs and need to come up with spending pattern or project style of spending in each category. The data in statement contains different boxes and views. Would like to see if you any videos
This is super helpful. Thank you so much for sharing. One question that I have is on missing values. I see that your invoice happens to have no missing values for the columns . What if I have missing values in few columns. Is it possible to read those missing/ blank values of the table also while reading the text to maintain consistency of columns.
Very informative thanks. I have a problem though in constructing a regular expression where the invoice may have a "Disc Amount" or not. At this time the expression assumes it is always blank.
excellent series, so useful!!! If you can, do one where someone can upload a pdf of a balance sheet and it returns a data frame of all the entries ( the column names could be "entry type(assets, liabilities...)", "entry name(ppl, inventory, retained earnings)", "value")
Thank you! Perfect explained, easy to follow your code. I have a question, related Hebrew/Arabic languages. When extracting the text, it's reversed. How can I fix it? Thank you!
Wow. find this cool tutorial after struggling with with pricy proprietary app. Btw, how to extract more than one line as single value. for instance there is an invoice with 3 lines in its description. Thank you.
Absolutely doable! Takes a few extra steps, particularly taking some steps to identify what row you are in and when you reach a last row that you need to combine all the data. I will keep this in mind for a future video
@@PythonicAccountant I lied about one thing: It's a lot above my head! Haha. But, there's literally never been a better time to learn, especially with resources like this.
Fantastic video! It just solved my work problem easily! But I also tried to extract description which is in a line after all the invoice line items. I am not sure how to get the description and append it as the last column in the DataFrame. i have found no text pattern for the description, cant use re.compile to fetch. Could you tip me on that pls?
You can usually grab descriptions not based on the pattern of the description itself, but the pattern of everything around it. For example if there are three sets of numbers that always show up before the description, then use those three numbers of the pattern to determine that the description is coming next. If you want to send me a redacted example I could try and take a look - pythoniccpa@gmail.com
Teres Lok can have a flag for that. Maybe “in_line_item = False”. When your regex picks up that you’re in the line items change it to “in_line_items = True”. Then each iteration of a new line, of in_line_items AND the regex doesn’t match a line item, you know you’re now in the first line after the line items, which is your description line.
Not sure if it is needed to do so, as every invoice is actually generated by the data system, so it should be in excel format at the very beginning, why don't we use SQL to generate all the info ?
Thank you so much. I am working with a 100 pages pdf file. I did your codes, but only 5 rows of data from 1 page were exported to the df. all of the pages are extracted. is there any solution?
Of course, regex doesn’t care what profession it’s supporting :) The biggest factor is whether you are dealing with a scanned PDF or a computer generated PDF. If it is computer-generated then you need to make sure the text you’re trying to capture is not in an image. If the PDF is scanned, or you are trying to capture data from an image, then you will need to use a different approach than just regex, as you’ll need to perform OCR. One way to do that is with Tesseract, but there are other options as well.
Thanks Steven! I learned pretty much everything I know about python through a few Coursera courses, a few python books, and some of the TalkPython trainings. Then it was just a bunch of playing around and having fun. All hands on, and trying to solve problems. Lost of googling also, that’s a programmer’s best friend no matter how experienced. I’ve included the code for most of these videos on github at github.com/danshorstein/pythonic-accountant
This Reg Expressions is the Army Swiss Knife on steroids. Is it used for Web Scraping in your experience ? Is there another Library you know of that is a jewel like this in the toolbox. I feel like I'm gulping from a River that is flowing with Pure Water.
This Content is Priceless indeed. Thank you for taking time out and create such a resourceful video step by step with detailed narration. However, I have a PDF which does not contain tabular data. It contains text data which I need to convert it into .csv excel tabular form. Can you help me with it please?
@@PythonicAccountant: I have sent you an email for the same, so please do have a look and suggest me the modifications I need to make for solving my problem statement
Amazing video! Thank you for posting! Just curious, is it possible to extract vendor # and invoice # as well, although some is empty in pdf? I have a similar problem here with a Xero journal report. The debit and credit figures are in two separate columns and not sure how to use RE to tell which number is debit, and which is credit.
Yep that’s a challenge but can usually be done. There are ways to use spacing to determine which is from which column but it gets a little tricky sometimes. If you want to send a sample page with any sensitive data redacted, I’d be happy to give it a shot - pythoniccpa@gmail.com
I have the same scenario as you with the vendor number and vendor name. How do I get it to print all vendors and not just the one on the last page? How do I use regex if my vendor number and name is separated by a hyphen? For example, 700 - Smith, Joe. Sorry I’m a python newb
Can we make something different if not a complete spreadsheet is given to us to extract data from, but if we generate such PDFs upon the transaction and we want to create a script that could directly take those generated PDFs from the chrome browser and manage to gather the name,invoice no.,date and the amount ,amount + gst= total amount,and then format them into a spreadsheet and finally make a sheet which could include the total amount collected at the end of the day or when specificed by the user. Hope it won't be difficult as it looks eagerly waiting for your reply.
Hi, can u know how to read entire pdf document using pdfplumber.. i guess, we are reading only one page at once .. can't we read entire pdf doc contains like 4 or more pages.. i am unable to do this .. pls can anyone share me the thoughts.
Yes. But pdfplumber is not very efficient (at least in my experience), so it takes a while for a large pdf document. There are other options for extracting text that may work better for larger files
I found that by opening a pdf file with Mozilla Firefox and inspecting it with the developer tools you can collect its text (with the help of JavaScript) after the web browser has converted it to HTML and maybe save it for further processing with someone programming language.
Yeah there are quite a few ways to get the text from PDF file, it’s just a matter of whether you need it to maintain information about where the text is on the page, and if you need to create an automated process versus manually clicking, that’ll impacts what method ends up being the best.
For me, in the xlsx file, date are being rendered as ######## with the message of 'possible error loss' in excel. Any idea why? (My dataframe looks completely fine) How shall I resolve, thanks!
Damn, that was awesome! I can only imagine, what goes through your head, when you simply write lines of code like if it was normal sentence and it actually works! Well done, you've deserved your like and subscribe.
Thank you!!
I am just mesmerised by the way you are playing around with python script and regex. Mind blowing!
Incredibly talented. The way you graciously remedied, remolded, and reimagined the landscape of that PDF report.
Inspired.
Superb detail and very straight forward. Thank you for showing me your workflow and insightful guide to parsing semi-structured data
Thanks!
YES! This is excellent! I am eight hours in on reports which I was loading into pdf2xl and finishing with PowerQuery. LOVE THIS!
Great!
Sir, I have no words to express my gratitude for this video. That was exactly what I needed. Thank you!
Now I convinced to learn regex and lambda to extract text from pdf, thank you!
Thank you dude! I finally found a real tutorial that helps me on my way getting data out of a pdf that Isn’t easy and well structured. I have another problem that you aren’t facing here. My invoice has items that are split on multiple pages and I need to figure out how to assign that data from the second page to the invoice item of the page before!
That’s a fun challenge - similar to when you have multiple lines you want to assign to one. You have a variable that tracks where you are in the page / document, and use that to determine when to combine / assign info as needed.
@@PythonicAccountant yes, I’m hyped to test it tomorrow, I tried some other libraries too, but none of them worked for me. In the first place I wanted to do it with OCR but quickly found out that there are easier options when the pdf is searchable :D but I also face an other Problem where the PDF isn’t searchable and these pdfs contain handwritten text + copied invoices, I worked with easyOCR at some point in a different project but it‘s not working for me right now
Thank you man soo much earlier i was working with PyPDF2 but it was not providing proper spacing while reading the file but with the help of pdfplumber i fixed this issue. and now code is running completely fine. Thanks man for helping with the name of library
pdfplumber has worked well when PyPDF2 doesnt
Very nice approach to solving this kind of problem. Thank you.
Execellent Tutorial - thank you for sharing 👏
Excellent! Thanks a lot for taking the effort to solve a realistic problem!
Bro... you saved me AF... I was trying to get data from a PDF and export that data to .csv or .xlsx and I was about to die trying to solve it :p. BTW I'm very new in scripting.
Cheer from Argentina. You deserve millons of likes and suscriptors! (now I'm one of them)
Glad to help!!
Wow fantastic video! Very informative and shows how to parse PDFs in the real world! Well done. This will help me learn about Pythonic parsing from various data sources. Please make equivalent for MS Word docx files. Thank you! 👍
Awesome ! Thanks for such a informative session, can you please continue the nlp lectire on data extractions from diff files like json, webscrapping,and pdf
I do have several on the web scraping, and more PDFs, so check out my other videos! I don’t have too much on JSON but I do have a lot of experience working with that format, so can certainly look at adding some videos on that!
Thank you so much Pythonic Accountant for sharing your awesome skills in this video, it helped me a lot!
My pleasure, thanks for watching!
Awesome - difficult example done with ease. Thanks for sharing knowledge.
Thanks, that was really helpful. Only a short side-note that you should have zoomed the screen to make it more readable.
I'm so glad I just found this channel. Haven't watched yet but Thank you
Love it...I'm new to python...learning as I want to do some data analysis on a dataset that can only be generated in a poorly laid out pdf (don't ask why!) So this a great intro / guide to some fantastic functions which gives me hope :-)
Awesome! Enjoy the journey!
dude i was just looking for something like this and this video saved me hours of work. Awesome content, thanks for sharing!
Glad I could help!
You are a literal Grandmaster.
WOW!! You are on fire with this code man! That was awesome!
This video was very helpful. Thank you very much.
Very nice! Thank you for sharing!!
Absolutely fantastic video. Quick question I have, what changes would you make to the syntax if the PDF consists of let's say 350 pages or more with the same form of data across the pdf? That would be the biggest help. Thank you
Thanks for the question! The only change id make is to start a list, and iterate through each page in the pdf, appending the resulting extract to the list. That would only hold one pdf page in memory at once, and the list wouldn’t get that big.
@@PythonicAccountant Cool Cool 👊
This is very helpful, thank you. I was wondering, is there a way to search for properties besides the text itself? For example, say every vendor number is always highlighted in blue, or it's always bold and a larger font size, or it is always directly above a thick horizontal line. Is there a way for PDF plumber to search for the text that matches those particular properties?
Thank you so much for your explaination
Thanks for the videos, this has been extremely helpful!
Great tutorial! I agree, it’s the best tutorial I found d so far when working with unstructured PDF files. I have a question although my docs are not invoices but forms. Some of these PDFs have graphics, i.e., blue circles/elipses to “select” the best answer to some questions. Is there a way to identify these graphics and then select the text enclosed in them? If needed, I can share one of PDF so you can see what I mean. TIA
Hello, that it recommended me to use in Python 3.9; Thank you.
It’s always best practice to use the most current version of python available.
Hey, Excellent video.Can you make a video on extracting text from multiple documents at once.Thanks
Extremely helpful and easy. I'm looking to extract transaction details from bank or credit card statements for analysis say 3yrs and need to come up with spending pattern or project style of spending in each category. The data in statement contains different boxes and views. Would like to see if you any videos
This is super helpful. Thank you so much for sharing. One question that I have is on missing values. I see that your invoice happens to have no missing values for the columns . What if I have missing values in few columns. Is it possible to read those missing/ blank values of the table also while reading the text to maintain consistency of columns.
Good question!
Amazing video, thank you so much!
This video is amazing! Thank you!
Very cool, thanks for making this video
You bet!
Wonderful video. Thanks much
Thanks!
Very informative thanks.
I have a problem though in constructing a regular expression where the invoice may have a "Disc Amount" or not.
At this time the expression assumes it is always blank.
Excellent ...
excellent series, so useful!!! If you can, do one where someone can upload a pdf of a balance sheet and it returns a data frame of all the entries ( the column names could be "entry type(assets, liabilities...)",
"entry name(ppl, inventory, retained earnings)", "value")
This is so damn awesome. Great work
Thanks!!
@@PythonicAccountant I’m trying to extract data nested table from pdf, is it possible for you to make a video on how to do that?
Thank you! Perfect explained, easy to follow your code. I have a question, related Hebrew/Arabic languages. When extracting the text, it's reversed. How can I fix it? Thank you!
line = line[::-1] # reverses order of the text
@@PythonicAccountant Works!!!!! Thank you very much!
How did you fix that error on 8:07? because I also get the same error
An amazing way to extract data with regex.
Btw, what editor are you using?
Good Tutorial, Does it works for Bularian PDF as well. Does it supports other Crllic languages as well
Awesome video!
Thanks!
Wow. find this cool tutorial after struggling with with pricy proprietary app.
Btw, how to extract more than one line as single value. for instance there is an invoice with 3 lines in its description.
Thank you.
Absolutely doable! Takes a few extra steps, particularly taking some steps to identify what row you are in and when you reach a last row that you need to combine all the data. I will keep this in mind for a future video
@@PythonicAccountant that would be awsome, i have the same problem, a "cell" in a table with more than one row
I think AI Builder in Power Automate is becoming a viable alternative to using Python for this. This is a little above my head, but thank you.
Thanks I’ll have to check that out
@@PythonicAccountant I lied about one thing: It's a lot above my head! Haha. But, there's literally never been a better time to learn, especially with resources like this.
@@daveg5857 haha you’ll get there, just practice and have fun!
Excellent !!! Great video !!!
Thank you! Cheers!
It's good, but I have a doubt if we want dynamically then how it will work because I think re is Checking the particular number of range
Thank you so much? Is data scraping easier via word than via pdf, would you say?
It depends on the structure and what you’re trying to get out of it.
Superb. ❤
Thanks!
This is amazing! Could you use this code if you wanted to grab text from multiple pages in a PDF?
It was very helpful,thank you for sharing!!
Thanks a lot man 👍
Excellent!!
Many thanks!
thanks man u are amazing! keep it up!
just what I was looking for, Thank You
great tutorial, thanks!
Incredible work! Congrats! Do you think it's possible to extract a nonpattern text from a PDF file and move it to another text file?
Yes, as long as there’s a way to identity which portion of text you want to grab
Fantastic video! It just solved my work problem easily! But I also tried to extract description which is in a line after all the invoice line items. I am not sure how to get the description and append it as the last column in the DataFrame. i have found no text pattern for the description, cant use re.compile to fetch. Could you tip me on that pls?
You can usually grab descriptions not based on the pattern of the description itself, but the pattern of everything around it. For example if there are three sets of numbers that always show up before the description, then use those three numbers of the pattern to determine that the description is coming next. If you want to send me a redacted example I could try and take a look - pythoniccpa@gmail.com
@@PythonicAccountant thank you for the prompt reply^^. Unfortunately, there is only the description in the line, nothing else around it :(
Teres Lok That makes it even easier! Just grab the whole line and that’s your description!
Pythonic Accountant how could I specify the description line right after the invoice line? I’m stuck here😢
Teres Lok can have a flag for that. Maybe “in_line_item = False”. When your regex picks up that you’re in the line items change it to “in_line_items = True”. Then each iteration of a new line, of in_line_items AND the regex doesn’t match a line item, you know you’re now in the first line after the line items, which is your description line.
One question....can you add the invoice number against each line item too?
As long as it’s in the data and can be identified / extracted from its pattern, definitely.
Noiiiiiiceeee
Thanks!
cool 👏
Not sure if it is needed to do so, as every invoice is actually generated by the data system, so it should be in excel format at the very beginning, why don't we use SQL to generate all the info ?
TY 😀
You bet!
Thank you so much.
I am working with a 100 pages pdf file. I did your codes, but only 5 rows of data from 1 page were exported to the df. all of the pages are extracted. is there any solution?
Really depends on a lot of factors, you’d need to do some debugging and looking at the format to see why it’s missing the rest
Can we do the same for medical forms? Regex does not seem to work for that.
Of course, regex doesn’t care what profession it’s supporting :) The biggest factor is whether you are dealing with a scanned PDF or a computer generated PDF. If it is computer-generated then you need to make sure the text you’re trying to capture is not in an image. If the PDF is scanned, or you are trying to capture data from an image, then you will need to use a different approach than just regex, as you’ll need to perform OCR. One way to do that is with Tesseract, but there are other options as well.
Great Tutorial
I have a pre-downloaded pdf to work with. But I get this erroe when I try to read the pdf: PDFSyntaxError: No /Root object! - Is this really a PDF?
This is great
I wish to be you! Great skill!
why didn't you capture the invoice number?
Probably didn’t think of it :)
This is amazing! Thanks for posting! How did you learn coding like this? By hands on? Do you have a web site or blog where we can get the codes?
Thanks Steven! I learned pretty much everything I know about python through a few Coursera courses, a few python books, and some of the TalkPython trainings. Then it was just a bunch of playing around and having fun. All hands on, and trying to solve problems. Lost of googling also, that’s a programmer’s best friend no matter how experienced. I’ve included the code for most of these videos on github at github.com/danshorstein/pythonic-accountant
Sincerely thank you! Great learning repository for individuals like me! Please keep posting!
When you are as brilliant as this guy, everything is easy :). The code is in the github link above.
Amazing!!!
Sooo cool 😁
Thanks!
waowwwwwwwww!!!!! that's sick!
This Reg Expressions is the Army Swiss Knife on steroids. Is it used for Web Scraping in your experience ? Is there another Library you know of that is a jewel like this in the toolbox. I feel like I'm gulping from a River that is flowing with Pure Water.
This Content is Priceless indeed. Thank you for taking time out and create such a resourceful video step by step with detailed narration.
However, I have a PDF which does not contain tabular data. It contains text data which I need to convert it into .csv excel tabular form. Can you help me with it please?
It’s doable but more challenging for sure.
@@PythonicAccountant: I have sent you an email for the same, so please do have a look and suggest me the modifications I need to make for solving my problem statement
Thank you so much! Could you please share the code 🙏
github.com/danshorstein/pythonic-accountant/tree/master/015%20Extract%20line%20items%20from%20PDF%20AP%20listing
Please Ihave many invoices each one with different structure of line items, sometimes you find first description qunatity ...
Amazing video! Thank you for posting! Just curious, is it possible to extract vendor # and invoice # as well, although some is empty in pdf? I have a similar problem here with a Xero journal report. The debit and credit figures are in two separate columns and not sure how to use RE to tell which number is debit, and which is credit.
Yep that’s a challenge but can usually be done. There are ways to use spacing to determine which is from which column but it gets a little tricky sometimes. If you want to send a sample page with any sensitive data redacted, I’d be happy to give it a shot - pythoniccpa@gmail.com
@@PythonicAccountant Just sent. Thank youuuuu!!!
enjoy! th-cam.com/video/P_ECMWfA3MU/w-d-xo.html
Amazing tutorial, my question is: "is there any method in python for doing same with scanned images as well like .png, jpg etc..."
Hey Sibhat, yes, but it is a little different. It’s basically the same as using a scanned PDF file. I’ll try and do a video on that soon.
@@PythonicAccountant, Extracting original layout of any scanned document and exporting it to word, excel what ever is easy :)
@@PythonicAccountant sir plz make the video on this
Harshita Gupta here you go! th-cam.com/video/glJi3LBgn9U/w-d-xo.html
I have the same scenario as you with the vendor number and vendor name. How do I get it to print all vendors and not just the one on the last page? How do I use regex if my vendor number and name is separated by a hyphen? For example, 700 - Smith, Joe. Sorry I’m a python newb
Can we make something different if not a complete spreadsheet is given to us to extract data from, but if we generate such PDFs upon the transaction and we want to create a script that could directly take those generated PDFs from the chrome browser and manage to gather the name,invoice no.,date and the amount ,amount + gst= total amount,and then format them into a spreadsheet and finally make a sheet which could include the total amount collected at the end of the day or when specificed by the user.
Hope it won't be difficult as it looks eagerly waiting for your reply.
How can we print a line on the basis of a word we are searching for in the pdf?
If word in line:
print(line)
See i have pdf and from that pdf i want to extract the task name and only the time taken by them
Hi, i'd like to aoly to a table where every cell has many rows of text, i cannot use this method cuz the multilines in every cell of the table
So cool
is it somehow possible to use python to find and LOCATE Mathematical equations in PDF's?
Great material, thank you!
Hi, can u know how to read entire pdf document using pdfplumber.. i guess, we are reading only one page at once .. can't we read entire pdf doc contains like 4 or more pages.. i am unable to do this .. pls can anyone share me the thoughts.
Yes. But pdfplumber is not very efficient (at least in my experience), so it takes a while for a large pdf document. There are other options for extracting text that may work better for larger files
I found that by opening a pdf file with Mozilla Firefox and inspecting it with the developer tools you can collect its text (with the help of JavaScript) after the web browser has converted it to HTML and maybe save it for further processing with someone programming language.
Yeah there are quite a few ways to get the text from PDF file, it’s just a matter of whether you need it to maintain information about where the text is on the page, and if you need to create an automated process versus manually clicking, that’ll impacts what method ends up being the best.
Hi,
In tabula,
Is it possible to get page number from which we got dataframe
Thanks in advance
can u share file fot testing please. Sorry my ignorance.
Here you go! www.tabs3.com/support/sample/apreports.pdf
is it possible to read read pdf from online location like google drive, sharepoint using python without download pdf
yessir! check out th-cam.com/video/8JDQMW7PuUY/w-d-xo.html
awesome , thx!!!!!!!
I have a really complex pdf table that i have to read. Would you mind helping me with it?
file_loc = 'directory'
with pdfplumber.open(file_loc) as pdf:
page = pdf.pages[0:148:]
text = page.extract_text()
print(text)
it just returns: 'AttributeError: 'list' object has no attribute 'extract_text'
please help
For me, in the xlsx file, date are being rendered as ######## with the message of 'possible error loss' in excel. Any idea why? (My dataframe looks completely fine) How shall I resolve, thanks!