Download Data from the Web in Python

【Ｏｎｌｉｎｅ　Ｃｏｕｒｓｅｓ】
⚡Getting Started with Stata: (24 lectures + 4 assignments = 5.5 hours content): available on Udemy: www.udemy.com/course/getting-...
⚡Applied Time Series using Stata (29 lectures + 4 assignments = 6.5 hours content): available on Udemy: www.udemy.com/course/applied-...
This tutorial covers the Python HTTP library, Requests, which can be used to obtain data from the web. We write a Python code that downloads climate data from berkeleyearth.org/. The same principle can be used to download files from other urls. I show you how to download the data into a text file. Then we use NumPy to load the text file into a NumPy array. Finally, we convert the data into a Pandas DataFrame. If this sounds like crazy data fun for you, please join us!
The material including the code is on GitHub (github.com/GerhardKling/DataW....
I show you how to create and activate a virtual environment (which is optional - but useful to do). We use the pip installer.
If you do not have the pip installer, download get-pip.py. The download needs to be in same path as your Python installation - then change the directory into the folder. Using the command line, type: python get-pip.py, and finally check the installation: pip -V.
To install virtual environments, use: pip install virtualenv
We develop the code step-by-step. This will enable you to learn how to modify the code to suit your specific requirements. Please leave a comment if you have any questions. Python is the way!
Chapters
0:00 Data from the Web
0:36 Virtual Environment
2:26 Using Requests
4:02 Save Data to File
5:03 DataFrame
The channel
YUNIKARN focuses on publishing educational content in applied statistics, mathematics, and data science. In these fields, programming skills have become essential. Hence, we cover various programming languages including Python, Stata, and C++ to tackle problems and for fun.
Stay in touch
Please leave comments or follow us on Twitter ( / gerhardklings . DMs are open.
Hashtags
#datascience #python #datafromweb

[@namazishmakhametov3649]

Nice tutorial, Gerhardt, thanks for hard work! Do you have a dedicated website or personal blog besides github?

[@YUNIKARN]

Dear Namaz, thanks for your comment. This is much appreciated! The channel only started about three months ago. Hence, we do not have a website/blog yet - but it is on the to-do list for the first year. Python is the Way!

[@namazishmakhametov3649]

@@YUNIKARN thanks, looking forward for future contents. As a former policy maker, I enjoyed the STATA playlist but as a data scientist I agree - Python (and statsmodels in particular) is the way to go!

[@bhargavimurthy]

Hi, how do you use python if the file has to be downloaded from the website?

[@YUNIKARN]

Sorry, I am not quite sure what exactly your question is. Which website do you refer to?

[@mad1337nes]

You can use the same setup code as this...but you have to use response.content instead of .text. And "wb" write binary instead of "w" on the "with open" line. Should work for pdfs, etc. There's a few more lines that can be added if you're downloading large files...where you go by a block of bytes and loop through and write them in chunks. I'll look it up after I hit enter (on mobile now)