- 97
- 1 085 813
Jonathan Soma
เข้าร่วมเมื่อ 27 ธ.ค. 2013
Mostly boring data journalism tutorials for classes I teach for the Lede Program at Columbia's J-School - ledeprogram.com
Scraping to a pandas CSV using BeautifulSoup and Python
An overview of how to scrape with BeautifulSoup into a pandas dataframe and eventual CSV with "parent" elements.
Quick website version: jonathansoma.com/everything/scraping/convert-web-pages-to-csv/
Quick website version: jonathansoma.com/everything/scraping/convert-web-pages-to-csv/
มุมมอง: 354
วีดีโอ
Scrape by faking a browser request (to avoid bot detection??) in Python
มุมมอง 1.9Kปีที่แล้ว
Scraping isn't always easy, and this was an especially tough case. Requests and BeautifulSoup? Failed! Playwright or Selenium? Blocked! But by passing the bot check and kidnapping your cookies into a Python session, it's easy enough to pretend to be your browser without needing any fancy tools! This example is incredibly, incredibly, INCREDIBLY specific because it was for someone doing a person...
How to disable a workflow on GitHub Actions
มุมมอง 731ปีที่แล้ว
How to get a workflow to stop running on GitHub Actions
Fix "Error: Process completed with exit code" errors on GitHub Actions
มุมมอง 4.6Kปีที่แล้ว
When you get an "Error: Process completed with exit code" error on GitHub Actions, that isn't the actual error message! In this example we look at how to debug GitHub Actions errors with two examples: naming files incorrectly and forgetting to pip install the right Python modules. Even if those aren't your problems you'll at least be able to see the actual issue!
Auto-updating data visualizations from scraped data with GitHub Actions and Datawrapper
มุมมอง 418ปีที่แล้ว
An auto-updating web site makes for a quick and impressive portfolio piece: we set up an automatic scraper, feed the CSV results into a Datawrapper graphic, and host it all for free on GitHub Pages! You can see the finished product at jsoma.github.io/bad-air-cities Also features a pandas scraping trick you might not know about! NOTE: After 30 days your graphic won't be self-updating - you'll wa...
Scrollytelling with ai2html and scrollama.js
มุมมอง 2.9K2 ปีที่แล้ว
How to build a scrollytelling experience from an Illustrator file using the NYT tool ai2html and the delightful scrollytelling library scrollama.js. ai2html scrollytelling template: github.com/jsoma/ai2html-walkthroughs/blob/main/layers-scrollytelling/scrollytelling-incomplete/template.html ai2html and data viz playlist: th-cam.com/play/PLewNEVDy7gq3MSrrO3eMEW8PhGMEVh2X2.html github repo of ai2...
JavaScript interactivity with ai2html Illustrator exports (including D3)
มุมมอง 2.3K2 ปีที่แล้ว
If you want to use JavaScript or D3 with ai2html-exported Adobe Illustrator .ai graphics, it doesn't very work by default! But by exporting some layers to SVG and embedding them in your HTML you'll almost be there. THE STEPS 1) Export some of your layers as SVG by adding :svg to their name 2) Add inline_svg: true to your settings 3) Use the layers palette to name single elements you want to acc...
Generating ideas for data-driven stories and visualizations (a boring video)
มุมมอง 5682 ปีที่แล้ว
This isn't a fancy video and it isn't about programming! I'm just drinking coffee, clicking links, and sounding like an idiot who doesn't know what the Warsaw Pact is. But hey, maybe you spend your time struggling with awful datasets, trying to coax stories out of them, and you'd like an alternative approach? SPOILER: Don't look for data! Just read Wikipedia, find interesting facts, click citat...
ai2html text overflow and other annotation/label issues: an easy fix
มุมมอง 5622 ปีที่แล้ว
ai2html text overflow and other annotation/label issues: an easy fix
Cleaning up a Python data visualization in Adobe Illustrator (pandas to ai2html)
มุมมอง 2K2 ปีที่แล้ว
Cleaning up a Python data visualization in Adobe Illustrator (pandas to ai2html)
Python to Adobe Illustrator to ai2html data viz workflow (2022 update)
มุมมอง 2.4K2 ปีที่แล้ว
Python to Adobe Illustrator to ai2html data viz workflow (2022 update)
How to download files in Python with progress bars
มุมมอง 5K3 ปีที่แล้ว
How to download files in Python with progress bars
Selenium fix: SessionNotCreated "this version only supports Chrome version" error in Python
มุมมอง 22K3 ปีที่แล้ว
Selenium fix: SessionNotCreated "this version only supports Chrome version" error in Python
Data Analysis with Pandas: K-Pop idols (value_counts, filtering, datetimes)
มุมมอง 6513 ปีที่แล้ว
Data Analysis with Pandas: K-Pop idols (value_counts, filtering, datetimes)
Data Analysis with Pandas: Lumber Prices (cleaning currency, datetimes, time series)
มุมมอง 8923 ปีที่แล้ว
Data Analysis with Pandas: Lumber Prices (cleaning currency, datetimes, time series)
Data Analysis with Pandas: StockX Sneaker Sales (datetimes, resample, groupby)
มุมมอง 1.6K3 ปีที่แล้ว
Data Analysis with Pandas: StockX Sneaker Sales (datetimes, resample, groupby)
Selenium + pandas web scraping tutorial for Python (2021) but also I'm late to pick up a cat
มุมมอง 3.6K3 ปีที่แล้ว
Selenium pandas web scraping tutorial for Python (2021) but also I'm late to pick up a cat
Intro to GitHub: How to host code online with GitHub Desktop
มุมมอง 6663 ปีที่แล้ว
Intro to GitHub: How to host code online with GitHub Desktop
How to write Python in Visual Studio Code and run it in Terminal
มุมมอง 17K3 ปีที่แล้ว
How to write Python in Visual Studio Code and run it in Terminal
Customize website with bulma CSS framework: easy design tutorial with no CSS and no templates!
มุมมอง 1.4K3 ปีที่แล้ว
Customize website with bulma CSS framework: easy design tutorial with no CSS and no templates!
Copy a website's style by stealing CSS!
มุมมอง 12K3 ปีที่แล้ว
Copy a website's style by stealing CSS!
Build a portfolio site with Google Docs and node.js (plus ai2html, d3, HTML, handlebars, etc)
มุมมอง 8643 ปีที่แล้ว
Build a portfolio site with Google Docs and node.js (plus ai2html, d3, HTML, handlebars, etc)
ai2html: exporting, custom fonts and using CSS-only templates for responsive data viz
มุมมอง 1.8K3 ปีที่แล้ว
ai2html: exporting, custom fonts and using CSS-only templates for responsive data viz
How to make grouped bar charts in pandas/Python with crosstab and pivot tables and more
มุมมอง 13K3 ปีที่แล้ว
How to make grouped bar charts in pandas/Python with crosstab and pivot tables and more
Hide API keys in Python scripts using python-dotenv, .env, and .gitignore
มุมมอง 47K3 ปีที่แล้ว
Hide API keys in Python scripts using python-dotenv, .env, and .gitignore
Using paginated APIs with Python (four ways!)
มุมมอง 22K3 ปีที่แล้ว
Using paginated APIs with Python (four ways!)
How to build news applications with Svelte, Part 1 (JavaScript)
มุมมอง 1.4K3 ปีที่แล้ว
How to build news applications with Svelte, Part 1 (JavaScript)
Building a scrollytelling site with scrollama.js (no d3, no magic)
มุมมอง 6K3 ปีที่แล้ว
Building a scrollytelling site with scrollama.js (no d3, no magic)
Intro to Interactive Data Viz with D3 (the must-know pieces!)
มุมมอง 7143 ปีที่แล้ว
Intro to Interactive Data Viz with D3 (the must-know pieces!)
UPDATED! Valid JSON CORS configuration for AWS S3 public file hosting
มุมมอง 11K4 ปีที่แล้ว
UPDATED! Valid JSON CORS configuration for AWS S3 public file hosting
Very well explained
Great JOB Appreciate
Thank you so much for this playlist! Went through all of it today and learned a lot. I am very experienced in Pandas and there's something I would like to simplify in your process for any new learners here: After you perform your spatial join (I've changed some variables names here) ``` states_power = gpd.sjoin(states, powerplants, how='inner', predicate='contains') ``` You can easily map the powerplant count by converting that count into a dictionary and then mapping it to the desired dataframe using the `map` function of geopandas (also in regular pandas): ``` # convert count to dictionary state_map = states_power.name.value_counts().to_dict() # map powerplant count to states states['count'] = states.name.map(state_map) ``` And just like that! For instances that do not have a match, it will have a NaN count. I know this an old video, but I hope this helps!
Noticed the tutorial is from 2016. How relevant is the code syntax(d3js) to apply it nowadays?
Young Brenda Moore Gary Martin Sarah
Thank You, it was helpful and I was able to use the tool properly!! Much appreciated!
Excellent video, thank you!
Ah Finally Its workin in 2024 i add this cors policy for images and other stuff [ { "AllowedHeaders": [ "*" ], "AllowedMethods": [ "GET", "PUT", "POST", "DELETE" ], "AllowedOrigins": [ "*" ], "ExposeHeaders": [ "x-amz-server-side-encryption", "x-amz-request-id", "x-amz-id-2" ], "MaxAgeSeconds": 3000 } ]
Very informative! Clear explanation!
thank you so much
I'm polyGONE leave a comment.
Went through complete series. Great practical series, with useful examples and informative Videos. Kindly Continue making such videos.
Thank you !!!
nice man, tks!!
hello from Lede2024, Bulma totally changed their interface and now they have even more pages inside of pages inside of pages and you never know what is where unless you use their search box!!
:0
Thanks for making the most complete set of ai2html tutorials on the planet. Have you used figma2html yet?
Haha, it was completely accidental! Every year I hope someone who actually uses it daily will do better ones. I haven't used figma2html! but one of my instructors taught it last year so no one would have to buy Illustrator. It was a little rough around the edges but definitely useful
How would I spatial join 2 data sets with longitude and latitude coordinates and also with dates - aka how do i match up the data both temporaly and spatially
Very helpful - ty sir.
ow men thanks a lot you saved me!
cant thank enough !!
thank you it works
many thanks, I would love to join your class one day!
Do you have a link to download the zip file you used?
Hey, I got one error while plotting a different lat lon coordinates: ValueError: aspect must be finite and positive plz help...
for anyone watching this recording, there's a sound issue at 43:27...just fast forward to 44:12. It's not your computer, there's no need to refresh or restart
Just watch 5 of your videos in a row. Good content! You got a new subscriber :)
You have saved my day!, Thank you
Thankyou this helped!
python -m pip install ipykernal Defaulting to user installation because normal site-packages is not writeable ERROR: Could not find a version that satisfies the requirement ipykernal (from versions: none) ERROR: No matching distribution found for ipykernal
Excellent playlist. Thanks!
You're really good at explaining things. Please do one for windows using Git Bash or Command Prompt.
Hey great Video but I got a question, I am currently using a .env to hide my mySQL connection data inside of my python script, and when using nuitka this .env is not hidden and instead its shown in the path of the exe. Is there a way to fix it, or do you maybe know a alternative I can do to hide my mySQL connection data like the password? Or is nuitka so safe that people cant get my sourcecode at all?
thank you so much
You are such a fucking G BRO I LOVE YOU
You are awesome.
7:00
Notification was so annoying.
Thank you!
Glad I came across this tutorial of yours! Very helpful!
GOLDEN CONTENT!!! As beginner/self learner, these are hands down the best introductory explanations to HTML, CSS and Bootstrap. I get now.
THANK YOU!! As a hobbiest/newbie, this is golden content!!💯
Thanks so much. This helped me troubleshoot some S3 problems.
Thank you, Jonathan, for a fantastic explanation
what if we don't have access to change the server config and allow cors?
That's usually the case when you're on one server and trying to get data from another server. But unless you can change the other server, you won't be able to have the browser get the data... In those situations I usually set up something to automatically download the data from the remote location to somewhere I *do* control. This is kind of the same situation as th-cam.com/video/QNKxzkNpsko/w-d-xo.html except that you don't need to scrape.
Thankfully, I don't need documentations. I can just watch this video for developing concepts. Excellent ❤
Hello sir, I have a question, I'm trying to click on an icheckbox with box of value changing everytime i open the website. How can I do that? Thank you
THank you so much!!! Been sitting here to find out how to install chrome driver from cmd for hours. THanks again
Great video, thanks!
I always had a headache regarding setting up paths in mac. You opened my eyes. Thanks a lot!!
Excelente aula.