Web scraping at scale with Scrapy and ScrapingHub
Episode #50,
published Tue, Mar 15, 2016, recorded Tue, Feb 16, 2016
Overcast
Apple
Castbox
PocketCasts
RSS
RadioPublic
Spotify
Pro Edition
YouTube
Every episode in your player of choice
What do you do when you are working with an amazing web application that, for whatever reason, doesn't have an API? One option is to say I wish that site had an API and give up. Or, you could use scrapy, an open source web scraping framework from Pablo Hoffman and scrapinghub.com and create your own API!
On episode 50 of Talk Python To Me, we'll talk about how to do this, when it makes sense, and even when it's allowed.
Links from the show:
Scrapy Project: scrapy.org
Scraping Hub: scrapinghub.com
Pablo on Twitter: @PabloHoffman
Pablo' site: pablohoffman.com
Michael's Video Project:
blog.michaelckennedy.net/2016/02/16/im-building-20-online-python-courses-and-i-need-your-help-video-course-library-announced
On episode 50 of Talk Python To Me, we'll talk about how to do this, when it makes sense, and even when it's allowed.
Links from the show:
Scrapy Project: scrapy.org
Scraping Hub: scrapinghub.com
Pablo on Twitter: @PabloHoffman
Pablo' site: pablohoffman.com
Michael's Video Project:
blog.michaelckennedy.net/2016/02/16/im-building-20-online-python-courses-and-i-need-your-help-video-course-library-announced
Want to go deeper? Check out our courses
Episode sponsored by
Ads served ethically