Web scraping is pulling the HTML of a website down and parsing useful data out of it.
The use cases for this type of functionality are endless.
Have a bunch of data on governmental sites that are only listed online in HTML without a download?
There's an API for that.
Do you want to keep abreast of what your competitors are featuring on their site?
There's an API for that.
Need alerts for changes on a website?
For example, enrollment is now open at your college,
and you want to be first and avoid that 8 a.m. morning slot?
Well, there's an API for that as well.
That API is Screen Scraping, and Attila Toth from Scraping Hub is here to tell us all about it.
This is Talk Python to Me, episode 283, recorded July 22, 2020.
Welcome to Talk Python to Me, a weekly podcast on Python,
the language, the libraries, the ecosystem, and the personalities.
This is your host, Michael Kennedy.
Follow me on Twitter, where I'm @mkennedy.
Keep up with the show and listen to past episodes at talkpython.fm,
and follow the show on Twitter via at Talk Python.
This episode is brought to you by Linode and us.
Python's async and parallel programming support is highly underrated.
Have you shied away from the amazing new async and await keywords because you've heard it's
way too complicated or that it's just not worth the effort?
With the right workloads, a hundred times speed up is totally possible with minor changes to your code.
But you do need to understand the internals, and that's why our course,
Async Techniques and Examples in Python,
show you how to write async code successfully as well as how it works.
Get started with async and await today with our course at talkpython.fm/async.
Attila, welcome to Talk Python to me.
Thanks for having me, Michael.
Yeah, it's great to have you here.
I'm looking forward to talking about web scraping.
It's the API that you didn't know it exists,
but it's out there if you're willing to find it.
It's like the dark API.
Yeah, it's like a secret API that you need a little work
to sort of like be able to use it, to earn the right to use it.
Yeah.
But yeah, I like it.
It's useful.
It definitely is.
And we've got some pretty interesting use cases to talk about
and a little bit of history and stuff.
I'm obviously going to talk about Scrapey,
some stuff about Scraping Hub, what you guys are doing there.
So a lot of fun stuff.
But before we get to all that, let's start with your story.
How'd you get into programming in Python?
Yeah, sure.
So actually, I got into programming back in elementary school.
I was the sort of kid, I think I was like eighth grade.
And at the time, everybody in my class in school,
everybody was asking for like,
I think it was Xbox 360 that just came out.
Everybody was asking for that for Christmas.
Yep.
And I was the one who was asking for a book about programming.
What language did you want at that time?
Yeah.
So actually, I did like a lot of research.
I think for like three months,
I was researching which programming language is the best for beginners.
So I was on different forums, Yahoo Answers,
in every place on the internet.
And many people suggested at the time, Pascal.
So I got this book.
I think it was called something like
Introduction to Turbo Pascal or something like that.
Right.
Okay.
Yeah.
So I got the book.
And that was the first step.
And then after like 40 pages into the book,
I threw it away because I found that just Googling stuff,
Googling stuff in Google and Stack Overflow,
it's just easier to learn it that way.
Nice.
So did you end up writing anything in Turbo Pascal?
Yeah.
I was just, you know,
the regular things beginners usually do.
Like in the terminal,
I, you know,
I printed my name in red.
I printed like figures with characters,
you know,
just the regular things beginners do.
But then I quickly switched to Java after Pascal.
Right.
Java, you can build web apps,
you can build GUIs,
more interesting things.
Yeah.
At the time I was interested in Android,
Android development.
Actually,
it was a few years later when I got to high school,
I started to like program in Java to develop Android apps.
I created like,
I don't know,
maybe like five or six apps for Google Play that I just found useful,
like minor things like a music player that plays music randomly,
which there are on your phone.
And the only input is how long you want to listen to music.
Oh, cool.
So I was like,
I want to listen for 20 minutes.
And they just randomly plays music for 20 minutes.
So, you know,
little things like that.
Yeah.
Cool.
How'd you find your way over to Python?
Thanks to,
thanks to web scraping actually.
Yeah.
You know,
I'm not sure which was the first point.
No,
I actually remember which was the first time I came across web scraping.
I was trying to develop like a software.
Like I was pretty into sports betting,
football and soccer.
And I didn't play high stakes because I wasn't that good to afford to do it.
But I was interested in the analysis part,
you know,
looking up stats,
looking up injury information,
looking at different factors that would determine the result,
the outcomes of games.
And I wanted to create a software that would predict the outcome of these games or some of the,
we have,
I don't know,
25 games in a weekend.
And it would predict the like,
okay,
look at these three games.
And these are going to be the results.
So I wanted to create a software like this.
Right.
And if that beats the spread that they are offering at Las Vegas,
you're like,
oh,
they're going to win by 14.
Oh,
they said they're only winning by seven.
I want to take that.
Maybe.
Yeah.
I tried that.
And I also tried.
How did it go?
Yeah.
Well,
the thing is,
I wouldn't say it went too good,
but it wasn't too bad.
I mean,
I experimented,
like,
first of all,
I was betting on like over and under,
because I found that that's like a more predictable thing to bet on.
That's what I thought,
because in reality,
it isn't,
but I bet on over and under.
And so this software would tell me that,
hey,
in this game,
you know,
if it's soccer,
there will be more than two goals in the first half.
And so what I would do is I would watch the game live,
because in life,
there are better odds,
because as the time progresses,
the odds for scoring the first half goes up.
So I just waited for like the,
I think it was the 15th minutes or something like that.
And if there is still no goals scored in the first half,
I would bet on it.
So I was doing this for like,
I think four months without money,
just testing the system.
Right.
I was watching and okay,
I would bet right now.
And I was calculating,
I had this Excel spreadsheet and everything.
And in that four months period,
I was just,
I think it was not profitable at all,
but it was close to being just a little profitable long term.
Yeah.
Well,
maybe you were onto something.
If you get some more factors,
maybe bring in a little machine learning,
who knows?
Yeah,
actually,
maybe I should pick it up.
But that's when I came across web scraping.
Right.
Because I needed data.
I needed stats.
And a lot of these places,
they don't want to give out that data.
They'll show it on their website,
but they're not going to make an API that you can just stream.
Right.
Exactly.
At the time,
there were like different commercial APIs that big news corporations use,
but it was just a hobby project.
And the only way for me to get data was to extract it from a publicly available website.
And so I started to look into it.
And at the time I was programming in Java.
So I found Jsoup.
I found HTML unit,
which is a testing library,
but it can be used for web scraping.
I found,
is it Jount?
I think it's pronounced Jount.
And I found other libraries.
And everybody was saying that,
which one is the best?
Which one is the best?
Everybody was talking on the forums about Scrappy.
And I didn't know what Scrappy was.
So I looked it up.
It's in Python.
I don't know how to program in Python.
I barely can program in Java.
But then I eventually learned Python just to learn Scrappy.
Cool.
Yeah.
And it's such a nice way to do web scraping.
It's a weird language.
It's Python.
It doesn't have semicolons,
curly braces.
There's not so many of those.
Yeah.
It was like comparing it to Java.
It's very clean.
Yeah.
That's for sure.
Java is a little verbose in all of its type information,
even for a typed language,
like say compared to C or I don't know,
maybe even C#.
Like it likes its interfaces and all of its things and so on.
Right.
Yeah.
The best thing I love about Python,
which I cannot do in Java,
is that in Python,
you can read the code like an actual sentence.
Right.
I mean,
not always,
but sometimes you can,
it's like a sentence.
Yeah.
There's this cartoon joke about Python that there's these two scientists or something like that.
And they've got this pseudocode.
It says pseudocode on their file.
And it's like pseudocode.txt.
They're like,
oh,
how do we turn this into a program?
Oh,
you change the txt to a .py and it'll run.
Yeah.
It's already there.
Yeah,
exactly.
All right.
Well,
so that's how you got into it.
And obviously your interest in web scraping.
So,
you've continued that interest today,
right?
What are you doing these days?
Yeah.
So I've been developing like a lot of spiders,
a lot of scrapers with Scrappy and other tools over the years.
And nowadays I joined the company who actually created Scrappy,
Scraping Hub.
And I'm with Scraping Hub for over a year now.
And what I'm doing is as a day-to-day is educating people about web scraping,
both on the technical side of things and business side of things,
like how to extract data from the web,
why is it useful for you or why it can be useful for you.
And so nowadays I don't really code that much unless it's for
an article or a tutorial or to showcase some new stuff.
I spend more time on creating videos and just,
you know,
teaching.
Yeah.
Well,
that sounds really fun.
You just get to explore,
right?
Yeah.
And the biggest part or the best part I love about it is that like,
I get to speak to customers who are doing web scraping or who are sort of
enjoying the benefits of web scraping.
And there are some really cool stories,
like what these customers do with data.
Yeah.
And it's really creative.
Like people can get super creative,
like how to make use of,
you know,
web data,
which is like,
it's in front of you.
You can see the data on your website.
It doesn't look like interesting or like,
you know,
but when you extract it and you structure it,
you can actually make use of it and drive,
you know,
you can do many different things,
but you can drive better decisions in companies,
which is pretty exciting.
Yeah.
And I guess if you work at a company and you're looking to think about,
I'd like to do some comparative analysis against say our competitors and what
are they doing?
What are they doing with sales?
Like in terms of discounts,
what are they doing in terms of things are featuring,
right?
You could theoretically write a web scraper that looks at your competitors'
data,
their sites,
their presentation,
and sort of gives you that over time relative to you,
right?
Or something like that.
Yeah.
Like many use cases are,
you know,
as you said about monitoring competitors or monitoring the market.
And especially it's a big thing in e-commerce where most of the sectors in e-commerce are
really like,
there are a lot of companies doing the same thing,
selling the same thing.
And it's just really hard for them to sell more products.
So with web scraping,
they can monitor the competitor prices.
They can monitor the competitors' stock information.
They can monitor pretty much everything that is on their website,
publicly available.
And they can gather this information.
And,
you know,
it can be like tens of thousands of products or more.
And you can see where you should raise your prices,
lower your prices,
and how to price your products better and do marketing better.
Yeah.
And there's a lot of implicit signals as well that are not,
you know,
they're not most companies.
Some do,
but most won't go,
there are 720 of these sold today or whatever.
Yeah.
but you could say graph,
like the number of reviews over time or the number of stars over time,
things like that would give you a sense of like a proxy,
that kind of information.
Right.
you could come up with a pretty good analysis and pretty interesting analysis.
This portion of Talk Python to Me is brought to you by Linode.
Whether you're working on a personal project or managing your enterprise's infrastructure,
Linode has the pricing,
support,
and scale that you need to take your project to the next level.
With 11 data centers worldwide,
including their newest data center in Sydney,
Australia,
enterprise-grade hardware,
S3 compatible storage,
and the next generation network,
Linode delivers the performance that you expect,
at a price that you don't.
Get started on Linode today with a $20 credit,
and you get access to native SSD storage,
a 40 gigabit network,
industry-leading processors,
their revamped cloud manager,
cloud.linode.com,
root access to your server,
along with their newest API,
and a Python CLI.
Just visit talkpython.fm/Linode
when creating a new Linode account,
and you'll automatically get $20 credit for your next project.
Oh, and one last thing,
they're hiring.
Go to linode.com slash careers,
to find out more.
Let them know that we sent you.
also, like, some of the things that people monitor with web scraping
is more like higher level.
We have the price, stock,
but those are, like, really specific things,
or values.
But with web scraping,
you can actually monitor, like,
company behaviors,
what the company is doing on a high level.
And you can achieve this by monitoring news,
looking for, like,
the text,
or the blog of the company,
and getting out some useful information from that.
And, you know, setting up alerts and things like that.
Yeah, yeah, super cool.
So there's a couple of examples that I wanted to
put out there,
and then get your thought on it.
One is, you know,
most of the world has been through the ringer on COVID-19
and all of this stuff.
In the U.S.,
we definitely have been recently getting our fair share of
disruption from that.
So there's an article on
towardsdatascience.com
where some,
the challenge was that
in the early days of the lockdowns,
at least in the United States,
we had things like Instacart
and other grocery delivery things,
like there's local grocery stores,
you could order stuff
and they would bring it to your car outside
so you didn't have to go in,
other people,
fewer people in the store,
all sorts of stuff
that was probably a benefit from that.
But if you tried to book it,
it was like a week and a half out.
Yeah.
And it was,
it was a mess, right?
And so this person,
as they wrote it,
Emmy GIL wrote this article
that talked about using web scraping
to find a grocery store delivery slots, right?
So you could basically say,
I'm going to go to my grocery store
or to like something like,
Instacart
and I'm just going to watch
until one of these slots opens up.
And then what he had to do
was actually send him a text,
like go now and book it,
place my order now or whatever.
And I think that that's a really creative way
of using web scraping.
What do you think?
Yeah, it's definitely really creative.
It's useful.
So he wasn't just only getting alerted
when there's a new spot available,
but he actually created the bot.
So it like,
it chooses that spot.
Right.
It would actually place the order.
Yeah, that's smart.
I mean, there are,
we had the same situation in Hungary
where you couldn't find any free spot.
And the thing I love about web scraping
is that it can be useful
for huge companies
to do web scraping at scale,
but it can be also useful
for literally everyone.
Yeah.
In the world,
just doing little things like this.
I have a friend who is looking
to buy an apartment
and currently the real estate market
is really like fragile,
I would say.
Yeah, yeah.
Yeah.
And what he wants to do
that I suggested him to do it
is to create,
set up a bot
that monitors
like 10 websites,
10 real estate listing websites.
And so he will get alerted
when there is a apartment available
with the attributes
he's interested in.
Right.
Okay.
That's awesome.
Yeah.
So there are so many
little things like this
in web scraping
and it's just awesome.
Yeah, I agree.
People talk often
about this concept
popularized by Al Swigert,
like automate the boring stuff.
Like, oh, I got to always
do this thing in Excel.
So could I write a script
that just generates
the Excel workbook
or I've always got to
do this other thing,
copy these files
and rename them,
write something that does that.
But this feels like
the automate the boring stuff
of web interfaces,
automate tracking
of these sites
and let it like finding out
when there's a new thing.
Yeah.
Automate the boring stuff
is a really good resource.
In this case,
it's really like
what it's about.
If you didn't create
a web scraper for this,
what you would do?
You would probably
look at the website
every single,
as often as you can,
every day,
every hour.
You'd hear somebody say,
I checked it every day
until I found
the apartment I wanted,
right?
That's what they would say.
Yeah, exactly.
Yeah.
And what you don't hear
them say so often is
my bot alerted me
when my dream apartment
was available.
Yeah.
I created a scrappy spider
that alerts me
when there is a new
apartment listing
in that area.
Yeah.
And,
but I think it's a really
great example
and it's like,
there's so many things
on your computer
that you do
that you kind of feel like,
ah,
this stuff I just got to do.
But on the web,
there's way more
even that you probably end up,
I always go to this site
and then I do this thing
and I got to do that,
right?
And web scraping
will kind of open
up a ton of that stuff.
So,
let me give you two examples.
One,
really grand,
large scale.
One,
like,
extremely small
but like,
really interesting to me
in the next,
you know,
hour.
So,
grand scale first.
When Trump was elected
in the United States
2015
and he was going to be
come president
in 2016,
there was a lot of concern
about what was going to happen
to some of the data
that had so far
been hosted
on places like
whitehouse.gov.
So,
like,
there's a lot of
climate change data
hosted there
from the scientists
and there was
a thought
that he might
take that down,
not necessarily destroy it,
but definitely
take it down
publicly from the site
and there's,
you know,
there's all these
different places,
the CDC
and other
organizations
that have a bunch
of data
that a bunch
of data scientists
and other folks
were just worried
about.
So,
there were these
big movements
and I struggle
to find the articles,
you know,
almost four years
out now.
Can't quite find
the right search term
to pull them up
but there was these
like,
save the data
hackathon type things.
So,
people got together
at different universities,
like 20,
30 people
and they would spend
the weekend
writing web scraping
to go around
to all these
different organizations,
download the data.
I think they were
coordinated across locations
and then they would
put that all
in a central location,
I think somewhere
in Europe,
like in Switzerland
or something,
so the data
could live on.
And I think
the day that
Trump was elected,
I think the
climate change data
went off of
whitehouse.gov.
So,
at least one
of the data sources
did disappear
and that's
a pretty large
scale web scraping
effort.
Some of it was
just like download
CSVs,
but a lot of it
was web scraping,
I think.
Yeah,
like the thing
is people say
that once
you put something
on the internet,
it's going to
stay there forever.
And why
this story is
interesting is that
this is information
that is useful
for everyone.
Yeah.
And this is the
kind of information
that you want
to make publicly
available.
And I think
this is one of
the things that
this movement
is about,
if I can call
it movement,
open data
to make data
open and accessible
for everyone.
And web scraping
is really
a great tool
to do that,
to extract data
from the web,
put it in a
structured format
in a database
so you can use it,
can just store
information or you
can get some
kind of intelligence
from the data.
And it's really
like without
web scraping,
what you couldn't
do anything else.
will you just
copy-paste
the whole thing
or what will
you do?
It would take
a lot of
by-hand work.
Yeah,
it wouldn't be good.
Yeah.
Yeah,
so that's really
cool.
And that was
an interesting
large scale,
like,
hey,
we have to
organize.
We've got
one month
to do it.
Let's make
this happen.
So on a
much smaller
scale,
I recently
decided to
realize that
some of my
pages on my
site were
getting indexed
by Google,
not because
anything on the
internet was
linking to them
publicly,
but something
would be in
Gmail or
somebody would
write it down
and then Google
would find it.
So I just went
through and put a
bunch of no index
meta tags on a
bunch of my
parts of my
site.
But it really
scared me because
I changed a
whole bunch of
it and I'm
pretty sure I
got it right.
But what if I
accidentally put
no index on
some of my
course pages
that advertise
them and all
of a sudden the
traffic goes from
whatever it is
to literally
zero because I
accidentally told
all the search
engines,
please don't tell
anyone about
this page.
what I was
doing actually
right before we
called unrelated
to the fact that
this was the
topic was I
was going to
write a very
simple web
scrappy thing
that goes and
grabs my
site map,
hits every
single page and
just tells me
the pages that
are not indexed
just for me to
double check.
But you know,
that template
didn't get reused
somewhere where I
didn't think it
got reused,
you know,
like the little
shared bit that
has the head
in it and so
on.
Yeah,
it's sort of
like in SEO,
like search engine
optimization,
we have this
thing,
technical SEO.
Yeah.
And actually,
it's another
use case for
web scraping or
like crawling,
where you want
to crawl your
own website,
just as you
know,
you did and
to learn the
site map or
to find pages
where you don't
have proper SEO
optimization,
like you don't
have meta
description on
some pages,
you have
multiple like
H1 tags or
whatever.
Right.
and with
crawling,
you can figure
these out.
And I just
remember the
story from
like 2016,
where web
scraping was
used to
reveal lobbying
and corruption
in Peru.
Oh,
wow.
Okay.
Yeah,
I'm not 100%
sure,
but I think
it was related
to, you
know,
the Panama
Papers.
Yeah,
I was going
to guess it
might have
something to
do with the
Panama
Papers,
which was
really interesting
in and of
itself.
Yeah,
yeah,
and they
actually,
they use
many tools
to scrape
the web,
but they
also use
Scrappy to
get like
information from
these,
I guess,
government
websites or
other websites.
But yeah,
like it's
crazy that a
web scraping
can reveal
these kinds
of really,
this is a
big thing.
Yeah.
Corruption
and web
scraping can
tell you
that,
hey,
there is
something wrong
here,
you should
pay attention.
It lets you
automate the
discovery and
the linking
of all this
different data
that was never
intended to be
put together,
right?
There's no
API that's
supposed to
take like
these donations,
this investment,
this other
construction deal,
or whatever the
heck it was,
and like link
them together.
But with
web scraping,
you can sort
of pull it
off,
right?
Yeah,
you can
structure the
whole web.
I mean,
technically,
you could
structure the
whole web,
or you could
get like a
set of
websites that
are considered
or that has
the same kind
of data you
are looking
for,
like you
can be
e-commerce,
fintech,
real estate,
whatever.
You can grab
those sites,
extract the
data,
the publicly
available data,
structure it,
and then you
can do many
sorts of
things.
You can do
like NLP,
you can just
search in the
data,
you can do
many different
kinds of
things.
Yeah,
that's cool.
So let's talk
a little bit
specifically about
Scrappy and
maybe Scraping Hub
as well,
and some of the
more general
challenges you
might run into
with web scraping.
So right on
the Scrappy
website,
homepage,
project site,
there's a
real simple
example that
says here's
why you should
use it.
You can just
create this
class,
it derives
from Scrappy.
spider,
and you
give it
some URLs,
and it
has this
parse
function,
and off
it goes.
Do you
want to
talk to us
real quickly
about what
it's like
to write
code to do
web scraping
with this
API?
With Scrappy?
Yeah.
Yeah,
so with
Scrappy,
if you use
Scrappy,
what you really
get is like a
full-fledged
web crawling
framework,
or like a
web scraping
framework.
So it really
makes it easy
for you to
extract data
from any
website.
And it
has,
there are other
libraries like
BeautifulSoup,
or LXML,
or in Java,
we have JSUP,
that are only
focused around
parsing HTML
and getting,
or getting data
from HTML and
XML files.
Right,
BeautifulSoup is
I have the
text,
I have the
HTML source,
now tell me
stuff about it,
right?
Right.
That's sort of
its whole job,
yeah.
Right,
but Scrappy,
I would say
it's like maybe
10% of the
whole picture
or the big
picture,
because with
Scrappy,
like,
yeah,
you can parse
the HTML,
but in
real-world
project,
that's not the
only task you
need to do.
In real-world
project,
you need to
process the data
as well.
You need to
structure the
data properly,
you need to
maybe put it
into a database,
maybe export it
as a CSV file
or JSON,
you need to
clean the data,
you need to
normalize the
data.
In real-world
scenarios,
you need to
do a lot of
things,
not just
scraping the
data,
and Scrappy
really makes
it easy to
develop these
spiders that
grab the
data,
and then
right there in
Scrappy,
you can clean
it,
you can process
it,
and when the
data leaves
Scrappy,
it's usable
for whatever
you want to
use it for.
Right,
maybe you get
a list or
sequence of
dictionaries that
have the pieces
of data that
you went after
or something like
that.
Yeah,
right.
So that's why
I really like
Scrappy,
but also I think
it's the best
framework to
maintain your
spiders,
because a big
problem in
web scraping
is that once
you write the
spider,
it works,
you get the
data,
it will
probably not
work in
the six
months in
the future
because the
website changes
or layout
changes.
They've decided
to switch
JavaScript
front-end
frameworks or
redesign the
navigation or
something,
right?
Exactly.
And so you
need to adjust
your code,
adjust your
selectors,
you need to
maybe adjust
the whole
processing
part of
Scrappy,
because maybe
in the past
you were able
to extract
only messy
code from
the HTML,
but they did
some changes
on the website.
Now it's
clean,
you don't need
so much
processing.
And with
Scrappy,
it's really
easy to
maintain the
code,
which is,
you know,
it's really
important if
you rely on
web-extracted
data on a
daily basis,
you need to
maintain the
code.
Yeah,
really neat.
So to me,
when I look
at this simple
API here,
it's like you
create a class,
you give it a
set of start
URLs,
and there's a
parse function,
and that
comes in,
you run some
CSS selectors on
what you get.
For each thing
that you find,
you can yield
that back as
part of the
thing, like the
result you found,
and then you
can follow the
next one.
I think that's
probably the
biggest differentiator
from what I can
tell is,
it's like as
you're parsing
it, you're
directing it to
go do more
discovery, maybe
on linked pages
and things like
that.
Yeah, like on
websites where
you need to go
deeper, maybe
because, you
know, you need to
go deeper in the
sitemap or you
need to like
paginate, you
need to do more
requests to get
the data.
Yeah.
In tools like
BeautifulSoup, you
would need to
sort of create your
own logic or like,
you know, how you
want to do that.
But in Scrappy, it's
already sort of
straightforward how
you want to do
that or how you
can do that.
And you just need
to just click this
button, click this
button, and you
paginate with
Scrappy because you
have this high-level
functions and
methods that let
you paginate and
let you do things
in a chain, you
know?
Yeah.
Yeah, very cool
API.
So let's talk about
some of the
challenges you might
get.
I mean, probably to
me, the biggest
challenge, well,
there's two problems
that I would say, the
two biggest
challenges.
One is probably
maybe generally you
could call it data
quality, but more
specifically like the
structure changed,
right?
they've added a
column to this
table and it's not
what I thought it
was anymore.
Like it used to be
temperature and now
it's pressure because
they've added that
column and I wrote it
to have three columns
in the table, now it's
got four.
The other one is I go
and I get the source
code of the page and
it says, you know,
view.application or
there's like an
Angular app or
something and you get
just a bunch of
script tags that
have no data in
them because it's not
actually executed yet.
Yeah, I mean, this
is like really all
the things that all
the reasons why you
need to maintain the
code, which can be,
you know, it takes
time to maintain the
code.
Actually, that's why
there are more
advancing machine
learning technologies
that pretty much
maintain the code
for you.
Oh, wow.
Okay.
Give us an example
of that.
Like find me the
title or find me the
stock price off of
this page, machine
magic.
Yeah.
So actually it's
not that big of a
magic.
Like in the future,
you won't need to
write things like
selectors.
You don't need to
tell Scrappy or
like, you know,
whatever tool you
use to, hey, this
is the price title,
whatever.
I want to extract
it.
This is the selector
where you can find it
in the HTML.
In the future, you
will not need to do
that.
In the future, what
you will only need to
say is that this is
the page.
this is real estate
listing or this is a
product page or this
is some kind of
financial page.
So you just need to
say the data type.
Yeah.
And then the machine
will know what
fields to look for.
So it will know that
if it's a product page,
you probably need
like price.
You probably need
product title or
like product name,
sorry, description,
stock information,
maybe reviews, if
any.
So in the future,
you just need to
specify the data type
and you will get all
the data fields that
you want with ML.
Yeah.
That's like a dream.
Well, it's sort of
like it's a reality in
some cases because
like, you know, there
are many, there are
some products out
there that do this
at Scraping Hub.
We have like, for
example, news API,
which does exactly
this thing for
articles and news.
Okay.
Yeah.
So what you do is
that, hey, this is
the page URL.
give me everything
and you get the
article title, you
get the text body,
you get, you know,
everything that is on
the page.
Well, and if any
place is going to
have enough data on
scraping to like
answer those kinds of
questions more
generally, it sounds
like you guys might.
You guys do so much
web scraping, right?
That like way more
than a lot of people
in terms of as a
platform.
I don't know how
you like share that
across projects.
I don't think that,
I don't see a good
way to do that, but
it seems like you
should be in a
position that somehow
you could build some
cool tools around all
that.
Yeah, we do a lot of
web scraping, as you
said.
And it's really about
the biggest problem is
that there is no one
schema that websites
follow when it comes
to data.
So when you scrape,
you need to like
figure out a structure
that would work for
all the websites.
and then you extract,
you sort of like
standardize the data.
And that's why the
data type is important
to specify.
Right.
What about JavaScript?
That's all the cool
kids say they should,
you don't write server
code anymore.
You don't write
templates.
Just you write just
APIs and you talk to
them with HTML and
JavaScript.
But that doesn't
scrape very well.
I'm not sure that's
necessarily what you
should be doing,
but entirely, but a
lot of people are and
it makes web scraping
definitely more
challenging.
Yeah.
Like nowadays,
all the websites
have some kind of
JavaScript running.
And if the data you
need to extract is
rendered with
JavaScript, that can
be a challenge.
And first, if you're
writing a spider,
first, you should
look for ways to get
the data without
executing JavaScript.
You can do this
sort of, you know,
I call them hidden
APIs when there is
no documented API
for the website.
But if you look in
the background,
background, that is
actually a public
API available.
Right.
So it's clearly going
back, like if you've
got a Vue.js front
end, it's probably
calling some function
that's returning like a
JSON result of the
data you actually
want.
You just got to look
in your web browser
tools to see that
happening, right?
Yeah, exactly.
Like with AJAX calls,
many times that's the
case that you just
need to sort of grab
the JSON and you can
get the data easily.
Well, and it's
formatted JSON, right?
It's not just HTML.
I mean, it's like
pretty sweet actually
to get a hold of
that feed.
Oh, yeah.
But on the other
side, if you cannot
find an API like
this, you need to
execute JavaScript using
some kind of headless
browser like Selenium
or Puppeteer or
Splash.
And because, you know,
you need an actual
browser to execute
JavaScript.
Right.
But then it's
going to take more
hardware resources to
run JavaScript.
So it's going to be
slower and it's going
to make the whole
process a little bit
more complicated.
Yeah.
There's probably no
mechanism built right
into Scrapey that
will render the
JavaScript, is there?
Like Scrappy itself
cannot render JavaScript,
but there is a really
neat integration with
Splash.
Okay.
Splash is also
made, you know, it was
created by Scraping Hub,
so it's really easy to,
and Splash is sort of
like a headless browser
created only for web
scraping.
and you can integrate
Splash with Scrappy
really easily.
It's literally just like
a middleware in Scrappy.
And then it will
execute JavaScript for
you.
Yeah.
Sweet.
Yeah.
You will be able to
see the data and then
you will be able to
extract it.
Right.
It says the headless
browser designed
specifically for web
scraping.
Turn any JavaScript
heavy site into data.
Right.
Cool.
I'll put a link to
that.
That's pretty neat.
And that's open
source.
People can just use
that as well or
what's the story with
that?
Yeah.
Just like Scrappy,
they can set up
their own server
and you can use it
that way or you can
also host it on
Scraping Hub.
Talk Python to me
is partially supported
by our training
courses.
How does your team
keep their Python
skills sharp?
How do you make
sure new hires get
started fast and
learn the Pythonic
way?
If the answer is a
series of boring
videos that don't
inspire or a
subscription service
you pay way too
much for and use
way too little,
listen up.
At Talk Python
Training, we have
enterprise tiers for
all of our courses.
Get just the one
course you need for
your team with full
reporting and
monitoring.
Or ditch that
unused subscription
for our course
bundles which
include all the
courses and you
pay about the same
price as a
subscription once.
For details,
visit training.talkpython.fm
slash business or
just email sales at
talkpython.fm.
Yeah, so let's
maybe round out
our conversation
talking about
hosting.
So I can go and
run web scraping on
my system and that
sometimes works pretty
well, sometimes it
doesn't.
There was this time
a friend of mine was
thinking about getting
a car and putting it
on one of these car
sharing sites, keep it
a little bit general
and we wanted to
know like, okay, for
a car that's similar to
the kind of car he's
thinking about getting,
how often is it
rented?
What does it rent
for?
Like, so would it
be worthwhile as
just a pure investment,
right?
So I wrote up some
web scraping stuff
that would just
monitor like a couple
of cars I picked.
After a little while,
they didn't talk to
me anymore.
The website was
mad at me for like
always asking about
this car.
Yeah.
So that seems like
a common problem and
if I do that for my
house, I probably
can't check out that
site anymore for a
while, right?
I probably shouldn't
do it for my house,
but even from a
dedicated server, like
that might cause
problems.
Yeah, it's not just
with web scraping.
Like I remember I
was doing some kind
of financial research.
It was about like one
of the cryptocurrencies
and I was using an
actual API.
Like I wasn't
scraping.
I was using an
actual API and it
was pretty basic.
Like I was just
learning how to use
this thing and I
created a for loop
and I put the
request, the API
request in the
for loop and I was
just testing and by
accident, I put like
a huge number in
the for loop, like
I don't know, like
a million or
something.
And it created a
million requests.
I mean, it tried to
create a million
requests, but it
stopped after like,
I don't know, maybe
like 10,000 requests
in a minute.
Right.
And because, you
know, the server
didn't like that I
was doing that.
And it's the same
thing with web
scraping.
You need to be
really careful not
to hit the website
too hard because
when you scrape the
web, it's really
important.
I mean, it's very
important to be
ethical and be
legal.
And one of the
things that you can
do to achieve
these is to just
put some kind of
delay limit just
over a respect the
website and make
sure that you don't
cause any harm to
the website.
Right.
As soon as people
think that what
you're doing, you
think you're just
getting data, but
they might perceive
it as this is the
distributed denial
of service attack
against my site.
We're going to
block this person,
right?
That's not a good
situation you want to
be in.
And it's also not
super nice to
whoever runs that
website to hammer
it.
Exactly.
The website doesn't
know what you want
to do.
It just sees a bot
and it's really
like, you know,
because web scraping
itself is legal.
You can do it if it's
a publicly available
website and you
need to make sure
that you are being,
I mean, we use this
in the web scraping
community.
You need to be nice.
You need to be nice
to the websites and
you know, you can
look at the robots
text file.
Usually there is a,
like a,
a delay specified
in the robots
text file that
tells you that,
hey, you should
put like three
seconds between
each request.
And, but just
in general, even
if there is no
such a rule defined
in the robots
text file, you
should pay
attention.
You should be
careful with your,
how many requests
you make, how
frequently you make
those requests.
So yeah, it's
important.
Yeah.
Interesting.
And so I guess
one option is you
guys have a cool
thing, which I just
discovered.
I don't know how
new it is, but
scrappy D, a daemon
that lets you
basically deploy your
scrappy scripts to
it and it'll schedule
and run them and
that's pretty cool and
you can control that.
So you could set that
up yourself or
obviously you have the
business side of your
whole story, which is
scraping hub.
So what does
scraping hub do that
makes this better?
Yes.
Then just like getting
a server at AWS or
Linode or something
like that.
Yeah.
So scraping hub is
really about, you can
deploy your spiders, but
it's really, it's made
for web scraping and
there is all the tools
available in the
scraping hub platform
that you would want to
use for web scraping.
Like, you know, we
mentioned scrappy and
splash in the
scraping hub platform.
You can upload your
spider, your scrappy
spider, and then you
can, with a click of
a button, you can add
splash.
So it renders
JavaScript or you can
also, if you want to
use proxies for web
scraping, you can, with
a click of a button, you
can add proxies to your
web scraping.
You can also, if you
have a website, like we
mentioned, you know,
articles and news, if you
want to extract data, we
have an API, so you
don't have to actually
write the spider.
You can just use the
API and get the data so
you don't need to
maintain the code.
So it's scraping
up is really just the, I
work at scraping
up, but I truly
believe that it's the
best platform if you're
doing web scraping at
scale, because you have
all the tools in one
place that you would
want to use for web
scraping.
Right, right.
So a lot of the stuff
you might bring your own
infrastructure for, you
guys have like as a
service type stuff?
Yeah, exactly.
You can sign up.
We have a free trial for,
I think, for most of our
services.
You can try it.
You can see how it
works.
And then there's a
monthly fee and that's
it.
Okay.
Yeah, that seems pretty
awesome.
And also we have a lot
of clients who don't want
to deal with the stuff,
deal with the technical
details.
So they just say that,
hey, I want this type of
data from these websites,
give me the data.
And so we just do the
hardware for them.
Right.
Maybe they're not even
Python people.
Yeah.
They might be like, they
don't even know how to
code or like.
Right.
They could be political
scientists.
They're like, we need to
just scrape this.
But like you guys do
scraping, right?
Yeah.
Yeah, exactly.
So let's round out our
conversation on this.
Just a quick comment on
Scraping Hub or a quick
thought on it.
So when I talked to Pablo
back in 2016, so I had a
show with him about
Scraping and Scraping Hub
and all that back in
episode 50, four years ago
or something.
One of the things that
really impressed me about
what you guys are doing
and continues to be the
case is you've taken
pretty straightforward
open source projects,
not something completely
insane, like a whole new
database model or
something, but just pretty
straightforward, but
polished model of web
scraping.
And you built a really
powerful business on top
of it.
And I think it's an
interesting model for
taking open source and
making it your job more
than just I'll do
consulting in this thing,
right?
If I wrote this web
framework, I could maybe
sustain myself by
consulting with people who
want to write websites in
that framework or so.
But this is a pretty
interesting next level
type of thing that you
guys pulled off here.
And what are your
thoughts on this?
Yeah, 100% agree.
Like I think, you know,
Pablo and Pablo Hoffman
and Shane Evans, they did
an awesome job when they
created Scrappy.
Like at the time, it was
the first real web
scraping framework, as I
know.
And they open sourced it.
And it's really like if
you look at the GitHub
page of Scrappy, it's
really like I feel like
it's a community.
Like people are chatting
about this.
People are talking about
how the community can
improve this tool.
And then the fact...
You've got like 17,000
GitHub questions or
something like that.
Oh, yeah.
Tagged on it.
Yeah, it's quite a bit.
Yeah.
And the fact that Pablo
and Shane were able to
pull this thing into
business, but in a
really...
Like people can still
use Scrappy without
using our services.
They can still use
Splash without paying.
They can use it.
It's open source.
So the fact that they
built on top of the
open source tools in a
way that you don't need
to be affiliated with the
company to get the
benefits of Scrappy and
other open source tools.
Like there are many
other tools.
It's not just Scrappy.
Like there are so many
other tools for web
scraping.
And I agree.
I think it's just really
amazing that they have
been doing this.
Yeah.
It seems like it's
really adding value
because if not the core
thing is already open
source and people would
just use it, you know?
Yeah.
Like especially if they
want to do it at scale.
Yeah.
You just cannot do huge
things like millions of
extracted records or
things like that on your
computer or on your
laptop.
You just cannot do that.
No, you get about 50,000
and then you get blocked.
Yeah.
That's an option as well.
Yeah, exactly.
So yeah, very cool.
I think it's a really neat
example because a lot of
people are thinking about
how do I take this thing
that's got some traction
I'm doing with open source
and make this my job
because I'm tired of
filling Jira tickets at
this thing that's kind of
okay but not really my
passion, right?
And cool example.
So congrats to you guys.
Keep up the good work.
Yeah, we are.
We definitely try our best.
There are some awesome
things to come in the
web scraping world,
I believe, with the
advancements of machine
learning and other stuff.
So it's going to be really
interesting to see what's
going to happen in the
next few years.
Yeah, for sure.
All right.
Well, I think that's probably
it for the time we got to
talk about web scraping.
But the final two questions
before you get out of here,
you're going to write some
code.
What editor do you use?
Python code?
PyCharm.
Right on.
And then notable PyPI
package.
I mean, there's obviously
pip install, scrappy, but
maybe some project that you
came across, you know, like,
oh, this is so awesome.
People should know about
X.
Oh, actually, it's a hard
question because I've been
sort of out of the game for
a lot of months now.
Aside from scrappy, I just
really like to, I don't have
one specific example.
I just really like to find
these GitHub repositories
where, when it's not like
a library, but it's like a
collection of, like, there
are many libraries, GitHub
repos with a collection of
templates.
Right.
You know, like, hey, if
you're a beginner, you can
use these templates to get
started.
And for me, that was
really useful when I started
out with using a new tool
like scrappy or other
libraries is that you can
use these starter codes
sort of.
Right, right.
I wonder if there's a
better scrappy, possibly.
Probably.
Yeah.
There is.
I'll throw it into the
show notes, but yeah.
Cool.
But just cookie cutter, make
me a scrappy project.
Let's roll and do some
web scraping.
Pretty neat.
Also, there is a, you
know, just last note,
awesome.
I think it's got awesome
web scraping.
OK.
Or something like that,
which has hundreds of
tools for web scraping.
Oh, yeah.
We can link it later.
Maybe if I find it.
OK.
Yeah.
It's got awesome web
scraping on GitHub.
Yeah.
Nice.
I'll put that in the show
notes as well.
I love those awesome lists.
Yeah.
It's easy to get lost
because you were like, oh,
I have this one thing and I
use it.
Then all of a sudden you're
like, oh, look at that.
There's like 10 other
options in this thing that I
thought I knew.
Yeah, exactly.
Awesome.
All right.
Attila, it's been great to
chat with all these, share
all these web scraping
stories and chat with you.
So thanks for being here.
Yeah, it's been amazing.
Thank you.
Thank you very much.
Yeah, you bet.
Bye-bye.
Bye.
This has been another episode
of Talk Python to Me.
Our guest in this episode has
been Attila Toth and it's
been brought to you by Talk
Python Training and Linode.
Start your next Python project
on Linode's state-of-the-art
cloud service.
Just visit talkpython.fm
slash Linode, L-I-N-O-D-E.
You'll automatically get a
$20 credit when you create a
new account.
Want to level up your Python?
If you're just getting
started, try my Python
Jumpstart by Building
10 Apps course.
Or if you're looking for
something more advanced,
check out our new Async course
that digs into all the
different types of Async
programming you can do in
Python.
And of course, if you're
interested in more than
one of these, be sure to
check out our Everything
Bundle.
It's like a subscription
that never expires.
Be sure to subscribe to the
show.
Open your favorite podcatcher
and search for Python.
We should be right at the
top.
You can also find the
iTunes feed at /itunes,
the Google Play feed at
slash Play, and the
direct RSS feed at
/rss on
talkpython.fm.
This is your host,
Michael Kennedy.
Thanks so much for
listening.
I really appreciate it.
Now get out there and
write some Python code.
you're welcome.
Bye.
Thank you.