When you think about processing tabular data in Python, what library comes to mind?
Pandas, I'd guess.
But there are other libraries out there, and Polar's is one of the more exciting new ones.
It's built in Rust, embraces parallelism, and can be 10 to 20 times faster than Pandas out of the box.
We have Polar's creator, Richie Vink, here to give us a look at this exciting new data frame library.
This is Talk Python to Me, episode 402, recorded January 29, 2023.
Welcome to Talk Python to Me, a weekly podcast on Python.
This is your host, Michael Kennedy.
Follow me on Mastodon, where I'm @mkennedy, and follow the podcast using @talkpython, both on fosstodon.org.
Be careful with impersonating accounts on other instances.
There are many.
Keep up with the show and listen to over seven years of past episodes at talkpython.fm.
We've started streaming most of our episodes live on YouTube.
Subscribe to our YouTube channel over at talkpython.fm/youtube to get notified about upcoming shows and be part of that episode.
This episode is brought to you by Typy.
Typy is here to take on the challenge of rapidly transforming a bare algorithm in Python into a full-fledged decision support system for end users.
Check them out at talkpython.fm/Typy, T-A-I-P-Y.
And it's also brought to you by User Interviews.
Earn extra income for sharing your software developer opinion.
Head over to talkpython.fm/userinterviews to participate today.
Hey, Richie.
Welcome to Talk Python to me.
Hello, Argo.
Thanks for having me.
Great to be here.
I feel like maybe I should rename my podcast TalkRust to me or something.
I don't know.
Rust is taking over.
As the low-level part of how do we make Python go fast?
There's some kind of synergy with Rust.
What's going on there?
Yeah, there is.
I'd say Python already was low-level languages that succeeded that made Python a success.
I mean, like NumPy, Pandas, everything that was reasonable fast was so because of C or Cytan, which is also C.
But Rust, different from C, Rust has made low-level programming a lot more fun to use and a lot more safe.
And especially if you regard multi-threaded programming, parallel programming, computer programming, it is a lot easier in Rust.
It opens a lot of possibilities.
Yeah, my understanding.
I've only given a cursory look to Rust, just sort of scan some examples.
And we're going to see some examples of code in a little bit, actually, related to pullers.
But it's kind of a low-level language.
It's not as simple as Python.
No.
Maybe a JavaScript, but it is easier than C, C++, not just in the syntax, but it does better memory tracking for you and the concurrency especially.
Yeah, well, so Rust has got a, brings a whole new thing to the table, which is called ownership and a bower checker.
And Rust is really strict.
There are things in Rust, you cannot do a C or C++.
Because at the time, there can only be one owner of a piece of memory.
And other people can, you can lend out this piece of memory to other users, but then they cannot mutate.
So it can be only one owner, which is able to mutate something.
And this restriction makes Rust a really hard language to learn.
But once you, once it clicked, once you went over that, that steep learning curve, it becomes a lot easier because it doesn't allow you things that you could do in C and C++.
But those things were also things you shouldn't do in C and C++ because they probably led to sack all the central memory issues.
And this bower checker also makes writing concurrent programming safe.
You can have many threads reading a variable all they want.
They can read concurrently.
It's when you have writers and readers that this whole thread safety, critical section, take your locks or the locks re-entering.
All of that really difficult stuff comes in.
And so, yeah, it sounds like an important key to making that.
And the same bower checker also knows when memory has to be freed and not.
But it doesn't have to, unlike in Go or Java, where you have a garbage collector, it doesn't have to do garbage connection and it doesn't have to do reference counting by Python.
It does so by just statically.
So at compile time, it knows when something is out of scope and not used anymore.
And this is real power.
I guess the takeaway for listeners who are wondering, you know, why is Rust seemingly taking over so much of the job that C and variations of C, right?
Like you said, Cython have traditionally played in Python.
It's easier to write modern, faster, safer code.
Yeah.
Probably more fun too, right?
Yeah, definitely.
And it's a language which has got its tools, right?
So it's got a package manager, which is really great to use.
It's got a real great.co, which is similar to the PyPy index.
It feels like a modern language.
Yeah.
Builds low-level, more low-level code.
You can also write high-level stuff like REST APIs, which is, I will say, also for high-level stuff, I like to write it in Rust because of the safety guarantees.
And also the correctness guarantees if my program compiles REST, I'm much more certain it is correct than when I write my Python program, which is dynamic and types can, are not enforced.
So it's always been preying on that side.
Yeah.
Python is great to use, but it's harder to write correct code in Python.
Yeah, and you can optionally write very loose code, or you could opt in to things like type hints and even mypy, and then you get closer to the static languages, right?
Are you a fan of Python typing?
Definitely.
But because they're optional, they are as strong as the weakest link.
So one library which you use, if it doesn't do the static correct or doesn't do it, it breaks.
It's quite brittle because it's optional.
I hope we get something really enforcative and really can check it.
I'd love to know if it's possible because of the dynamic nature of Python.
Python can do so many things, jump dynamically.
And technically, we just cannot know from, I don't know how far it can go.
But yeah.
In PowerRest as well, we use mypy type hints, which prevent us from having a nutbox.
Most of the way, the IDE experience much nicer.
Yeah.
Pypins are great.
They really help you also think about your library.
I think you really see a shift in modern Python and Python 10 years ago where it was more dynamic.
And then the dynamic, I remember, I thought it was more seen as a strength than currently I've made.
Yeah, I totally agree.
And I feel like when type hints first came out, you know, this was, yes, wow, at this point, kind of early Python 3.
But it didn't feel like it at the time, you know.
Python 3 had been out for quite a while.
When type hints were introduced, I feel like that was Python 3, 4.
But anyway, that was, put it maybe six years into the life cycle of Python 3.
But still, I feel like a lot of people were suspicious of that at the moment.
Yeah.
You know, they're like, oh, what is this weird thing?
We're not really sure we want to put these types into our Python.
And now a lot less.
There's a lot less of those reactions.
Yeah.
I see.
Yeah.
I see Python having two, probably more.
But I often see Python as the really fun, nice to own duct tape language where I can, in my, for instance, in Jupyter notebook, I can just hack away and try interactively what happens.
And for such code, type hints don't matter.
But once I write more of a library or product or tool, then type hints were really great.
I believe they came about that Dropbox really needed them.
They have a huge Python project.
It's a really tough one.
I'm not really sure.
Yeah.
And I heard some guy who has something to do with Python used to work there.
Yeah.
Guido used to work there.
I think even at that time.
All right.
So a bit of a diversion from how I often start the show.
So let's just circle back real quick and get your story.
How did you get into programming and Python and Rust as well, I suppose?
I got into programming.
I just wanted to learn programming.
A friend of mine who did program a lot of PHP said, learn Python.
Like that.
He gave me an interactive website where I could do some puzzles.
And I really got hooked to it.
It was a fun summer.
Yeah.
I was programming a lot.
I started automating.
My job was a civil engineer at the moment.
And I started.
It was a lot of mundane tasks.
It's been repetitive.
And I just found ways to automate my job.
And eventually I was doing that for a year or three or four.
And then I got into data science and I switched jobs.
I became a data scientist and later a data engineer.
Yeah.
So that was Python mostly.
I've always been looking for more languages, playing with Hasbro,
playing with Go, playing with Dubstrip, or just playing with Scala.
And then I found Rust.
Rust really, really.
And you had, like, you learn a lot about how computers work.
Yeah.
So I had a new renaissance of the first experience with Python.
And another summer with graphs.
And I'm doing a lot of the project.
But right there.
I don't know.
A lot of projects.
And all this game, one of those hobby projects just to use last month.
Now it's got quite the following.
And we're going to definitely dive into that.
But let me pull it up.
It does right here.
13,000 GitHub stars.
That's a good number of people using that project.
Yeah.
Crazy, isn't it?
Yeah, it is.
And it's on GitHub stars.
It's the process where I will get it to.
Wow.
Incredible.
You must be really proud of that.
Yeah.
If you would have told me this two years ago, I would never be.
But it happens slow enough so you can get accustomed to that.
Yeah.
That's cool.
Kind of like being a parent.
The challenges of the kids are small.
They're intense, but there are only a few things they need when they're small.
And you kind of grow with it.
Yeah.
So a couple of thoughts.
One, you had the inverse style of learning to program that I think a lot of computer science
people do and certainly that I did.
It could also just be that I learned it a long time ago.
But when I learned programming, it was, I'm going to learn C and C++.
And then you're kind of allowed to learn the easier languages.
But you will learn your pointers.
You'll have your void star, star, and you're going to like it.
You're going to understand what a pointer to a pointer means.
And we're going to get, I mean, you start inside of the most complex,
closest to the machine.
You work your way out.
You kind of took this opposite.
Like, let me learn Python where it's much more high level.
It's much, you know, if you choose to be often say very much more away from the hardware and
the ideas of memories and threads and all that.
And then you went to Rust.
So was it kind of an intense experience?
You're like, oh my gosh, this is intense.
Or had you studied enough languages by then to become comfortable?
Well, yeah, yeah, no.
So the going from high level to low language, I think it makes natural sense.
You're learning it yourself.
There's no professor telling me you learn your pointers.
Yeah.
I think this also helped a lot because at that point you're really custom programming to algorithms.
Yeah, but you can, I believe you should learn one thing, one new thing at a time.
And that you can really own that knowledge that you're on.
But for us, I wouldn't say you should learn Rust as in first language.
It'd be really terrible because you meet, terrible.
But other languages also don't help you much because the power checker is quite unique.
It doesn't let you do things you can do in other languages.
So what you, what you learn there, the languages that allow you to do that, they just encourage
you because you were, they encourage the wrong behavior, right?
Well, yeah.
So nine out of, nine out of 10 times, it turns out by compiling, not letting you do that one
thing, that one thing you wanted was probably really bad to begin with.
that to really, so in Rust, your code is always a lot flatter.
It's always really clear who owns the memory, how deep your nesting is.
It's always one degree or most of the times it's, it's not that complicated.
You, you make things really flat and really easy to reason about.
And in the beginning of the project, it seems okay, a bit over constraining, but when, I mean,
software will become complex and complicated and then you're happy to compile a notch to this.
Yeah, absolutely.
In this direction.
It seems like a better way.
Honestly, you know, you get a sense of programming in a more simple language that doesn't ask so many
low level concepts of you.
And then you're ready.
You, you can add on these new ones.
Uh, so I, I feel like a lot of how we teach programming and how people learn programming is
a little bit backwards, to be honest.
Yeah.
Anyway, enough on that.
So you were a civil engineer for a while and then you became a data scientist and now you've
created this library.
Still working as a data scientist now?
No, no.
I got sponsored two years ago for two days a week.
And yeah, just to use the time to get a polar.
And currently I stopped all my day jobs.
I'm going full time on all the other side and trying to live on sponsorships, which is not
really working.
It's not enough at this time.
I hope to start a foundation and get some proper sponsors.
Yeah.
That'd be great.
Yeah.
That's awesome.
Yeah.
It's still awesome that you're able to do that, even if, you know, you still needed to grow
a little bit.
Yeah.
We'll have you on a podcast and let other people know out there who, who may be using your
library.
Yeah.
Maybe they can, you know, put a little sponsorship and get up sponsors.
I feel like get up sponsors really made it a lot easier for people to, to support.
Cause there used to be like PayPal donate buttons and other, other things like that.
And one, those are not really recurring.
And two, you've got to go find someplace and put your credit card.
You know, many of us already have a credit card registered at GitHub.
It's just a matter of checking a box and monthly it'll just go, you know, it's kind of like the
app store versus buying independent apps.
It just cuts down a lot of the friction.
I feel like it's been really positive mostly for open source.
Yeah.
I think it's good to, as a way to say, thank you.
With, it isn't enough to pay the bills.
I think for most developers, it isn't, but, I hope we get there.
The companies, who use it to give a bit more, bit more back.
I mean, we have a lot of money.
I agree.
It's really, really ridiculous that there are banks and VC funded companies and things like
that, that have not necessarily in terms of the VC ones, but definitely in terms of
financial and other large companies that make billions and billions of dollars in profit on top of
open source technology.
Yeah.
And many of them don't give anything back, which is, it's not criminal because the licenses allow it,
but it's, it's certainly borders on immoral to say, we're making all this money and not at all support.
The people who are really building the foundations that we build upon.
Most of my sponsors are developers.
Yeah.
So, yeah, we'll let's hope it changes.
I don't know.
Yeah.
Well, I'll continue to beat that drum.
This portion of talk Python to me is brought to you by type.
I type.
I is the next generation open source Python application builder with type.
I, you can turn data and AI algorithms into full web apps in no time.
Here's how it works.
You start with a bear algorithm written in Python.
You then use type.
I is innovative tool set that enables Python developers to build interactive end user applications quickly.
There's a visual designer to develop highly interactive GUIs ready for production.
And for inbound data streams, you can program against the type I core layer as well.
I buy core provides intelligent pipeline management, data caching, and scenario and cycle management facilities.
That's it.
You'll have transformed a bear algorithm into a full fledged decision support system for end users.
I buy is pure Python and open source and you install with a simple pip install type.
For large organizations that need fine grained control and authorization around their data.
There is a paid type I enterprise edition, but the type I core and GUI described above is completely free to use.
Learn more and get started by visiting talkpython.fm/type I that's T-A-I-P-Y.
The links in your show notes.
Thank you to type I for sponsoring the show.
Let's talk about your project.
So Polars and the RS is for rust, I imagine at the end.
Yeah.
But tell us about the name Polars, like polar bear, but Polars.
Yeah.
So I started writing a data from library and initially it was only for for us, my ID or two.
You can tell you, so all the people doing data science and Python, you're like, wow.
Yeah.
What can I do for these people?
Right?
Yeah.
And I wanted to give a week to the ponders project, but I wanted a bear that was better,
faster, I don't know, stronger.
So luckily a panda bear is in the most practical bear.
So I had a few to choose, but the Grizzly, yeah, the polar has the RS.
So that's a lot of the incentives.
Yeah.
So, yeah.
So the subtitle here is lightning fast data frame library for rust and Python.
And you have two APIs that people can use.
We'll get to dive into those.
Yeah.
Because we read an angle in rust.
It's a complete data frame library in rust.
You can expose it to many fun bits, but it is already front-end thing.
Well, Python, Node.js, R is coming up and normal JavaScript is coming up.
And Ruby, there is also a polarity group.
So, interesting.
So for the JavaScript one, are you going to use WebAssembly?
Yeah.
Right.
Which is pretty straightforward because rust comes from Mozilla.
WebAssembly, I believe also originated, they kind of originated as a, somewhat tied
together.
Yeah.
Story.
So, Rust C++ C can compile to WebAssembly.
It's not really straightforward because the WebAssembly virtual machine isn't like your normal
OS.
So there are a lot of things harder, but we're, we are working on the challenges.
Okay.
Well, that's pretty interesting.
But for now you've got Python and you've got Rust and that's great.
Let's, I think a lot of people listening, myself included, when I started looking into this,
immediately go to, it's like pandas, but rust, you know, it's like pandas, but instead of C at
the bottom, it's, it's rust at the bottom.
And that's somewhat true, but mostly not true.
So let's start with you telling us, you know, how is this like pandas and how is it different
from pandas?
Yeah.
So it's not like pandas.
I think it's different on two ways.
So we have the API and we have the application and which one should I start with?
I think.
Yeah.
Bottom up.
Sure.
All right.
So that was my critique from pandas.
And they didn't start by the library.
They do whatever was there already with work, do it for that purpose.
And pandas built on NumPy.
And NumPy is a great library.
It's built from numerical process and not from relational.
Versus in relational data is completely different.
You have string data, nested data.
And this data has probably put as Python object in those NumPy lines.
And if you know anything about memory, then in this array, you have pointer with where each
Python object is somewhere else.
So if you traverse this memory, every point you must look it up somewhere else.
But memory is not a cache where the cache means, which is a 200x slowdown per element to traverse.
Yeah.
So for people listening, what you're saying, the 200x slowdown is the L1, L2, L3 caches,
which all have different speeds and stuff, but the caches that are near the CPU versus
main memory, it's like two to 400 times slower, not aging off a disk or something.
It's really different, right?
It's really a big deal.
It's a big deal.
It's terribly slow.
It also, Python has a GIL.
It also blocks multi-treading.
If you want to read the string, you cannot do this from different threads.
If you want to modify the string, there's only one thread that can access Python.
They also didn't take into account anything from databases.
So databases are based in from the 1950s.
There's been a lot of research in databases, how we do things fast, write a query, and then
optimize this query because the user that uses your library is not the expert.
It doesn't write optimized query.
No, but we have a lot of information, so we can optimize this query and execute this in
the most, in a very efficient way.
Well, that's an interesting idea.
Yeah.
And Pandas just executes it and gives you what you ask.
And what you ask is probably not.
Yeah, that's interesting because as programmers, when I have my Python hat on, I want my code
to run exactly as I wrote it.
I don't want it to get clever and change it.
I, you know, if I said do a loop, do a loop.
If I, if I said put it in a dictionary, put it in a dictionary.
But when I write a database query, be that against Postgres with relational or MongoDB,
there's a query planner and the query planner looks at all the different steps.
Should we do the filter first?
Can we use an index?
Can we use a compo?
Which index should we choose?
All of those things, right?
And so what you tell it and what happens, you don't tell it how to do finding the data,
the database.
You just give it, here's kind of the expressions that I need, the predicates that I need you
to work with.
And then you figure it out.
You're smart.
You're the database.
So one of the differences I got from reading what, what you've got here so far is it looks
like, I don't know if it goes as far as this database stuff that we're talking about,
but there's a way for it to build up the code it's supposed to run.
And it can decide things like, you know, these two things could go in parallel or things along
those lines.
Right?
Yeah.
Well, it is actually very similar.
It is a vectorized query.
And you can, the only thing that doesn't make us a database is that we don't have any,
we don't bother with the, with file structures that we write, like the persistence and transactions
and all that.
Yeah.
So we have different kinds of databases, you have OLAP and OLTP, transactional
modeling, which works often on one.
So if you do a REST API query and you modify one user ID, then you're transactional.
And if you're doing OLAP, that's more analytical.
And then you do large aggregations of large whole tables.
And then you need to process all the data and those different databases,
these eyes lead to different query optimizers.
And OLAP is focused on OLAP.
But yeah, we, so as you described, you've got two ways of programming things.
One is procedural, which Python mostly is.
So you tell exactly if you want to get a couple coffee, how many steps should take forward.
They rotate 90 degrees, take three steps.
They rotate 90 degrees.
You can put, write down the whole algorithm, how to get a coffee.
Or you could just say, get me a coffee.
I'd like to sell sugar and then let the algorithm, let query engine decide how to best get.
Right.
And that's more declarative.
You describe the end result.
And as it turns out, this is also very readable because you declare what you want and the intent
is readable in the query.
And if you're doing more procedural programming, you describe what you're doing.
And the intent often needs to come from comments.
Like what are we trying to do when we follow this?
Right.
Yeah.
That makes a lot of sense.
And that's very interesting.
Yeah.
Sorry.
And that's why the, so the first thing is we write from, we write a database engine,
a premium engine scratch and really think about multiprocessing, about cache, caches, about also out
of core, we think process data doesn't fit into memory.
So we really built this web spread with all those things in mind.
And then in, at first we wanted to expose Pondus API, and then we noticed how bad it was for
writing fast data.
It, Pondus API just isn't really neat for this declarative analyzing of what the user wants to
do.
So we just got it off and took the freedom to design an API that makes most sense.
Oh, that's interesting.
I didn't realize that you had started trying to be closer to pandas than you ended up.
Yeah.
Well, it was very short-lived, I must say.
That was it.
Most faithful.
Yeah.
And that's not necessarily saying pandas are bad, I don't think.
It's approaching the problem differently and it has different goals, right?
Yeah.
So maybe we could look at an example of some of the code that we're talking about.
I guess also one of the other differences there is much of this has to do with what you would call,
I guess you refer to them as lazy APIs or streaming APIs, kind of like a generator.
Yeah.
So what you think about a join, for instance, in pandas, if you would write a join and then take
only one to first 100 rows with that result, then it would first do the join.
And then that might produce 1 million or 10 million rows.
And then you take only 100 of them.
And then you have materialized a million, but you take only a fraction of that.
And by having that lazy, you can optimize for the whole query at a time and just see,
yeah, we do this join, but we only need 100 rows.
So that's how we materialize normally.
It gets you more realistic.
That's really cool.
I didn't realize it had so many similarities to databases, but yeah, it makes a lot of sense.
All right.
Let's look at maybe a super simple example you've got on fuller.rs.
What country is RS?
I always love how different countries that often have nothing to do with domain names get grabbed
because they have a cool ending like Libya that was .ly for a while.
You know, it still is, but it was used frequently like bit.ly and stuff.
Do you know what RS is?
I believe it's Serbia.
Serbia.
Okay, cool.
I'm not sure.
Yeah.
Very cool.
All right.
So polar.rs, like polar.rs.
Over here, you've got on the homepage here, the landing page, and then through the documentation
as well, you've got a lot of places where you're like, show me the Rust API or show me the Python API.
People can come and check out the Rust code.
It's a little bit longer because it's, you know, that kind of language,
but it's not terribly more complex.
But maybe talk us through this little example here on the homepage in Python,
just to give people a sense of what the API looks like.
Yeah.
So we start with a scan CSP, which is a lazy read, which is so read CSP.
It tells what you do.
And then it reads the CSP and you get the data frame.
In a scan CSP, we start a computation graph.
called this a lazy frame.
And the lazy frame is actually just, it also, it remembers the steps of the operations you want to
do.
Then it tells it to the boiler, but it looks at this very plan and it will optimize it.
And we'll think of how to execute it.
And we have different engines.
So you can have an engine that's more specialized for data that doesn't fit into memory and engine
that's more specialized for data that does fit differently.
So we start with a scan and then we do a dot filter and we want to use verbs.
Verbs, that's the declarative part.
If on us, we often do indexes or a, and those indexes are a big years in my opinion, because
you can, you can pause in a new file array with booleans, but you can also pause in a new file array
with integers.
So you can do slicing.
You can also pause in a new file, a list of strings and then you do column selection.
So it has three, three functions.
One thing that I find really interesting about pandas is it's so incredible.
And people who are very good with pandas, they can just make it fly.
They can make it really right expressions that are super powerful, but it's not obvious that
you should have been able to do that before you see it.
You know, there's a lot of not quite magic, but stuff that that doesn't seem to come really
straight out of the API directly.
You know, you pass in like some sort of like a boolean expression that involves a
a vector and some other test into the brackets.
Like, wait, how, how did I know I could do that?
Whereas this, your API is a lot more of a fluent API where you say, you know, PD, you'd say PL, PL.scan,
CSV.filter.groupby.aggregate.collect.
And it kind of just flows together.
Does that mean that the editors and IDEs can be more helpful suggesting what happens at each step?
Yes, we are really strict on types.
So we also only return a single type common from a method.
And we only, a dot filter just expects a boolean expression that produces a boolean, not an integer, not a string.
So we want our methods from reading or code.
You should be able to understand what should go in there.
That's really important to me.
It should be unambiguous.
It should be consistent.
And you, your knowledge of the API should expand to different parts of the API.
And that's where I think we're going to talk about this later, but that's where expressions
may be coming over.
So, this portion of Talk Python to me is brought to you by User Interviews.
As a developer, how often do you find yourself talking back to products and services that you use?
Sometimes it may be frustration over how it's working poorly.
And if they just did such and such, it would work better.
And it's easy to do.
Other times it might be delight.
Wow.
They auto-filled that section for me.
How did they even do that?
Wonderful.
Thanks.
While this verbalization might be great to get the thoughts out of your head, did you
know that you can earn money for your feedback on real products?
User Interviews connects researchers with professionals that want to participate in research studies.
There is a high demand for developers to share their opinions on products being created for
developers.
Aside from the extra cash, you'll talk to people building products in your space.
You will not only learn about new tools being created, but you'll also shape the future of
the products that we all use.
It's completely free to sign up and you can apply to your first study in under five minutes.
The average study pays over $60.
However, many studies specifically interested in developers pay several hundreds of dollars
for a one-on-one interview.
Are you ready to earn extra income from sharing your expert opinion?
Head over to talkpython.fm/userinterviews to participate today.
The link is in your podcast player show notes.
Thank you to User Interviews for supporting the show.
I just derailed you a little bit here as you were describing this.
So you start out with scanning a CSV, which is sort of creating and kicking off a data frame
equivalent here.
Lazy frame.
And then you say a dot filter and you give it an expression like this column is greater than five.
Right.
Or some expression that we would understand in Python.
And that's the filter statement, right?
Yeah.
And then we follow a group by argument and then an aggregation where we say, okay, take all columns
and sum them.
And this again is an expression.
And these are really easy expressions.
And then we take this lazy frame and we materialize it into a data frame called
Comecton.
And collect means, okay, all those steps you recorded.
Now you can do your magic, query optimizer, get all the stuff.
And what this will do here, it will recognize that, okay, we've taken the iris.csv, which got
different columns.
And now in this case, it won't.
So if you would have finished with a CEMEC where we only selected two columns, it would
have recognized, oh, we don't need all those columns in the, in the CSV file.
We only take the ones we need.
What it will do, it will push the filter, the predicate down to the scan.
So during the reading of the CSV, we will take this predicate.
We say, okay, where the sample length is larger than five, the rows that don't match the predicate
will not be materialized.
So if you wrap a really large CSV file, if we really, let's say you have a CSV file with
a 10s of gigabytes, but your, your predicate only selects 5% of that.
Then you only materialize 5% of the 10 gigabytes.
Yeah.
So 500 megs instead of 10 gigabytes or something like that, or 200, 200 megs, whatever it is,
quite a bit less.
That's really interesting.
And this is all part of the benefits of what we were talking about with the lazy, lazy frames,
lazy APIs, and, and building up all of the steps before you say go, because in pandas,
you would say, read CSV.
So, okay, it's going to read the CSV.
Now what?
Yes.
Right.
And then you apply your filter if that's the order you want to do it in, and then you group
and then, and so on and so on.
Right.
It's interesting in that it does allow more database like behavior behind the scenes.
Yeah.
And yet, in my opinion, the data frame is, should be seen as a table in a, in a database.
It's, it's the final view of computation.
Like you can see it as a materialized view.
It's, we have some data on this and we want to get it into another table, which we would feed into
our machine learning models or whatever.
And we do a lot of operations on them before we get there.
So I wouldn't see a data frame as a, as a data.
It's not, it's not only a data structure.
It's not only a list or a dictionary.
There are lots of steps before we get into those tables.
And eventually.
Right.
So here's an interesting challenge.
There's a lot of visualization libraries.
There are a lot of other data science libraries that know and expect
and as data frames.
So like, okay, what you do is you send me the pandas data frame here,
or we're going to patch pandas so that if you call this function on the data frame,
it's going to do this thing.
And they may say, Richie, fantastic job you've done here in Polars,
but my stuff is already all built around pandas.
So I'm not going to use this.
Right.
But it's worth pointing out.
There's some cool pandas integration.
Right.
Yeah.
So he said, so Polars doesn't want to do plotting.
I don't think it should be in a different line.
Maybe another length, another library can do it on top of Polars.
Just like it shouldn't be a Polars in my opinion.
But often when you do plotting, you're plotting the number of rows will not be billions.
I mean, there's no plotting engine that can deal with that.
So you will be reducing your, your big data set to something small.
And then you can send it to the plot.
Yeah.
There's hardly a monitor that has enough pixels to show you that anyway.
Right.
So yeah.
Yeah.
You can call to pandas and then we transform our polars data frame to pandas.
And then you can integrate with I could learn with, and we often find that progressively rewriting
things from pandas to polars already is cheaper than keeping it in pandas.
If you do it, if you call from pandas, or let's do a join in polars and then back to pandas,
we probably made up for those double copies.
Pandas does a lot of internal copies.
If you do a reset index copies all data.
There are a lot of internal copies and pandas which aren't listed.
So I wouldn't worry about an explicit copy in the end of your ETL to go to plotting when the data is
already.
Right.
So let's look at the benchmarks because it sounds like to a large degree, even if you do have to do this
conversion in the end, many times, it still might even be quicker.
So you've got some benchmarks over here and you compared, I'm going to need some good vision for
this one.
You compared polars, pandas, Dask, and then two things which are too small for me to read.
Tell us what you compared.
Modding and facts.
Okay.
And for people listening, you go out here and look at these benchmarks,
like linked right off the homepage.
There's like a little tiny purple thing and a whole bunch of really tall bar graphs.
It's got the rest.
Yes.
And the little tiny thing that you can kind of miss if you don't look carefully,
that's the time it takes for polars.
And then all the others are up there in like 60 seconds, a hundred seconds.
And then polars is like a quarter of a second.
So, you know, it's easy to miss it in the graph.
But the quick takeaway here, I think, is there's some fast stuff.
Yeah.
We're often orders of magnitude faster than pandas.
So it's not uncommon to hear it's 1020x times fast, especially if you do write proper
pandas and for apropotos.
It's probably 20 except if we deal with IO as well.
So what we see here are the TPCH benchmarks.
And TPCH is a database query benchmark standard, which this is used by every query engine to show
how fast it is.
And those are really our questions that really, really collect some muscles of the query engine.
So you have joints on several tables, different group buys, different nested group buys, etc.
And yeah, yeah, I really tried to make those other tools faster.
But so in memory, it does then mode in.
But it was really hard to make stuff faster than pandas.
Except for polars.
So I think that's a good idea.
I think that's a good idea.
So I think that's a good idea.
I think that's a good idea.
And then you have problems.
Or we need to do multiprocessing.
Or we need to send those Python objects
to to to another project.
And we copy data, which is slow.
Or we need to do multi-threading
and we're bound by the gill
and we're single thread.
And then there are key defenses.
Yeah, I think there's some
interesting parallels for Dask and Polars.
On these benchmarks, at least,
you're showing much better
than performance than Dask.
I've had Matthew Rocklin on a couple
of times to talk about Dask
and some of the work they're doing
there at Coiled.
and it's very cool.
And one of the things that
I think Dask is interesting for
is allowing you to scale your code out
to multi cores on your machine
or to even distributed grid computing
or process data that doesn't fit in memory
and they can behind the scenes
juggle all that for you.
I feel like Polars kind of
has a different way,
but attempts to solve
some of those problems as well.
Yeah, but Polars has full control
over it over everything.
So it's built from the ground up.
It controls I/O, it controls their own memory,
it controls which strap gets which data.
And in Dask, it goes through,
it takes this other tool
and then parallelizes that.
But it is limited by what this other tool
also is limited by.
But I think, so on a single machine,
it has those challenges.
I think Dask distributed
and does that these challenges.
And I think for distributed,
it worked really well.
Yeah, the interesting part with Dask,
I think, is that it's kind of like Pandas,
but it scales in all these interesting ways.
Across cores,
bigger memory,
but also across machines
and then, you know,
across cores, across machines,
like all that stuff.
I feel like Dask is a little bit,
maybe it's trying to solve
like a little bit bigger computer problem.
Like how can we use a cluster of computers
to answer these questions?
The documentation also says it themselves.
they say that they're probably not faster
than Pandas on the single machine.
So they're more for the large, big data.
Yeah.
But Paulus wants to be
and a lot faster on the single machine,
but also wants to be able to do
out of core processing on the single machine.
So if you,
we don't support all queries yet,
but we want to,
we already do basic device group by sorts,
predicates, element wise operations.
And then we can process,
I process iPhone gigabytes on my laptop.
Matt, that's pretty good.
Your laptop probably doesn't have 500.
No, no, no, no, no.
It's 16 gigs.
Yeah.
Nice.
It's probably actually a value to,
as you develop this product,
to not have too massive of a computer to work on.
If you had a $5,000 workstation,
you know, you might be a little out of touch
with many people using your code.
And, you know, it's so awesome.
Although I think there,
I think others like scaling on a single machine
makes sense for different reasons as well.
I think a lot of people talk about distributed,
but if you think about complexity of distributed,
you need to send data,
shuffle data over the network to other machines.
So there are a lot of people using polars in our discord
who have one terabyte of red and say,
it's cheaper and a lot faster than Spark,
because they can,
well, all this is faster on a single machine.
And one too,
they have a beefy machine with like 120 cores
and they don't have to go over the network to parallelize.
And yeah, so I think times are changing.
I think also scaling out data on a single machine
is getting more worried.
It is.
One of the areas in which it's interesting is GPUs.
Do you have any integration with GPUs or any of those sorts of things?
No.
I'm suggesting that necessarily is even a good idea.
I'm just wondering if it does.
No, I get this question,
but I'm not really convinced I can get memory.
I can get the data fast enough into the memory.
We want to process gigabytes of data.
The challenge already on the CPU is,
is getting the data or cache or memory fast enough
on a CPU piece.
Just, I don't know.
Yeah.
So maybe we could talk really quickly about platforms that it runs on.
You know, I just,
this is the very first show that I'm doing on my M2 Pro processor,
which is fun.
I literally been using for like an hour and a half,
so I don't really have much to say, but it looks neat.
Anyway, you know,
that's very different than an Intel machine,
which is different than a Raspberry Pi,
which is different than, you know,
some version of Linux running on ARM or on AMD.
So where, where do these,
what's the, the reach?
Well, we support it.
We support it.
We don't.
So Poilers also has a lot of like SIMD optimizations.
SIMD starts for a single instruction mock data,
where for instance,
if you do a floating point operation,
it's doing a single floating point at a time,
you can fill in those vector lanes into your CPU,
which can fit eight floating points.
And in a single operation,
can include eight of the five.
And they have eight times the parallelism on a single core.
Those instructions are only activated for
Intel.
So we don't have these instructions activated for ARM,
but we do compile to ARM.
How it forms?
I think it performs far.
Yeah.
But so if the standard machines, right?
macOS, Windows, Linux,
or we're all good to go.
Yeah.
And it ships as a wheel.
So you don't have to have any,
you don't have to have rusty or anything like that.
Hanging around.
Yeah.
Okay.
We also have condo,
but condo is always a bit lagging the eye.
So I could try to answer a bit because we can control this.
Yeah, exactly.
You push it out to IPI and that's what pip sees.
And it's going to go, right?
Pretty much instantly.
I guess it's worth pointing out while we're sitting here is,
um, not that thing I highlighted this.
You do have a whole section in your user guide,
the Polar's book called coming from pandas that actually talks about the differences,
not just how do I do this versus, you know, this operation and pandas versus Polar's,
but it also talks about some of the philosophy,
like this lazy concepts that we've spoken about and a query optimization.
I feel like we covered it pretty well.
Yeah.
Unless there's maybe some other stuff that you want to throw in here really quick,
but I mostly just want to throw this out as resource.
Cause I know many people are coming from pandas and they may be interested in this,
and this is probably a good place to start.
I'll link to it in the show notes.
I think the most controversial one is that we don't have the multi-index.
You don't have anything other than zero base zero one, two, three,
where is it in the array type of.
Yeah.
Well, we can, we will support data structures that make you cooks faster,
like index in a database sense, but it will not involve the,
it will not chase the cement.
Great.
That's important.
Okay.
Yeah.
So I encourage people who are mostly pandas people that come down here and,
you know, look through this.
It's, it's pretty straightforward.
Another thing that I think is interesting and we're talking about maybe is we could touch
a little bit on some of the, how can I, and your user guide, you've got,
how can I work with IO?
How can I work with time series?
How can I work with multiprocessing and so on?
What do you think is good to highlight out of here?
Yeah.
How do you regard it?
It's a bit outdated.
So you can see your own.
So the Francis IO is changing.
All of this writes as its own IO readers.
So we've written our own CSP reader, JSON reader, RK, IPC, Arrow.
And that's all in our control, but for interaction with databases, often a bit more complicated.
Deal with different drivers, different ways.
And currently we do this with connector X, which is really great and allows us to read from a lot
of different databases, but it doesn't allow us to write from databases yet.
And this is happy.
This is not changing.
I want to play a bit why.
So Parler is built upon the Arrow memory specification and the Arrow memory specification
is sort of the standard of how memory or data, our memory for columnar data should look into,
how columnar data should be for, should be represented in memory.
And this is becoming a new standard and Bark is using it, Dremel, Pondos itself.
For instance, if you read a parquet in Pondos, it reads in first into Arrow memory and then
copies that into Pondos memory.
So the Arrow memory specification coming in standard, and this is a way to share data to processes,
to other, also to other libraries within a process without copying data.
We can just swap our pointers if we know that we both support Arrow.
Oh, so Arrow defines basically a, in memory, it looks like this.
Yes.
And if you both agree on that, we can just swap our pointers.
Right.
Because a .NET object, a C++ object and a Python object, those don't look like anything similar
to any of them, right?
In memory.
And yeah.
So, so this is from the Apache Arrow project.
Yeah.
And this is really, really used by a lot of different tools already.
And currently there is coming the ADBC, which is the Apache Arrow database connector,
which will solve all those problems because then we can write,
read arrives from a lot of databases in Arrow and then it will be really fast and very easy for us to do.
So luckily we, we, that's one of those foundations of folders I'm really happy about because supporting Arrow and using Arrow memory gives us a lot of interaction,
which interwoven with other libraries.
Yeah.
That's interesting.
And when you think of Pandas, you know, it's kind of built on top of NumPy as its core foundation,
and it can exchange NumPy arrays with other things that do that.
So Apache Arrow is kind of, kind of your, your base.
Yeah.
Well, it's kind of full circle because Apache Arrow is started by Wes McKinney.
Wes McKinney being known as the creator of Pandas.
And when he got out of Pandas, he thought, okay, the memory representation of NumPy is just not, we should not use it.
And then he was inspired to build Apache Arrow, which made from our master.
Yeah.
So that's how you learn about these projects, right?
This is how you realize, oh, we, we had put this thing in place.
Maybe we'll work better, right?
You, you work on a project for five years and you're like, if I got a chance to start over, but it's too late now.
But every now and then you do actually get a chance to start over.
Yeah.
Interesting.
I didn't realize that Wes was involved with both.
I mean, I knew from Pandas, but I didn't realize it's.
Yeah.
He's a CEO of Holter and with Shiro.
He started Pajero and that's, Pajero is sort of super big, like use everywhere, but sort of middleware.
Like it's end users are developers and not end users are developers who build tools and not developers who use
like that.
Right.
You might not even know that you're using it.
You just use, I just use Polars.
And oh, by the way, it happens to internally be better because of this.
Yeah.
Very cool.
Okay.
Let's see.
We've got a little bit of time left to talk about it.
So for example, this, some of these, how can I let me just touch on a couple that are nice
here.
So you talked about connector X, you talked about the database, but it's like three lines of code to
define a connection string, define a SQL query, and then just
you can just say PL dot read SQL.
Yeah.
And there you go.
You call it data frame or what do you call the thing you get back here?
So reading is always a data frame.
Okay.
Scanning will be a base.
Got it.
Okay.
Is there a scan SQL as well?
We know this might happen in the future.
The challenge is, are we going to push back our optimizations?
Sorry, we write out others query, and then we must translate that into SQL into the SQL we send
to the database.
But that needs to be consistent over different databases.
That's all the rabbit hole we might get into.
I'm not sure it's worth it because you can already do many of these operations in the SQL
query that you're sending over, right?
You have sort of two layers of query engines and optimizers and query plans.
And it's not like you can't add on additional filters, joins, sorts, and so on before it ever
gets back to you.
It would be terrible if someone writes select star from table and then writes the filters in
polars and then the database has sent all those data over the network.
So yeah, ideally we'd be able to push those predicates down into the SQL.
Yeah.
But you know somebody's going to do it because they're more comfortable writing
polar API in Python than they are writing in T-SQL.
Yeah.
You will not.
Yeah.
If it's possible, someone will write it.
It's not optimal.
That's right.
That is right.
Let's see what else can you do here.
So you can, we've already talked about the CSV files and this is the part of that I was talking
about where you've got the toggle to see the rust code and the Python code.
So I think people might appreciate that parquet files.
So parquet files is a more efficient format.
Maybe talk about using parquet files versus CSV and why you might want to get rid of your
CSV and like store these intermediate files and then load them.
But this is really policy here reader.
I really did my best on that.
But you can use parquet or arrow IPC because your data is typed.
There's no ambiguity on reading.
We know type it is.
Right.
Because CSV files, even though it might be representing a date, it's still a string.
Yeah, we need to parse it.
Yeah, it's slow to parse it.
Yeah.
But also we can just, so it parquet interacts really nicely with query optimization.
So we can select just a single column from the file without touching any of the other
columns.
We can read statistics.
So a parquet file can write statistics, which knows, okay, this page has got this maximum
value, this minimum value.
And if you have written a photo square, which says, oh, so I'll give me the result where the
value is larger than this.
And we see that the statistics say it cannot be in this file.
We can just skip the whole column.
We don't have to read.
Yeah.
Oh, interesting.
Wow.
Okay.
So there are a lot of optimizations, which, so the best work is work you don't have to do
and partake a lot.
Exactly.
Or you've done it when you created the file and you never do it again or something like that.
Yeah.
So you've got a read parquet, a scan parquet, I suppose that's the data frame versus lazy frame.
And then you also have the ability to write them.
That's pretty interesting.
JSON, multiple files.
Yeah.
There's just a whole bunch of how do I, how can I rather, a bunch of neat things.
What else would you like to highlight here in the next couple minutes?
The most important thing I want to touch on is the expression API.
So that's a bit, if you go a bit higher.
So you swallow, photos expression.
We got our own chip.
There you go.
One of the goals of the photos API is to keep the API so small, but give you a lot of things
you can do.
And this is where the photos expressions come in.
So photos expressions are expressions of what you want to do, which are
run and parallelized on a query in Japan and can combine them in depth.
So an expression takes a series and produces a series and does the input.
It's the same as the output.
You can combine them.
And as you can see, we can do pretty complicated stuff and you can keep chaining them.
And this is the same.
Like, I would, I'd like to see it.
Transistor Python vocabulary is quite small.
So we have a while we have a loop, we have a variable assignment.
But if you, I think it fits into maybe two, two pieces of paper, but with this, you can write
any program you want with the combination of all those, all those, yeah, this vocabulary.
Yeah.
And that's what we want to do with the photos expressions as well.
So we, you've got a lot of small building blocks, which can be combined into.
Yeah.
So somebody could say, I want to select a column back, but then I don't want the actual
values.
I want the unique ones, a uniqueness.
So if there's duplicate, remove those, and then you can do a dot account.
Then you can add an alias, which gives it a new, which basically defines the column name.
You could read it as a, well, it's not names.
It's a, it's a, it's.
So take column names as, you need names to, you see, but as is the keyword and pipe.
So I'm allowed to use it.
Right.
It means something else.
Yeah.
That's, that's interesting.
Okay.
Yeah.
So people, they use these expressions to do lots of transformations and filterings and, and things
like that.
Yeah.
So these expressions can be used in a select on different places, but the knowledge of expressions
extrapolates to different locations.
So you can do it in a, in a select statement and then you select column.
Then you select this expression and you get a result.
But you can also do this in a group by aggregation.
And then the same logic applies.
It runs on the same engine and we make sure everything is possible.
And this is really powerful because because it's so expressive, people don't have to use
custom apply with Lambda because when you use a Lambda, it's like box to us.
It will be slow because it's Python and we don't know what happened.
So a Lambda is, it will be slow.
It will kill parallelization because it builds.
But yeah.
A Lambda is three times a day.
Right.
It gets in the way of a lot of your optimizations and a lot of your, your speed ups there.
That's why we want to make this expression API very complete.
So you, you don't need that as much.
Yeah.
So people are wanting to get this, get seriously into this.
They should check out chapter three expressions, right?
And just go through there.
And probably, especially, you know, sort of browse through the Python examples that they
can see where, go back and see what they need to learn more about.
But it's a very interesting API.
The speed is very compelling thing.
Yeah.
I think it's a cool project.
And like I said, how many people we got here?
13,000 people using it already.
So that's, that's a big community.
Yeah.
So if you're interested in project, we have a discord where, where you can chat with us
and ask questions and see how you can best do things.
Pretty active there.
Cool.
The discord's linked right off the homepage.
So that's awesome.
People can find it there.
Contributions.
People want to make contributions.
I'm sure you're willing to accept PRs and other feedback.
Or you put in a really large PR, please first open an issue with a, with a, with a,
to start with discussion of business, this contribution is welcome.
And we also have a few getting started.
Good for the contributors.
Okay.
Yes.
You've, you've tagged or labeled some of the issues as look here, if you want to get,
get into this.
Yeah.
I must say, I think we're an interesting project to contribute to because we're,
you can, it's not, not everything is set in stone.
So there are still places where you can play.
I'm not sure.
And there, there's still interesting work to be done.
It's not completely 100% polished and finalized.
Yeah.
On the periphery.
Yeah.
Very cool.
Let's wrap it up with a comment from the audience here.
Ajit says, excellent content guys.
It certainly helps me kickstart my journey from pandas to pollers.
Awesome.
Glad, glad to help.
I'm sure it will.
Many people do that.
So Richie, let's close it out with final call action.
People are interested in this project.
They want to start playing and learning pollers.
maybe try it out on some other code that is and is at the moment.
What do they do?
I'd recommend if you have a new project, just start in pollers.
Because you can also rewrite some comments, but the most fun experience will just start a new
project in pollers.
And because then you can really enjoy what pollers offers.
The only expression API, learn how you use it declaratively.
And yeah, we'll be, then it will be most fun.
Absolutely.
Sounds great.
And like we did point out, it has the to and from hand as data frame.
So you can work on a section of your code and still have it consistent, right?
With, with other parts that have to be.
Yeah.
You can progressively rewrite some performance heavy parts.
Or I also think supporters have really strict on the, on the schema and the types.
It's also, if you write any ETL, you will be really happy to do that.
But also because you can check the scheme of lazy frame before executing it.
But you know, the apps core running the query and if the data comes in and it doesn't
oblige to this schema, you can fail fast.
Instead of having strange outtakes.
Oh, that's interesting because you definitely don't want zero when you expected something else
because it could parse or other weird, you know, whatever, right?
Yeah.
So this was my, for missing data and polar doesn't change the schema.
Yeah.
So polar is great.
The schema is defined by the operations and the data and not by their values in data.
So you can definitely check.
Got it.
Excellent.
All right.
Well, congratulations on a cool project.
I'm glad we got to share with everybody.
Thanks for coming on the show.
Bye.
You bet.
Bye.
This has been another episode of Talk Python to me.
Thank you to our sponsors.
Be sure to check out what they're offering.
It really helps support the show.
Type I is here to take on the challenge of rapidly transforming a bare algorithm in Python
into a full-fledged decision support system for end users.
Get started with Type I core and GUI for free at talkpython.fm/Typy.
T-A-I-P-Y.
Earn extra income from sharing your software development opinion at user interviews.
Head over to talkpython.fm/userinterviews to participate today.
Want to level up your Python?
We have one of the largest catalogs of Python video courses over at Talk Python.
Our content ranges from true beginners to deeply advanced topics like memory and async.
And best of all, there's not a subscription in sight.
Check it out for yourself at training.talkpython.fm.
Be sure to subscribe to the show, open your favorite podcast app, and search for Python.
We should be right at the top.
You can also find the iTunes feed at /itunes, the Google Play feed at /play, and the direct
RSS feed at /rss on talkpython.fm.
We're live streaming most of our recordings these days.
If you want to be part of the show and have your comments featured on the air,
be sure to subscribe to our YouTube channel at talkpython.fm/youtube.
This is your host, Michael Kennedy.
Thanks so much for listening.
I really appreciate it.
Now get out there and write some Python code.
I'll see you next time.
Bye.