How do you build and maintain a complex suite of Python packages?
Of course, you want to put them on PyPI.
The best format there is as a wheel.
This means that when developers use your code,
it comes straight down and requires no local tooling to install and use.
But if you have complex dependencies, such as C or Fortran,
then you have a big challenge.
How do you automatically compile and test against Linux, macOS,
that's Intel and Apple Silicon, Windows, 32 and 64-bit,
and so on?
That's the problem solved by CI Buildwheel.
On this episode, you'll meet Henry Schreiner.
He's developing tools for the next era of the Large Hadron Collider
and is an admin of Scikit-HEP.
Of course, CI Buildwheel is central to that process.
This is Talk Python to Me, episode 338, recorded October 14th, 2021.
Welcome to Talk Python to Me, a weekly podcast on Python.
This is your host, Michael Kennedy.
Follow me on Twitter, where I'm @mkennedy,
and keep up with the show and listen to past episodes at talkpython.fm.
And follow the show on Twitter via at Talk Python.
We've started streaming most of our episodes live on YouTube.
Subscribe to our YouTube channel over at talkpython.fm/youtube
to get notified about upcoming shows and be part of that episode.
Hey there, I have some exciting news to share before we jump into the interview.
We have a new course over at Talk Python.
HTMLX plus Flask, modern Python web apps, hold the JavaScript.
HTMLX is one of the hottest properties in web development today,
and for good reason.
You might even remember all the stuff we talked about with Carson Gross back on episode 321.
HTMLX, along with the libraries and techniques we introduced in our new course,
will have you writing the best Python web apps you've ever written.
Clean, fast, and interactive, all without that front-end overhead.
If you're a Python web developer that has wanted to build more dynamic, interactive apps,
but don't want to or can't write a significant portion of your app in rich front-end JavaScript
frameworks, you'll absolutely love HTMLX.
Check it out over at talkpython.fm/HTMLX,
or just click the link in your podcast player's show notes.
Now let's get on to that interview.
Henry, welcome to Talk Python to Me.
Thank you.
Yeah, it's great to have you here.
I'm always fascinated with cutting-edge physics
with maybe both ends of physics, right?
I'm really fascinated with astrophysics and the super large,
and then also the very, very small.
And we're going to probably tend a little bit towards the smaller,
high-energy things this time around,
but so much fun to talk about this stuff and how it intersects Python.
Some of the smallest things you can measure
and some of the largest amounts of data you can get out.
Yeah, the data story is actually really, really crazy,
and we're going to talk a bit about that.
So neat, so much stuff.
We used to think that atoms were as small as things could get, right?
I remember learning that in elementary school.
There are these things called atoms.
They combine to form compounds and stuff,
and that's as small as it gets.
And yeah, not so much, right?
Yeah, that was sort of what atom was supposed to mean.
Exactly, the smallest bit, but nope.
But that name got used up, so there we are.
All right, well, before we get into all that stuff, though,
let's start with your story.
How did you get into programming in Python?
Well, I started with a little bit of programming
that my dad taught me.
He was a physicist.
And I remember it was C++ and sort of taught the way you would teach Java,
you know, all objects and classes.
Yeah.
Just a little bit.
And then when I started at college,
then I wanted to take classes,
and I took a couple classes again in C++.
But I just really loved objects and classes.
Unfortunately, the courses didn't actually cover that much,
but the book did.
So I really got into that.
And then for Python, actually, right when I started college,
I started using this program called Blender.
Oh, yeah.
Blender.
I've heard of Blender.
It's like a 3D animation tool, like Maya or something like that, right?
And it's very Python-friendly, right?
Yes.
It has a built-in Python interpreter.
So I knew it had this built-in language called Python,
so that made me really want to learn Python.
And then when I went to an REU,
a research experience for undergraduates
at Northwestern University in Chicago.
And when I was there,
we had this cluster that we were working on.
This was in solid-state physics, material physics.
And we would launch these simulations on the cluster.
And so I started using Python,
and I was able to write a program that would go out,
and it would create a bunch of threads,
and it would watch all of the cluster,
all the nodes in the cluster.
And as soon as one became available, it would take it.
So I could just,
my simulation would just take the entire cluster.
After a few hours, I would have everything.
So at the end of that,
everybody hated me,
and everybody wanted my scripts.
Exactly.
They're like,
this is horrible.
I can't believe you did that to me,
but I'll completely forgive you
if you just give it to me and only to me,
because I need that power.
Yeah, that's fantastic.
How neat.
So I think that is one of the cool things about Python, right,
is that it has this quick prototyping approachability.
They're like,
I'm just going to take over a huge hardware, right?
Like a huge cluster of servers,
but it itself doesn't have to be like intense programming.
It can be like this elegant little bit of code, right?
You can sort of do things that,
normally I think the programming gets in the way more,
but Python tends to stay out.
It looks more like pseudocode.
So you can do more and learn more,
and eventually you can go do it in C++ or something.
Yeah.
Yeah, absolutely.
Great way to start.
Or maybe not.
Sometimes you do need to go do it in some other language,
and sometimes you don't.
I think the stuff at CERN and LHC has an interesting exchange between C++
and maybe some more Python and whatnot,
so that'll be fun to talk about.
Yeah.
We've been C++ originally,
but Python is really showing up in a lot more places,
and there's been a lot of movement in that direction.
And there have been some really interesting things that have come out.
A lot of interesting things have come out of the LHC,
computing-wise as well as physics.
Awesome.
Yeah.
As a computing bit of infrastructure,
there's a ton going on there.
And as physics,
it's kind of the center of the particle physics world, right?
So it's got those two parallel things generating,
all sorts of cool stuff.
I want to go back to just really quickly to,
you know,
you talked about your dad teaching a little programming.
If people are out there and they're the dad,
they want to teach their kids a little bit of programming,
I want to give a shout out to CodeCombat.com.
Such a cool place.
My daughter just yesterday was like,
hey, dad, I want to do a little Python.
Remember that game that taught me programming?
Like, yeah, yeah, sure.
So she's like,
she logged in and started playing
and basically solve a dungeon interactively by writing Python.
And it's such an approachable way,
but it's not the like draggy, dropy, fake stuff.
You write real Python,
which I think is cool to introduce kids that way.
So anyway, shout out to them.
I had them on the podcast before,
but it's cool to see kids like taking to it in that way, right?
Whereas you say it like,
you could write a terminal app.
They're like, I don't want to do that.
But solve a dungeon.
Yeah, they could do that.
Yeah.
I've actually played with a couple of those.
They're actually really fun just to play.
Yeah, they are.
Exactly.
I did like 40 dungeons along with my daughter.
It was very cool.
How about now?
What do you do now?
So I work in a lot of different areas
and I jump around a lot.
So I do a mix of coding.
I do some work on websites
because they just needed maintenance
and somehow I got volunteered.
And some writing.
So less coding than I would like,
but I definitely do get to do it,
which is fun.
Yeah.
And this is at CERN or at your university
or where is this?
So now I'm at Princeton University
and I'm part of a local group
of RSEs, research software engineers.
And I'm also part of Iris Hep,
which we'll talk about a little bit.
But that's sort of a very spread out group.
Some of us are at CERN,
a few are in some other places,
a few in some Fermilab.
And energy physicists are just used
to working remote.
The pandemic wasn't that big
of a change for us.
We were already doing
all our meetings remote.
We just eventually changed
from video to Zoom.
But other than that,
it was pretty much the same.
Exactly.
It was real similar for me as well.
That's interesting.
Fermilab, that's in Chicago,
outside Chicago, right?
Yes.
Is that still going?
I got the sense that
that was shutting down.
They're big in neutrino physics.
So they do a lot of neutrino things there.
And then they're also very active
just in the particle physics space.
So you may be at Fermilab,
but working on CERN data.
I see.
Okay.
Interesting.
Yeah.
I got to tour that place a little bit
and it's a really neat place.
It is.
CERN's a neat place too.
I would love to tour CERN,
but it wasn't 20 minutes down the street
from where I happened to be.
So I didn't make it there.
Sadly, I hope to get back there someday.
All right.
Well, let's talk about
sort of the scikit-hep side of things
and how you got into
maintaining all of these packages.
So you found yourself in this place
where you're working on tools
that help other people build packages
for the physicists and data scientists
and so on, right?
So where'd that all start?
So with maintenance itself,
the first thing I started maintaining
was a package called Plumbum back in 2015.
And at that point,
I was starting to submit some PRs
and the author came to me and said,
I would like to have somebody do the releases.
I need a release manager.
I don't have time.
And I said, sure, I'd be happy to do it.
And it was exciting for me
because it was the first package
or like real package I got to join.
And so I think on the page,
it might even still have the original news item
when it says, welcome to me.
But-
Nice.
So that was the first thing I started maintaining.
And then I was working on a IG physics tool
called Goofit when I became a postdoc.
And I worked on sort of really renovating that.
It started out as a code written by physicists.
And I worked on making it actually installable
and packaged nicely
and worked with a student to add Python bindings to it,
things like that.
And as part of that,
I wrote a C++ package, CLI 11.
It was just a first package I actually wrote
and then maintained.
And it's actually in C++.
And that was written for Goofit,
but now it's a fairly,
I think it's done pretty well on its own.
Nice.
What's that one do?
Microsoft Terminal uses it.
Yeah.
Microsoft Terminal uses it?
Mm-hmm.
Oh, nice.
Yeah, I'm a big fan of Microsoft Terminal.
I've for a while now
kind of shied away from working on Windows
because the terminal experience
has been really crummy.
You know, the cmd.exe command prompt style
is just like,
oh, why is it so painful?
And people who work in that all day,
they might not see it as painful.
But if you get to work in something
like a macOS terminal
or even to not quite the same degree,
but still in like a Linux one,
then all of a sudden,
yeah, it kind of gets there.
But I'm kind of warming up to it again
Windows Terminal.
Yeah, the Xterm is one of the reasons
I use,
I really moved to Mac
because I loved Xterm.
And then Windows Terminal is amazing.
Now it's a great,
great team working on it,
including the fact that they used my parser.
But it's actually quite nice.
The only problem I have in Windows 10
is it's really hard to get the thing
to show up instead of cmd prompt.
Yeah.
But Windows 11,
I think it's supposed to be the only one.
Yeah.
I definitely think it's included now,
which is great.
So CLI 11,
this is a C++ 11 command line parser,
right?
Like click or arg parse
or something like that,
but for C++, right?
Yes.
It was designed off of
the Pumbum command line parser.
Pumbum is sort of a toolkit
and it has several different things.
I wish those things had been pulled out
because I think on their own,
they might have
maybe even been popular on their own.
it has a really nice parser,
but it was sort of designed off of that
and off click.
It has some similarities
to the both of those.
Yeah.
I think probably that's a challenge.
I mean,
we're going to get into CiteGithub
with a whole bunch
of these different packages,
but finding the right granularity
of what is a self-contained unit
that you want to share with people
or versus things like
pulling out a command line parser
rather than some other library, right?
This is a careful balance.
It's a bit challenging.
I think in Python,
there's a really strong emphasis
to having the individual
separate pieces and packages,
especially in Python,
partially because it has
a really good packaging system.
And being able to take things,
have just pieces
and be able to swap out one
that you don't like
is really, really nice.
And that's one of the things
we'll talk about the PyPA as well.
And that's one of the things
that they focus on
is small individual packages
that each do a job
versus all-in-one poetry.
Yeah.
Well, you'll have to
do some checking
or some fact-checking,
balancing,
modernizing for me.
I did professional C++ development
for a couple of years
and I really enjoyed it
until there were better options.
And then I'm like,
why am I still doing this?
I would go work on those.
But one of the things
that struck me
as a big difference
to that world
is basically
the number of libraries
you use,
the granularity
of the libraries you use,
you know,
the relative acceptance
of things like pip
and the ease
of using another library,
right?
In C++,
you've got the header
and you've got the linked file
and you've got the DLL
and there's like
all sorts of stuff
that can like
get out of sync
and go crazy
and like make weird crashes.
Your app just goes away
and that's not great.
Is that still true?
I feel like that
that difference
is one of the things
that allows for people
to make these smaller
composable pieces
in Python.
I think that has
a lot to do with it.
What has happened
in C++
is there's sort of
a rise of a lot
of header-only libraries
and these libraries
are a lot easier
to just drop
into your project
because all you do
is you put in the headers
and there's no,
you don't have to deal
with a lot of the
original issues.
So a lot of these
small standalone libraries
are header-only
and one of the next
things that I picked up
as a maintainer
was Pybind 11,
which,
and I've sort of
been in that space
sort of between C++
and Python
for quite a bit.
I kind of like being
in that area,
joining the two.
I get a sense
from listening to the things
that you've worked on
previously
and things like this
that you're interested
in connecting
and enabling,
like piecing together,
like here's my script
that's going to pull together
the compute
on this cluster
or here's this library
that pulls together
Python and C++
and so on.
Yes,
making different things
work together
and combining things
like C++ and Python
or combining different
packages in Python
and piecing together
a solution.
I think that's one
of Python's strengths
versus something like
MATLAB.
I spent quite a bit
of time in MATLAB
early on
and got to move
a lot of stuff
over to Python.
Right on,
that's awesome.
It was really nice.
We didn't have to have
a license
and things like that.
I know,
it's so expensive
and then you get the,
what are they called,
toolkits,
the add-on toolkits
and they're like,
each toolkit
is the price
of another $1,000 a year
or $2,000 a year.
It's ridiculous.
So I know of CFFI,
which is a way
for Python and C
to get clicked together
in a simple way.
How's Pybind 11
fit into that?
This is seamless
interoperability
between C++11
and Python.
How are they different?
So CFFI,
I teach like
a little short course
where I can go through
some of the different
binding tools
and it usually ends
with me saying
Pybind 11 is my favorite.
Yeah, cool.
Give us an overview
of what the options are
and stuff.
CFFI is closer
to C types.
It's more of,
it's focused on C
versus C++
and it's actually
the one I've used
the least.
I was just helping,
we're just talking
with the CFFI developer
but I've used it
the least of those
but I think it
basically parses
your C headers
and then sort of
automates a lot
of what you would
have to manually
do with C types
or you have to
specify what
symbol you want to call
and what the arguments
are and what the return
type is and if one
of those things is wrong
you get a seg fault
and that sort of thing.
Whereas Pybind 11,
this is about building
modules, extension modules.
So, and it's,
and it's,
the interesting thing
about this
is that it's written
in pure C++.
The other tools
out there,
so Cython can do this,
it's not what
it was designed for
but it immediately
became popular
for doing this
because Cython
turned code,
Python,
Python-like code
is a new language
into,
it transpiled it
into C
or C++
that had a toggle
you could change,
has a toggle
you can change
and then
when you're there
you can now call
C or C++
but it's extremely verbose
and you repeat yourself
and you have to learn
another language.
This weird combined
Python thing
and just thinking
in Cython is difficult
because you have to think
about well am I
in Python
or am I in Cython
that can,
that's going to be
bound to Python
or am I in Cython
that's just going
straight to C,
C++
or am I just in C++
or C
but I've actually used it.
It's a lot of layers there,
yeah.
But Pybind 11
is just C++
and it's just,
it's basically,
it's like the,
C API
for Python
but a C++ API.
It's quite,
it's quite natural
and you don't have
to learn a new language.
It uses some fairly
advanced C++
but that's it.
You're learning
something useful anyway.
Right.
So do you do
some sort of like
template type thing
and then say
I'm going to expose
this class to Python
or something like that
and then it figures out,
does it write
the Python code
or what is it?
It's writing the,
build like .so files
or what do you do here?
It,
it compiles
into the C API calls
and then that
would compile
into a .so
and there's no
separate step
like Cython
or Swig
or these,
or these other tools
because it's just
C++.
You compile it
like you do
any other C++
but it's actually
internally using
the CPython API
or PyPy's
wrapper for it
and the language
looks a lot like Python
but the names
are similar.
You just do a def
to define a function
and you give it
the name
and then you just
pass it the pointer
to the,
the underlying thing.
It can figure out
things like types
and stuff like that
for you.
Give it a doc string
if you want.
Give the arguments
names.
You can make it
as Pythonic as you want.
It's verbose
but it's not overly verbose.
Yeah,
that's really neat.
Nice.
And for people
who haven't used
those kind of outputs,
basically,
it's just import
module name
whether it's a
.py file
or it's a
.so file.
PyTorch
if you've used
.py
if you've used
any of those things
you have,
you've been importing
some PyBind11 code.
So let's talk
a little bit
about
Scikit-Hep.
This is
one of the projects
that it has
a lot of these
packages
inside of it
and your
library
CBuild
Wheel
is
one of the things
that is used
to maintain
and build
a lot of
those packages
because I'm sure
they have a lot
of interesting
and oddball
dependencies,
right?
I mean,
C++ is kind of
standard,
but there's probably
others as well,
right?
It is.
So one thing
that is kind of
somewhat unique
to HEP
is that we are
very heavily invested
in C++.
So it's usually
either you're going
to see Python
or you're going
to see some sort
of C++
package of some sort.
I mean,
it could be
varies in size there,
but it's mostly
C++ or Python.
We really
haven't used
other languages
much for the past
early 90s or so.
Is that inertia
or is that
by choice?
You know,
why is that?
I think it's
partially the community
is a fairly
cohesive community.
We're really
used to
sort of working
together.
The experiments
themselves
are often,
you know,
might be a thousand
or several thousand
physicists working
on a single experiment.
And we have been
fairly good about
sort of meeting
together and
sort of deciding
the direction
that we want
to go in
and it's
sort of sticking
to that.
So for C++,
it was heavily
root,
which is a
giant C++
framework.
And it's got
everything in it.
And that was
C++ and that's
what everybody
used.
So root is the
library.
If I was going
to write code
that would run
and interact
with like the
grid computing
or the data
access and all
that kind of
stuff at LHC,
I would use
this root library
if I was doing
that in C++,
right?
Yes.
You might be
using interpreted
C++,
which is something
we invented.
Oh, okay.
This is interesting.
Is this something
people can use?
Oh, yes.
We actually,
so Cint
was the original
interpreter
and then it got
replaced by
Cling,
which is built
on the LLVM.
And I think
recently it was
merged to
mainline LLVM
as Clang
Repl,
I think it's
called,
but it's sort
of a lightweight
version.
Yeah, it's
a C++
interpreter.
You can actually
get Zeus
Cling,
which I think
Quantstack has,
but they package
it as well,
I think,
Zeus Cling.
Okay, yeah,
very interesting.
It's not,
C++ really wasn't
designed for a
notebook though.
It does work,
but you can't
rerun a cell
often because
you can't
redefine things.
Python is just
really natural
in a notebook
and C++
is not.
Yeah, especially
if you change
the type,
you compile it
as an int
and then you're
like, ah,
that should be
a string.
Yeah, that's
not going to
be a string.
It's compiled.
Yeah, interesting.
So it seems to
me like the
community at
CERN has
decided, look,
we need some
low-level stuff
and there's some
crazy low-level
things that
happen over there.
People can check
out a video,
maybe I'll mention
a little bit
later.
But for that
use, they've
sort of gravitated
towards C and
then for the
other aspects,
it sounds like
Python is what
everyone agreed
to.
It's like, hey,
we want to
visualize this,
we want to do
some notebook
stuff, we want
to piece things
together,
something like
that, right?
It's certainly
moving that way.
They definitely
have sort of
agreed that
Python should
be a first-class
language and
join C++.
That was decided
a few years ago.
And I think
that's been a
great step in
the right direction
because what was
happening, people
were coming in
with Python
knowledge.
They wanted to
use Pandas.
I came in
that way as
well.
Pandas and
Numba and all
these tools were
really, really
nice.
And we were
basically just
having to write
them all
ourselves in
C++.
It has a
data frame,
but why not
just use Python,
which is what
people know
anyway?
Panda exists.
There's a ton
of people already
doing the work
maintaining it
for us.
Root literally
has a string
class.
Literally,
they do
everything.
So the idea,
and this is
sort of the
idea behind
Scikit-HEP
was to build
this collection
of packages
that would just
fill in the
missing pieces,
the things that
energy physicists
were used to
and needed.
And some of
them are general
and were just
gaps in the
data science
ecosystem,
and some
things are
very specific,
high energy
physics.
Scikit-HEP
actually sort
of originated
as a single
package.
It sort of
looked like
root red at
first,
and it was
invented by
someone called
Eduardo
Rodriguez,
who was
actually in
my office
at CERN,
and we're
office mates.
But he did
something I think
really brilliant
when he did
this,
and that is
he created
an organization
called Scikit-HEP
around it,
and then he
went out and
spoke with
people and
got some of
the other
Python packages
that existed
at the time
to join
Scikit-HEP,
moved them
over and
started building
a collection of
some of the
most popular
Python packages
at the time.
And I thought
that was great,
and I really
wanted Scikit-HEP
to become a
collection of
tools,
separate tools,
and for the
Scikit-HEP
package to just
be sort of a
meta package that
just grabbed all
the rest.
And that's
actually kind of
where it is now.
Right.
I can pip install
Scikit-HEP.
Is that right?
You can,
and mostly,
other than a few
little things that
are still in there
that never got
pulled out,
that will mostly
just install our
most popular,
maybe 15 or so
packages,
2015 of our
most popular
packages.
Yeah, so it
probably doesn't
really do anything
other than,
say, it depends
on those packages
or something like
that, right?
And then by
virtue of installing
it, it'll grab
all the pieces.
Yeah, yeah,
that's a really
cool idea and I
like it.
So maybe one of
the things I
thought would be
fun is to go
through some of
the packages there
to give people a
sense of what's
in here.
Some of these are
pretty particular
and I don't think
would find broad
use outside of
CERN.
For example,
Conda Forge
Root.
It sounds like
that's about
building root so
I can install it
as a dependency
or something like
that, right?
building root is
horrible and
you actually now
can get it as
part of a
Conda package
which is just
way better than
anything that was
available for
attaching it to a
specific version of
Python because it
has to compile
against a very
specific version of
Python but that's
what it does.
So unless you want
something in root
then that's very
HEP specific.
Yeah, absolutely.
Some more general
ones, probably
our first, briefly
Mitch, our very
first package that I
think was really
popular among
energy physicists
that we actually
produced was
Uproot which was
just a pure Python
package so you
didn't have to
install it that
read root files.
Again, very
specific for
somebody who was
in high energy
physics but you
could actually read
a root file and
get your data without
installing root and
that was a game
changer.
So now you can
actually install
root slightly easier
but normally it's a
multi-hour compile and
it's got.
gotten better but
it's still a bit of
a beast to compile
especially for Python.
Yeah, that does
sound like a beast.
Oh my gosh.
And now you can
just read in your
files.
Basically, Jim
Povarsky basically
just taught Python
to understand the
decompile the root
file structure and
actually can write
right now too but
originally reading.
But that actually
was really...
So this is like if
I want to do, if I
want to create a
notebook and maybe
visualize some of
the data but I
don't really need
access to anything
else, I shouldn't
depend on this
beast of almost its
own operating system
type of thing.
Yeah, we were
very close to being
able to use all the
data science tools in
Python, pandas, things
like that.
For most data
worked fine.
You just had to get
the data.
And I mean, I've
done this too where
I had one special
install of Python and
root together that I'd
worked several hours on
and it sat somewhere
and I would convert
data with it.
I'd move it to HDF5
and then I would do
all the rest of the
analysis in Python that
didn't have it because
then I could do
virtual environments and all
that read that HDF5
format, right?
Mm-hmm.
Yeah.
Right, okay.
The first package we
had that was really
popular on its own was
Awkward Array.
Yeah.
Awkward Arrays.
I definitely heard
about this one, yeah.
Yeah, that was
originally part of
Upproot, sort of grew
out of Upproot.
When you're reading
root files, you end up
with these jagged
arrays.
So that's an array that
is not rectangular.
So at least one
dimension is jagged.
it depends on the data
and this shows up in
all sorts of places
and not just particle
collisions or obviously
shows up lots of places
in particle collisions
like how many hits
got triggered in the
detector.
That's a variable length
list.
How many tracks are in
an event?
You know, that's a
variable length list
and can be a variable
length list of
structured data.
And to store that
compactly the same way
you'd use numpy was
one thing, but you can
arrow and there's some
other, there's some
other things that do
this, but Awkward Array
also gives you numpy
like indexing and data
manipulation.
And that was the sort
of the breakthrough
thing here.
It's like numpy.
The original one was
built on top of numpy.
The new one actually
has some pybind11
compiled bits and pieces,
but it makes working
with that really well.
In fact, Jim Povarsky
has now got a grant
to expand this to,
I don't remember the
number of different
disciplines that he's
working with, but lots
of different areas,
genomics and things
like that have all
use cases and he's
adding things like
complex numbers and
things that weren't
originally needed by
energy physicists, but
make it widely.
Almost an evangelism,
like dev evangelism
type of role, right?
Go talk to the other
groups and say, hey,
we think you should be
using this.
What is it missing for
you to really love it?
Something like that,
right?
How interesting.
Yeah.
So, yeah.
Yeah.
So looking at the
Awkward Array page
here says for a
similar problem,
10 million times
larger than this
example given above,
which one above is
not totally simple.
So that's pretty
crazy.
It says Awkward
Array, the one
liner takes 4.6
seconds to run and
uses 2 gigs of
memory.
The equivalent
Python list in
dictionaries takes over
two minutes and uses
10 times as much
memory, 22 gigs.
So, yeah, that's a
pretty appealing value
proposition there.
Yeah.
And it supports
Numba.
Jim works very closely
with the Numba team
and really is one of
the experts on the
Numba internals.
So, yeah, it has full
Numba support now and
he's working on adding
Dask.
He's working with
Anaconda on this grant
and then working with
adding GPU support.
Very cool.
Maybe not everyone out
there knows what
Numba is.
Maybe give us a quick
elevator pitch on
Numba.
Yeah.
I hear it makes
Python code fast,
right?
Yeah, it's a just-in-time
compiler and it takes
Python.
It takes Python.
It actually takes the
bytecode and then it
basically takes that
back to something or it
parses the bytes code
and turns it into LLVM.
So it works a lot like
Julia except instead of
a new language, it's
actually reading Python
bytecode, which is
challenging because the
Python bytecode is not
something that stays
static or is supposed to
be a public detail.
Yeah, there's no
public promises about
consistency of bytecode
across versions because
they play with that all
the time to try to
speed up things and
they add bytecodes and
they try to do little
optimizations.
Yeah, so every Python
release breaks Numba.
So they have to, they
just know the next
Python release will not
support Numba and it
usually takes a month
or two.
But it's very
impressive though.
It's the speedups,
you do get full sort of
C type speedups for
something that looks
just like Python.
It compiles really fast
for a small problem and
it's as fast as anything
else you can do.
I've tried lots of
these various
programming problems and
you just about can't
beat Numba.
It actually knows what
your architecture is
since it's just in time
compiling.
So you have to do
which is an advantage
over say like C, right?
It can look exactly at
what your platform is and
your machine architecture
and say we're going to
target, you know, I see
your CPU supports this
special vectorized thing
or whatever and it's
going to build that in,
right?
and then what sort of Jim
does with Awkward and
we've done with some
other things with Vector
does this too.
You can control what
Python turns into what
LLVM constructs any
Python turns into because
you can control that
compile phase.
That's incredibly
powerful because you can
say and it doesn't have
to be the same thing but
obviously you want it to
behave the same way.
You can say if you see
this structure, this is
what it turns into.
in LLVM machine code
which then gets compiled
or machine language
which then gets compiled
into your native machine
language.
Interesting.
Assembling.
So if you have like a
certain data structure
that you know can be
well represented or gets
packed up in a certain
way to be super efficient
you can control that?
Yeah, you can say that
well this like this
operation on this data
structure, this is what
this is what it should do
and then that turns into
LLVM and maybe it can
get vectorized or things
like that for you.
Yeah, yeah.
That's super neat.
Another package in the
list that I got to talk
about because just the
name and the graphic is
fantastic is a gassed.
What is a gassed?
It's got like this
the scream.
I forgot who was the
artist of that but the
scream sort of look as
part of the logo is good.
About half of the logos
come from Jim and I did
about half and he did
about half and then
use other around or from
the individual package
authors.
A gassed was so this is
sort of part of the
histogramming area which
is where sort of the
area I work in,
psychic hub.
But Jim actually wrote
a gassed and the idea
was that it would
convert between
histogram representations.
I think it came up
because Jim got tired of
writing histogram libraries.
I think he's written at
least five.
Yeah, one of the things
I got the sense of by
looking through all the
psychic hub stuff,
there's a lot of
histogram stuff happening
over there.
histograms are sort of the
area that I was in and it
ended up coming in in
several pieces.
But I think one of the
important things was
actually, and I think a
gassed may not really
matter, it may get
archived at some point
because instead of
translating between
different representations of
histograms in memory,
what you can do is define
a static typing protocol
and it can be checked by
mypy that describes
what a object needs
to be called a
histogram.
And so I've defined that
as a package called
UHI, Universal Histogram
Interface.
And anything that
implements UHI, it can
be fully checked by
mypy, will then be able
to take any object
from any library that
implements UHI.
And so all the libraries
we have that produce
histograms, so uproot,
when it reads a root
histogram or hist and
boost histogram, when they
produce histograms, they
don't need to depend on
each other.
They don't even depend on
UHI, that's just a
static dependency for
mypy time.
And then they can be
plotted in MPLHEP or
they can be printed to
the terminal with
histoprint and there's
no dependencies there.
One doesn't need the
other.
And that's sort of
making a gassed somewhat
unneeded because now
it really doesn't matter.
You don't have to
convert between two
because they both just
work.
They work on the same
underlying structure
basically, right?
They work through the
same interface.
Right.
Yeah.
So a gassed is a way to
work with different
histogramming libraries
that kind of is the
intermediary of that.
It's like an abstraction
layer on that.
Okay.
What are some other ones?
Yeah.
What are some other ones
we should kind of give a
shout out to?
we talked about GUFIT, which
is there.
It's an affiliated package.
It's not part of
scikit-hep, but it has.
So we developed this idea
of an affiliated package
for sure things that
didn't need to be moved
in, but had at least one
scikit-hep developer
working with them.
At least that's my
definition.
I was never able to
actually get the rest to
agree to exactly that
definition, but that's my
working definition.
And so that's why
pybind11 gets listed
there.
It's an affiliated package
because we share a
developer, me, with the
pybind11 library.
And we sort of have a
say in how that is
developed.
And most importantly, if
we have somebody come
into scikit-hep, we want
them to use pybind11
over the other tools
because that one we have
a lot of experience with.
Very cool.
Another one I thought
was interesting is
hep units.
So this idea of
representing units like
the standard units,
they're not enough for
us.
We have our own kind of
things like molarity and
stuff, but also luminosity
and other stuff, right?
Yeah.
Different experience
service can differ a bit.
So there's a sort of a
standard that got built up
for units.
And so this just sort of
puts that together and
has, and the unit that
we've sort of decided on
this should be the
standard unit that's one
and the rest are
different scalers.
It's a very tiny little
library.
It was the first one to be
fully statically
typed because it was
tiny.
That's easy to do.
It was like, because
mypy infers constants,
there was like two
functions or something
and then it was done.
Yeah.
Probably a lot of floats.
Mm-hmm.
So, but, and that's,
that's sort of what it is.
So you can use that if,
and the idea is that the
rest of the libraries
will, will adhere to
that system of units.
So then if you use this
and then use that, the
values it gives you,
then you can have a nice
human, human readable
units and be sure of your
units.
Yeah.
That's really neat.
Have you heard of pint?
Are you familiar with this
one?
Yes, I love pint.
Oh gosh, I think
pint is interesting as well.
It takes the, the types
through and I use pint
some, but it actually
gives you a quantity out
or a numpy quantity.
Whereas the happiness just
stays out of the way and
it's a way to be more
clear in your code, but it's
not enforced.
Pint is enforced, which I
like enforcing, but it's
also can slow down.
You can't, these are not
actual real numbers
anymore.
So you pay it.
Yeah.
So it's going to add a ton
of overhead, right?
But pint's interesting
because you can do things
like three times meter
plus four times centimeter
and you end up with
3.04 meters.
Yeah.
Those are actually real
quantities.
They're actually a different
object, which is the good
thing about it, but it's
also the reason that then
it's not going to talk to
say a C library that
expects a regular number
or something as well.
Sure.
Okay.
Maybe one or two more and
then we'll probably be out
of time for these.
What else should people
maybe pay attention to that
they could generally find
useful over here?
You mentioned vector.
It's a little bit newer,
but it's certainly for
general physics.
I think it's useful because
it's a library for 2D,
3D and relativistic
vectors.
And there aren't really,
it's a very common sort of
learning example you see,
but there aren't really
very many libraries that do
this, that actually have,
if you want to take the
magnitude of a vector in 3D
space, there just isn't a
nice library for that.
So we wrote vector to do
that.
And vectors is supported by
awkward.
It has an awkward backend.
It has a number backend,
a numpy backend, and then
plain object backend.
Eventually we might work on
more.
And it even has a number
awkward.
So you can, you can use a,
a vector inside an awkward
array inside a number jet
compiled loop and still take
magnitudes and do stuff like
that.
That's really cool.
That integration there.
Yeah.
vectors because we have a lot
of those in physics.
Sure.
And you can, you can do things
like ask if one vector is
close to another vector and
things like that, even in
different, it looks like a
one in polar coordinates and
one in, you know, a
Cartesian or something like
that.
It has different unit systems
and it can actually, it
actually stores the vector in
that.
So you don't waste memory or
something.
If that's, that's the
representation you have, that
was a feature from a route
that we wanted to make sure we,
we got.
And it also has sort of the
idea of, of momentums too
and stuff for the, for the
relativistic stuff.
We end up with a lot of that.
And then maybe just mention the,
since we mentioned the
histogramming stuff and that's
the area, that's the ones that I
really work on.
The ones I specifically work on
that are general purpose.
Boost histogram is a wrapper for
the C++ boost histogram library.
Boost is the sort of the big
C++ library, just one step below
the standard library.
And right at the time I was
starting at, at Princeton, the,
I met the author of boost
histogramming who's from
physics and he was in the
process, I believe, of getting
this accepted into boost.
And it got accepted after that.
But one of the things that he
decided to do is pull out his
initial Python bindings that
were written in boost Python,
which is actually very similar to
pybind11, but requires boost
instead of not requiring anything.
But the design is intentionally
very similar.
And so I proposed I would, I
would work on boost histogram
and write these, this, the
Python bindings for it inside
scikit-hep.
And that would be sort of the
main project I started on when I
started at Princeton.
And that's, you know, that's
what I did.
Boost histogram is a extremely
powerful histogramming library.
So it's a histogram as an
object rather than like a
NumPy, you can, there's a
histogram function and you give
it an array and then it spits a
couple of arrays back out at
you.
You are now, you now have to
manage these.
They don't have any special
meaning.
Whereas boost histogram,
histograms really are much more
natural as an object, just like a
data frame is more natural as an
object where you tie that
information together.
A histogram's really natural that
way, where you still have the
information about what the data
actually was on the axes.
If you have labels, you want to
keep those attached to those, to
that, to that data.
And you may need to fill again,
which is one of the main things
that,
energy physicists really wanted
because we tend to fill
histograms and then keep
filling them or rebinning them
or doing operations on them.
And you can do all those very
naturally.
And boost histograms, the
actual, the C++ wrapper in
PyBind 11 and a lot of, and,
I actually got involved in CI
BuildWell because of boost
histogram, because I, one of the
things I wanted to just make sure
it worked everywhere.
And it obviously requires C++.
It requires compilation.
Um, and then hist is a nice
wrapper on top of that that just
makes it a lot more friendly to,
to use because the original
boost histogram author wants to
keep this, Hans Dubinsky wants
to keep this quite, pure and
clean.
So hist is a more, the more
natural.
And even if you're not in hep,
I think that's still the more
natural one to use.
Yeah.
Gold.plot and plot.
Right, right.
There's a lot of people who do,
who use histograms across all
sorts of disciplines.
So that would definitely be one of
those that is generally useful.
All right.
So I think that brings us to CI
build wheel.
Uh, let's, let's talk a bit
about that.
And I mean, maybe the place to
start here is, you want
to wheels, right?
The, the first sentence
describing it as Python wheels
are great building them across
Mac, Linux, windows, and other
multiple versions of Python.
Not so much.
So no description.
Yeah, exactly.
Well, wheels are good.
There's times when there are no
wheels and things install slower.
They might not install at all.
It's generally a bad thing if
you don't have a wheel, but,
they're, they're not easy
to make.
Right.
So tell us what is a wheel and
then let's talk about why maybe
building them across all these
platforms and this, cross
product along with like versions
of Python and whatnot.
It's a mess.
When you distribute Python, you
have several options.
The most common one and most
packages have at least an S
dist, which is just basically a
tar ball of the, of the source.
Right.
When you pip install it,
it basically, you're missing
some things or adding some
things, but right.
Otherwise it's mostly
unzips.
Yeah.
It unzips your source and puts
it somewhere.
Python will find it.
And then that's that.
Yeah.
So it runs your build system.
So set up tools traditionally
that's become a lot more
powerful recently, but, it
has to run the build system to
figure out what, what do you do
with it?
This is just a bunch of files.
and then it puts it together in a
particular structure in your, in
your, on your computer.
And so a wheel was a package that
was already, everything was
already in place.
So it's already in a particular
structure.
It knows, knows the structure and
all Python has to do for a pure
Python wheel, one that does not
have any, binary pieces in
it.
It just grabs the contents inside and
dumps them following a specific,
set of rules into places into your,
um, site packages.
Right.
So then you now have something
installed.
There's no setup.py in your wheel.
There's no pyproject.tomol.
There's those sorts of things are
not in the wheel.
The wheel's already there.
It can't run arbitrary code.
Yeah, exactly.
That was one of the points I was
going to make is one of the
things that can be scary about
installing packages is just by
virtue of installing them, you're
running arbitrary code because
often that is execute, you know,
Python space, set up py space,
you know, install or something
like that.
And like, whatever that thing
does, that's what happens when
you pip install.
Right.
But not with wheels, as you said,
it comes down in a binary blob
and just like, boom, here it is.
Obviously the thinking is we have
this package delivered to a
million computers.
Why do we need to have every
million computer run all the
steps?
Why don't we just run it once and
then go here?
And then also that saves you a
ton of time.
Right.
like I just installed micro
whiskey and it took, I don't
know, 30 seconds, 45 seconds to
install because it didn't have a
wheel.
So it sat there and it just
grinded away compiling it, you
know?
Yeah.
So there's two possibilities.
A pure Python package, a wheel is
still superior because of the not
running arbitrary code.
Pip will actually go ahead and
compile all your PYC files.
Your, that goes ahead and makes
the bytecode for all those.
If it's a wheel, if it's an S, if
it's a tarball, it doesn't do
that.
If it doesn't pass through the
wheel stage anyway.
And then when, every time you open
the file, then it's going to have
the first time it's going to have
to make that, that bytecode.
So it'll be a little slower the
first time you open it.
There's, there's a variety of, of
reasons.
I think it's pythonwheels.com,
something like that.
That describes why you should use
wheels.
That's maybe that's not it, but
I think it is.
Yes.
Python wheels.
So they have like a list of
advantages there, but.
Yeah.
I also have a little like
checklist.
It says, how are we doing for the
top 360 packages?
And apparently 342 of them have
wheels.
And it shows you for your popular
packages, which ones like click does,
but future doesn't, for example, and so
on.
So.
Future's been there for a long time.
Yeah.
But, but yeah, so wheels are really
good.
And they actually replaced an older
mechanism that was trying to do
something somewhat similar called
eggs, but I avoid talking about
those.
I don't really understand.
Let it live in the past.
Wheels also are a great way if you
have compile and compile that
happens.
So if you compile some code as part of
your, as part of your build, then
that of course is much slower.
If you have the, if you just have
the example.
Yeah.
It's like it was doing GCC or something
forever.
And if you don't have a compiler, it
won't even work.
Right.
Exactly.
You have to have some setup, at least
a little setup.
You have to have a compiler setup at
the very moment.
Right.
How many windows users have seen
cannot find vcvars.bat?
Right.
Like what is this?
I don't want this.
In windows you have to be in a, in
the environment or you have to have
the, the right script sourced.
Yes.
So wheels had also can contain binary
components like .so's and things.
And they have a tag as part of their
name.
They have a very special naming scheme
for wheels and the tag is stored in
the wheel too.
And they can tell you what Python version
they're good for, what platform they can,
are supported on.
They have a build number and then they
have a, the Python's actually in two
pieces.
There's the, the AVI and the interface.
Yeah.
You can see there's some huge long name
that with a bunch of underscores
separating it.
And basically, when you try to
install it, sorry, go ahead.
I was saying it's also one of the
reasons that names are normalized.
There's no difference between a dash
and underscore.
It's because that special wheel name
has dashes in it.
So the package name at that point in
the, in the file name has to be
underscores.
Yeah.
And so basically when you pip install,
it says it, it builds up that, that
name and says, do you have this as a
binary?
Give it to me, right?
Something like this.
Yeah.
It knows how to pick out the, it
looks for the right one.
If it finds a binary, it'll just
download it depending slightly on the
system and how new your pip is.
Right.
And this is one of the main
innovations ideas or philosophies
behind Conda and, Anaconda,
right?
It's like, let's just take that and
make sure that we build all of these
things in a really clear way.
And then sort of package up the,
the testing and come compilation
and distribute, distributing all
that together.
Right.
Yes.
This is very similar to this.
This came, I think, I'm pretty sure it
came after Conda.
I think where they were still in
eggs when Conda was invented and
then sort of building up wheels was
challenging.
Building a wheel was, was challenging.
That's, that's, yeah, build wheel has
really changed that.
Um, if you want a pure Python, it's
really easy.
You use today, you should be using the
build tool, which I'm also, I'm a
maintainer of that as well.
Um, but build just builds an S
dist for you or it builds a wheel.
And so you would say something like
Python, set up PY, B dist or
something like that.
And then boom, I shouldn't be doing
that anymore.
Please don't.
But that is how you do it.
Yeah.
How would I do it?
Tell me the right way.
The best.
Um, well you could do Python, or
pip install build and then Python dash
M build, and that will build both an
S dist and a wheel and it'll build the
wheel from the S dist.
Uh, if you use pip X, which I would
recommend, then you can just say pip X
run build and you don't have to do
anything that'll, that'll download build
into a virtual environment for you.
It'll do it.
And then it eventually it'll throw away
the virtual environment, after a
week.
Interesting.
Okay.
So we could just use the build.
We should be using the build.
You should be using the build tool.
It's for an S dist.
There's a big, benefit to this.
And that is it will, it will, use
your piproject.toml.
And if you say you require numpy, then
it will go, like you're using the
numpy headers, the C headers, then it
will go and it will, when it's, when
it's building S dist, it will make the
pep, 517 virtual environment.
It'll install numpy, anything that's in
your, your, your, requires in
your piproject.toml.
And then it will run the setup.py,
inside that environment.
So you can now import numpy directly in
there.
Um, and it'll work even when you're
building an S dist.
If you do Python S dist or Python,
setup.py stuff, you can't do that because
you're literally running Python, giving it
setup.py import numpy.
now it's broken.
Right.
It, it, nothing, nothing triggers that, that
call to the, pyproject.toml to see what, what you need, for a wheel.
The best way to do it is with pip.
Uh, or the original way to do it was with pip wheel.
Um, because pip has to be able to build wheels in order to, install things.
The, that got added to pip before build existed.
Um, but now the best way to do it would be with build wheel.
And that's actually, it's doing the right thing.
It's actually trying to build the wheel you want.
Whereas pip wheel is actually just building a wheelhouse.
So if you depend on numpy and numpy doesn't have wheels, which they did
better with Python 3.10.
So I'm not going to complain about, about numpy for Python 3.10, but for 3.9, they didn't
have wheels for a while.
So it'll build the wheels there and it'll build your wheels and it'll dump them all in the
wheelhouse or whatever the output is.
So you'll get, you'll be building numpy wheels, which you definitely don't want to try to upload.
Yeah.
Yeah, definitely not.
All right.
Well, that's, that's really cool.
And I definitely learned something.
I will start using build instead of, doing it the other way.
You can now delete your setup.py too.
Yeah.
That's the big thing, right?
So you don't have to run that kind of stuff, right?
Yeah.
The, they're trying to move away from the any commands to setup.py because you don't
even need one anymore.
And, you can't control that, that environment.
It's, it's very much an internal detail.
Like wrapping up this segment of the conversation, we want a wheel because that's best.
It installs without requiring the compiler tools on our system.
It installs faster.
It's built just for our platform.
The challenge is when you become a maintainer, you got to solve this, this matrix of different
Python versions that are supported and different platforms.
Like for example, there's macOS Intel, there's macOS, M1, Apple Silicon.
There's multiple versions of windows.
There's different versions of Linux, right?
Like arm Linux versus AMD 64 Linux.
Yeah.
And now muscle, muscle Linux versus the other Linux varieties.
Yeah.
So one of the challenges with a wheel, is making it distributable.
So if you just go out and you build a wheel and then you try to give it to someone else,
it may not work.
Um, certainly on, Linux, if you try to pretty much, if you do that, that it just won't
work.
Um, because the systems are going to be different.
Uh, on macOS, it'll only work on the version you compiled it on and not anything, older.
And, you'll even see people trying to compile on, on macOS 10.14 because they're,
they want their wheels to work as in many places as you want.
Well, you can use the latest one.
There's ways to fix that.
Well, exactly.
It's fine.
The jankiest, like I've got a Mac mini from nine, from 2009.
We're building on that thing.
Cause it will work for most people.
Right.
I think that's how they actually build the official Python binaries.
Interesting.
I'm not sure.
But then Apple went in like last year around this time, they threw a big spanner in the
works and said, you know what?
We're going to completely switch to arm and our own silicon.
And, you got to compile for something different now.
Yeah.
And cross compiling has always been a challenge.
Um, yeah.
And then windows is actually the easiest of all of them.
You're most likely on windows to be able to compile something that you can give to someone
else.
But, the rest of the, that is one of the things that Microsoft's been really pretty good
at is backwards compatibility.
I get holds them back in other ways, but yeah, typically you can run an app from 20 years ago
and it'll still run.
Yeah.
And there are a few caveats, but not, not many compared, at least compared to the other
systems.
Uh, Apple's really good, but you do have to, you do have to understand how to, you do have
to set your minimum version and you have to get a Python that had that
minimum version set when it was compiled.
But if you do that, it works really well.
So what I actually did with what actually started with in scikit-heb, I had this, I had, I was
building boost histogram, which needed to be able to run anywhere.
That was something I absolutely wanted.
It had to be pip install boost histogram and it just worked no matter what.
And also we had several other compiled packages at the time, several we had inherited.
Um, and, I mean, you is, was compiled and that was quite popular.
We had a couple of specific ones and we had a couple, a couple more that ended up being,
becoming interested in, in that.
fact during this sort of period is when awkward started compiling pieces.
And so what I started with was building my own system to do this.
It was called, Azure wheel helpers, which, was, as you can guess by the name, Azure
was basically a set of Azure, DevOps scripts.
It was right after Azure had come out.
And I wrote a series of blog posts on this and described the exact process.
Um, and sort of the things I'd found out about how you build a compatible wheel.
Um, on macOS, you have to make sure you get the most compatible, CPython from, from
python.org itself.
You, you know, can't use, you can't use brew or something like that because those are
going to be compiled for whatever system they were targeting.
And on Linux, you have to, you have to run the mini Linux system and you should run audit
wheel and actually in Mac, you should run develop.
develop, bill wheel, although I might be getting him, I think it's the bill wheel.
Um, so there's this, this series of things that you have to do.
And I started maintaining this, this multi hundred line set of scripts to do this.
And, and I was also being limited by Azure at the time.
They didn't have these, all the templates and stuff they have now.
So everything had to be managed through get subtree because it couldn't be a separate
repository.
And, and I, and then when, Jim started working awkward, he went and just rewrote the
whole thing to, cause it thought it.
He wanted it to look simpler for him and took a couple of things out that were needed and
suddenly made it two separate things.
Now I had to, had to help maintain that.
So when Python 3.8 or whatever it was came out, now I had, I had a completely different
set of changes I had to make for that one.
And it was really just not, it was not working out.
Uh, it was not very easy to maintain.
And I was watching, CI build wheel.
I, and it was this package.
It was a Python package that would, would do this.
And it didn't matter what CI system you were on because it was written in Python.
Um, and it, it followed nice, Python principles for good package design and had unit tests and all that sort of stuff.
So it looked really good.
There were a couple of things that was missing.
I came in, I added, I made PRs for the things that I'd, I'd come up with that it didn't have.
And they got accepted.
And, there was a shared maintainer between PI bind 11 and CI build wheel as well.
I think that's one of the reasons that I sort of had heard about it.
It was really watching it.
And I finally decided just to make the switch.
And, I did at some point, a little later, I actually became a maintainer of CI build wheel.
But, I think I started doing the switch before I made it really easy once I was a maintainer to say, oh, this is a package that, you know, we have some control over.
It's okay.
Let's just.
Right.
Your package is a choice to depend upon this.
Cause we have a say.
It just took out all of that, that maintenance.
And now, depend about does all the maintenance for us.
It does the pin moves for the pin and see a build wheel.
And that's it.
Nice.
So if I want to accomplish, if I'm a package developer owner, and I want to share that package with everybody,
we've already determined we would ideally want to have a wheel, but getting that wheel is hard.
So CI build wheel will let you integrate it as the name indicates into your continuous integration.
And one of those steps of CI could be build the wheel, right?
But it pretty almost, it reduces it down to pretty much that, that there's a step in your CI that says, you know, run CI build wheel.
And then CI build wheel is designed to really integrate nicely with the build matrix.
So, you could, in, for a fairly simple package or for many packages, you can really just do Mac, Windows, and Linux have the same job.
I can get up actions.
It's easy to do the same job.
Um, and then I'll see, I build wheel.
And that's about it.
It just goes through all the different versions of Python that are supported.
It goes, it just goes through and makes, a wheel for each.
And, in fact, it even has one feature that was really nice that, I'd always struggled with a bit is testing.
So if you give it a test command, it will even take your, your package, it will install it in a new environment.
That's not, you know, in a different directory.
That's not related to your build at all.
And make sure it works and passes whatever test you give it.
And,
We'll do that across the platforms.
We'll do like a macOS test and a Windows test.
Yeah.
For each.
So CI build wheel really just sees the platform it's sitting on because it's inside the build matrix.
And so it's run, run for each.
And, yeah, it does for each, each one.
It, it, will run that, that test.
And the most, the simplest test is just echo.
And that will just make sure it installs.
Cause it won't try to install your wheel unless, there's something in that test command.
Even that's useful sometimes.
Even that's broken sometimes because of NumPy not supporting one of those things in that matrix.
Yeah.
It can't install the dependencies.
So that step fails or something.
So, it says it currently supports GitHub actions, Azure pipelines,
which I don't know how long those are going to be two separate things.
Maybe they'll always be separate, but Microsoft owning GitHub be like,
they're saying do stuff in Azure pipelines.
And then they're kind of moving.
Like, yeah, I think they're similar.
The runners are the same.
They actually have the same environments.
Um, so I think they'll exist just as two different interfaces probably.
And Azure is not so tied to GitHub and it has more of an enterprise type.
Yeah, for sure.
It definitely has a different focus.
It was just a rewrite and a better rewrite in most cases of it.
I got to learn.
Yeah.
They, I think GitHub actions came second.
All right.
So then Travis CI app, Bayer circle CI and get lab CI, at least all of those, right?
At least those are the, those are the ones we test on.
And then, it runs locally.
Uh, there are some limitations to running it locally.
Uh, if it's, if you target Linux and you can, any, any system that has Docker in target
Linux, you can just ask to build Linux wheels.
You can actually run it from like my Mac or from windows.
I assume from a Windows machine.
I tried Windows with Docker and, Windows.
It does install to a, a standard location, C colon backslash CI build wheel.
But other than that, it's safe to run it there.
And macOS, it will install to your macOS system.
It's installed system versions of Python.
So that's something we haven't solved yet.
Might be able to someday.
Um, so it's not a good idea unless you really are okay with installing every version of Python
that ever existed into your, into your system.
Uh, maybe get a VM of your Mac.
The Python.org Python.
Yeah.
I mean, it's somewhat safe.
Um, if you're on Windows, you could use, a Windows subsystem for Linux, WSL as well.
In addition to Docker, I suspect.
Although I haven't tried that.
Mini Linux has to run, you could, I'm sure as long as you can launch Docker, the thing
that you have to be able to do is launch Docker because you have to use the, mini Linux
Docker images or you should use that or derivative of that.
There's lots of rules to exactly what can be in the environment and things like that.
And PyPA maintains that.
One thing that also helps is that we have the main, mini Linux maintainer is also a CI
build wheel maintainer.
So it's one reason that those things tend that, they fit well together.
Features tend to match and come out at the same time.
Like, like Musa Linux, which is a big, big thing recently.
It's not actually in a released version of CI build wheel yet.
What is Musa Linux?
So a normal Linux, is based on Glib C and that's actually what controls.
It's one of two things that controls mini Linux.
So if, can you download the binary wheel or do you have to build?
If you have a old version of pip that will, they had to teach pip about each
version of mini Linux.
That was a mess.
So they eventually switched to not to a standard numbering system.
That is your G lib C number.
And now pip come.
Doesn't.
And the current pip will be able to install a future mini Linux as long as your systems.
But, that was a big problem.
So pip nine can only install mini Linux one.
It can't install mini Linux.
And even if your G lib C is fine for it.
So the real, the other thing is the G lib C version and, mini Linux one was based on
Sinto S five, grid hat five.
Um, mini Linux 2010 was Sinto S six mini Linux 2014 was Sinto S seven.
And then now they switched to Debian because of the, Sinto S sort of switching to the
stream model.
Um, so mini Linux two 14 or two 24 is G lib C 2.24.
And that's Debian eight or something like that.
And so, but that's G lib C based.
There are, distributions that are not G lib C based, most notably Alpine, very used
Alpine.
Um, it's this tiny, tiny little Docker image.
It's really fun distribution to use if you're on Docker, but, it actually sounds fun to
install, but I've never tried it without Docker, but, it's these five megabyte Docker
wheels or Docker is Docker doesn't do wheels.
Docker images, Docker images.
Yeah.
But, that doesn't use G lib C.
That uses muscle.
And so muscle Linux will run on Alpine.
Okay.
Got it.
So if you're building for the platform Alpine and similar ones, right.
So anything.
Yeah.
And you said, I can run this locally as well.
Um, I know I would use it in CI cause I'm trying, I've got that matrix of all the versions
of CPython and pipe pipe, P Y P Y, and then all the platforms that I want to check as many
of those boxes as possible to put wheels in it.
Right.
Um, yeah.
Suppose I'm on my Mac and I want to make use of this to fill in, maybe do some testing,
at least on some of these columns.
Like, how do I do that?
What's the benefit there?
Well, I can tell you the case where it happened.
Um, so we were shipping, CMake and the psychic build organization ran out of Travis credits
and, they were being built.
We hadn't switched them over to being emulated builds on, GitHub actions yet.
And it just ran out.
We couldn't, we couldn't build them.
And one of them had been missed and we also weren't waiting to upload.
So we had uploaded everything, but we had one, one set or maybe, maybe it was all of the emulated
builds.
I think it was one set that didn't, didn't work.
And so we wanted to go ahead and upload those, those missing wheels.
And I tried, but, I couldn't actually get emulation.
Docker Q, Q, Q, Q, E, M, U emulation.
What?
I couldn't get that working on my Mac.
So, the mini links maintainer used his Linux machine and he, yeah, had Q, Q, E, M, U emulation
on it and he built the emulated images a few hours, but he just built them locally and then sent
and then uploaded, filled in the missing wheels.
So if, if I'm, maintaining a package, I'm, I got some package I'm putting on pipe AI and I want
to test it.
Does it make sense to do it locally or does it just make sense to put it on a summit?
Some CI system.
Um, for C and build wheel, usually I do some local testing, but I'm also developing C and build
wheel, but, you know, usually it's probably fine to do this in your, just in your CI and
usually don't want to run the full, full thing.
Every time usually you have your regular unit tests, but C and build wheel is going to be
a lot slower because it's going through and it's making each set of wheels and launching
Docker images and things like that.
Um, and it's installing Python each time, for macOS and windows.
So, usually unless if you have a fairly quick build, I've seen some people just run C and
people just run CI build wheel as part of their test suite.
Um, but usually you just run it, say right before release.
Maybe I usually do it once before the release and then I'm the release.
Right.
Exactly.
Okay.
That makes sense.
Cause it's a pretty heavyweight type of operation.
So when I look at all these different platforms, I see macOS Intel, macOS, Apple Silicon, different,
businesses of windows.
And then I think about CI systems, you know, what CI systems can I use that support all these
things?
Like does GitHub actions support both versions of macOS, for example, plus windows.
GitHub actions is by far our most popular, platform.
It switched very quickly.
It used to be Travis.
Travis was a challenge cause they didn't do windows very well.
They still don't do windows very well.
Um, and it's a challenge for us because we actually can't run our macOS, tests on them anymore.
Because once we joined the pipe PA, the billing became an issue and we just basically just
lost, macOS running for it.
Um, but, circle, I think, Azure and GitHub actions, I think they do all three.
Um, and you can always flip things up.
I always do Travis for the Linux and then app fair for windows.
You can do it that way.
One of the big things that I had developed for CI build wheel was the, pipe project
dot Tomo or any Tomo configuration, usually that, configuration for CI build wheel.
That way you can get your CI build wheel configuration out of your, your, YAML files.
That way it works locally.
Um, which is one of the main, one of the things I was after, but also you can just do it and
then run on several different systems.
Like you might like the fact that Travis, Travis is, I think the only one that does the,
native strange architectures.
You have to emulate it other places, which is a lot slower, five times slower or something.
Yeah.
So kind of split that up, get the, the definition and then create maybe multiple
CI, jobs.
Your CI scripts are really simple.
Yeah.
Very cool.
The example script is just a few lines.
It doesn't, it does not take much to do this comparing.
Oh yeah.
Hundreds of lines it used to take.
Yeah, sure.
And I didn't even scroll down here.
You've got a nice grid on github.com/IPA slash CI build wheel that shows on GitHub actions,
which is supported on Azure pipelines.
What's supported.
It's not right.
Circle CI doesn't do this.
No, but yeah.
App there, Travis, Azure and GitHub do.
Where does the macOS, but we can't test it.
Theoretically, it does it.
Gotcha.
And then, yeah, I wonder about the, the M1, the Apple Silicon arm versions versus the Intel versions.
I don't know how, how well that's permeated into the world yet.
Um, but the fact that they have Mac at all is kind of impressive.
Nobody has an M1 runner yet.
Um, there are a few places I think now that you can purchase time on one, but no runners.
I mean, last I checked GitHub actions, you couldn't even write yourself on the M1.
Um, that may be, that may have changed.
I don't know.
That was a while back.
Yeah.
I mean, there are some crazy, places out there.
I think there's one called Mac mini Colo.
I think that's what it's called.
Let me see if that's, yeah, I think that's it.
Yeah.
So you can get, you can go to these places like Mac mini Colo and get, get a whole bunch
of Mac minis and put them into this crazy data center.
But you know, that's not the same as I upload a text file into GitHub that says run on Azure
on GitHub actions.
And then that's the end of it.
Right.
You probably got to set up your whole, like some whole build system into a set of minis.
And like, that doesn't sound very practical for most people.
Ideally with what you could do is, I mean, you just need one mini and then you set up
a GitHub actions, hosted runner, a locally hosted runner.
Um, and other systems in that too, get, get, get lab CI was big on that.
Um, you can, you can do anything on get lab CI.
We just haven't tested that because they don't have those publicly.
But, if you, if you have your own, you can do that.
I know, I know somebody who does this with basically with root and runs the, has a Mac mini
and runs the M1 builds on that.
But, you could do that.
And I have a Mac mini and the lead developer of CI build.
Will also has a, Mac mini or the M1.
He has an M1 of something.
I don't know.
I have a Mac mini.
Mine is Mac.
That's what I'm, talking to you right now on.
It's a fantastic little machine.
Yeah.
It's, it's very impressive.
I love the way the boost histogram.
It was fast.
I have a 16 inch, almost maxed out, MacBook and the Mac mini M1.
It was faster on boost histogram than this thing.
Wow.
Yeah.
I have a maxed out six, 15 inches, a little bit older, a couple of years, but I just don't
touch that thing unless I literally need it as a laptop because I want to be somewhere else.
But yeah, I'm definitely not drawn to it.
These, so you could probably set up one of these minis for 700 bucks and then tie it up.
But that's again, not as easy as, you know, just clicking the public free option that works,
but still it's, it's within the realm of possibility.
Yeah.
And Apple has actually helped out several, like, I know, homebrew and a few others they've
helped out with, by giving them either Mac minis or some, some, something that they
could build with.
So they, I believe, brew actually builds, homebrew actually builds on him on real
M1s.
I know it does because they're, the builds are super fast.
I remember that.
Like it builds root like 20 minutes, the root recipe, because I maintain that.
And the normal one takes about an hour because running on multiple cores, but, it's like
three times faster.
It's done in 20 minutes.
I just thought something was wrong when I first saw that.
That's it.
How could it be done?
Something broke.
What broke?
Interesting.
All right, Henry, we're getting really short on time, a little bit over, but it's been a
fun conversation.
How about you give us a look at the future?
Where are things going?
Um, with all the stuff.
Next thing I'm interested in, in, being involved with is scikit build, which,
is a, a package that currently sort of augments setup tools, but hopefully will eventually
sort of replace setup tools as your, as the thing that you, build with.
And it will call out to CMake.
So you basically just, you basically write a CMake, file.
And this could wrap an existing package, or maybe you need some of the other things that
CMake has.
And this will then let you build that as a regular Python package.
In fact, recently somebody, sort of put together, see, I build a wheel.
Yeah.
It's like it built in the CMake example and, and built, LLVM and pulled out just the
claim format tool and made wheels out of that.
And now you can just do pip install clang format.
It's one to two megabytes.
It works on all systems, including Apple Silicon and things.
I just tried it on Apple Silicon yesterday and it's a pip install.
Now you can claim format C++ code.
And that's just, you know, mind blowing.
You can add it to pre-commit.
The pre-commit CI, it runs in two.
I mean, I'd been fighting for about a week to reduce the, the, size of the claim format
recipe from 600 megabytes to just under the 250.
That was the maximum for pre-commit.CI.
And then you can now pip install under about a megabyte for Linux, that, that sort of thing.
And I, I think that would be really, that would be a really great thing to, to work on.
It's been around since 2014, but it needs some, some serious work.
And so I'm currently actually working on writing a grant to, to try to get funded, to just work
on, basically the scikit build, scikit build system and looking for interesting science
use cases that would be interested in, adapting, or switching, existing build
system over or adapting to it.
Um, or taking something that has never been available from Python and making it available.
And yes, root, root might be one.
Scikit build package.
I'm looking for wide variety.
Yeah.
How neat.
Scikit build package is fundamentally just the glue between set of tools, Python module and
CMake.
Yeah.
So it's a real way to take some of these things that were based on CMake and sort of expose
them to Python.
Yeah.
So you can just, have a CMake package that does all the CMake things well, you know,
like finding, finding different libraries and, and that I'm a big CMake person.
But how many of you physically uses it very heavily.
Most C++ does.
It's about 60%.
I think of all field systems are, are CMake based now.
Um, going from GitWare's numbers, but they make CMake.
But, it's, I think it's a, it's very powerful.
It can be used for things like that.
And, will really open up a much easier C++, more natural in C++ and C and, and
Fortran and things like that.
And CUDA then is currently available.
Setup tools is, disto tools is going away in Python 3.12.
Setup tools is not really designed to build C++ packages or packages.
It was really just a hack on top of disto tools, which happened to be build just Python itself.
So, well, scikit build sounds like the perfect tool to apply to the science space because
there's so many of these weird compiled things that are challenging to, you know, install and
deploy and share and so on.
So making that easier.
Sounds good.
All right.
Well, I think we're probably going to need, need to leave it there just for the sake of time,
but it's been, it's been awesome to talk about all the internals of supporting scikit
hep and people should check out CI build wheel.
It looks like it, you know, if you're maintaining a package either publicly or just for internal
for your organization, it looks like it'd be a big help.
Yeah.
If it's got binary, any sort of binary build in it.
Yes.
Yeah, absolutely.
If not build is fine.
Yeah.
Right.
And I learned about build, which is good to know.
All right.
So before you get out of your Henry, let me ask you the two final questions.
Uh, you're going to write some code, I mean, like Python code, what editor would you use?
Depends on how much it'll either be VI.
If it's a very small amount.
Um, if it's a really large project that let's say it takes several days, then I'll use,
PyCharm.
And then I've really started using VS Code quite a bit.
And that's sort of expanding to fill in all the middle ground and kind of eating in on both
of the other, both of the edges.
Yeah.
There's some interesting stuff going there.
Good choice.
But all with the VI, mode or plugins added, of course.
And then, notable PyPI package.
I mean, we probably talked about 20 already.
If you want to just give a shout out to one of those, that's fine.
Or if you got a new idea.
I'm going to go with one that's, unlike, I might not get mentioned, but I, I, I'm really
excited by it.
The development of it is, the, I think the developer is quite new, but what he's actually
done as far as the actual package has been, been nice.
It needs, it needs some, some nice touches.
But, and that is plot text, yellow T T E X T.
And I'm really excited about that because it makes these, the actual plots it makes are
really, really nice.
And they're plotted to the terminal and, it can integrate with rich.
Uh, and of course, I'm, I'm interested in it because I want to integrate it with tech.
Uh, I want to see it integrated with a textual, I think a textual app that combines this with,
uh, file browsers and things like that.
It'd be incredible.
Yeah.
So you can do things like the terminal, for example.
Yeah.
So you could like cruise around your files, use your, your root IO integration, pull
these things up here and, you know, put the plot right on the screen.
Right.
But in the terminal.
Okay.
Yeah.
This is really cool.
I had no idea.
And this is based on rich.
You say.
Uh, it can integrate with rich.
It integrates with rich.
Okay.
Got it.
Yeah.
So as soon as I saw it, I started trying to make sure the two people were talking to each
other.
Will and the person who is developing this.
Yeah, exactly.
All right.
These things work together.
That's very cool.
They seem like they should, right?
They're in the same general zone.
Yeah.
And they do now.
The, you had, there had to be some communication back and forth as far as what size the plots
were.
Right.
This should, this should work in it.
A good recommendation.
Uh, definitely one I had not learned about.
So I'm sure people will enjoy that.
All right, Henry, final call to action.
People want to do more with wheels, CI build wheel, or maybe some of the other stuff we talked
about.
What do you tell them?
Um, look through, I think one of the best places to go is the psychic developer pages.
If you have no interest in psychic tools or hep at all.
Um, and that sort of shows you how all these things integrate together really well.
And, has nice, has nice documentation.
Of course, CI build wheel itself is nice.
And the pipe PA, a lot of the IP projects have gotten, good documentation as well as packaging
of python.org.
We've updated that quite a bit.
Look like to reflect some of these things, but I would really, I really like the psychic
developer pages.
I mean, I'm biased because I wrote most of them.
Nice.
Yeah.
I'll link to those.
And I'll, I'll try to link to pretty much everything else that we spoke to as well.
So people can check out the podcast player show notes to find all that stuff.
I guess one final thing that we didn't call out that I think is worth pointing out is CI build
wheel is under the pipe, the Python packaging authority.
So it gives it some officialness, I guess you should say.
Yes.
That happened after, after I joined one of the first things I wanted to do was I thought this
should really be in the pipe PA.
And, I was sort of pushing for that and the other developers were fine with that.
And so we brought it up and, I actually joined the pipe PA just before that by becoming
a member of build.
Uh, so I got to vote on CI build wheel coming in, but it was a very enthusiastic vote, even
without my vote.
Um, and pipX joined right at the same time too.
So those were, it was fighting time.
PipX is a great library.
I really like the way pipX works.
It's a great tool.
All right, Henry, thank you for being here.
It's been great.
Thanks for all the insight on all these internals around building and installing Python packages.
There's also a lot more in my blog.
So I sign them, pie.
Dot.
Get lab.
Dot.
I.
Oh, so that's also a link to look that links to all those other things, obviously do.
Thanks again for being here.
Yeah.
See ya.
Thanks for having me.
Yeah.
You bet.
This has been another episode of talk Python to me.
Our guest on this episode was Henry Schreiner and it's brought to you by us over at talk
Python training and the transcripts were brought to you by assembly AI.
Do you need a great automatic speech to text API?
Get human level accuracy in just a few lines of code.
Visit talkpython.fm/assembly AI.
Want to level up your Python?
We have one of the largest catalogs of Python video courses over at talk Python.
Our content ranges from true beginners to deeply advanced topics like memory and async.
And best of all, there's not a subscription in sight.
Check it out for yourself at training.talkpython.fm.
Be sure to subscribe to the show.
Open your favorite podcast app and search for Python.
We should be right at the top.
You can also find the iTunes feed at /itunes, the Google Play feed at /play, and the
direct RSS feed at /rss on talkpython.fm.
We're live streaming most of our recordings these days.
If you want to be part of the show and have your comments featured on the air, be sure to
subscribe to our YouTube channel at talkpython.fm/youtube.
This is your host, Michael Kennedy.
Thanks so much for listening.
I really appreciate it.
Now get out there and write some Python code.
I'll see you next time.