WEBVTT

00:00:00.001 --> 00:00:04.220
Did you come to software development outside the traditional computer science path?

00:00:04.220 --> 00:00:07.580
This is common, and it's even how I got into programming myself.

00:00:07.580 --> 00:00:12.480
I think it's especially true for data scientists and folks doing scientific computing.

00:00:12.480 --> 00:00:15.460
That's why I'm thrilled to bring you an episode with Daniel Chen

00:00:15.460 --> 00:00:18.520
about maintainable data science tips and techniques.

00:00:18.520 --> 00:00:23.480
This is Talk Python To Me, episode 227, recorded August 6th, 2019.

00:00:23.480 --> 00:00:39.980
Welcome to Talk Python To Me, a weekly podcast on Python,

00:00:39.980 --> 00:00:43.040
the language, the libraries, the ecosystem, and the personalities.

00:00:43.040 --> 00:00:44.980
This is your host, Michael Kennedy.

00:00:44.980 --> 00:00:47.120
Follow me on Twitter, where I'm @mkennedy.

00:00:47.120 --> 00:00:50.860
Keep up with the show and listen to past episodes at talkpython.fm,

00:00:50.860 --> 00:00:53.300
and follow the show on Twitter via at Talk Python.

00:00:54.200 --> 00:00:56.800
This episode is sponsored by Indeed and Rollbar.

00:00:56.800 --> 00:00:59.140
Please check out what they're offering during their segments.

00:00:59.140 --> 00:01:00.540
It really helps support the show.

00:01:00.540 --> 00:01:03.180
Dan, welcome to Talk Python To Me.

00:01:03.180 --> 00:01:04.260
Hi, Mike. Nice to meet you.

00:01:04.260 --> 00:01:05.580
It's great to meet you as well.

00:01:05.580 --> 00:01:09.580
I'm so glad that we got a chance to run into each other at PyCon this year

00:01:09.580 --> 00:01:13.020
and learn about what you're up to because we're going to have a good time talking about it.

00:01:13.020 --> 00:01:16.000
Yeah, and this year was the first year I was at PyCon,

00:01:16.000 --> 00:01:18.500
and I typically live in the data science world,

00:01:18.500 --> 00:01:23.960
so it was one super cool to be at pretty much a convention of Python users.

00:01:23.960 --> 00:01:29.780
And I almost forgotten how Python is outside of data science.

00:01:29.780 --> 00:01:31.260
Django is a thing.

00:01:31.260 --> 00:01:34.980
It was a thing that was repeated back to me.

00:01:35.800 --> 00:01:36.640
Yeah, exactly.

00:01:36.640 --> 00:01:37.420
That's pretty interesting.

00:01:37.420 --> 00:01:39.800
What was your take on it?

00:01:39.800 --> 00:01:43.380
Do you recommend people, especially data scientists, attend PyCon?

00:01:43.380 --> 00:01:44.380
You're happy you went?

00:01:44.380 --> 00:01:44.780
Yeah.

00:01:44.780 --> 00:01:46.120
I mean, it was super cool.

00:01:46.120 --> 00:01:51.420
I mean, data science is sort of one of the growing parts of Python as a language.

00:01:51.540 --> 00:01:58.420
And I think a lot of people have said it's sort of the reason why Python has picked up in popularity recently.

00:01:58.420 --> 00:02:02.160
And so, yeah, it was super cool just to see all the booths.

00:02:02.160 --> 00:02:08.640
I personally gave a pandas tutorial there, so it is becoming more and more of a thing.

00:02:08.640 --> 00:02:15.160
And I think there were two or at least three pandas-related tutorials during the session.

00:02:15.520 --> 00:02:20.220
Yeah, I know Kevin Markham gave one as well, I'm pretty sure.

00:02:20.220 --> 00:02:22.360
At least something on data science there.

00:02:22.360 --> 00:02:23.840
So, yeah, there was definitely some interest.

00:02:23.840 --> 00:02:25.720
I think I met one other person who's doing one.

00:02:25.720 --> 00:02:27.400
So, yeah, it's pretty incredible, right?

00:02:27.400 --> 00:02:28.200
Yeah, yeah.

00:02:28.200 --> 00:02:35.420
And then, again, there's this whole web stack of things that I almost never really use,

00:02:35.420 --> 00:02:38.920
but a lot of people do use it as well.

00:02:38.920 --> 00:02:44.800
So, it's super cool just to see it and be reminded what Python can do as a language.

00:02:45.000 --> 00:02:45.540
Yeah, that's cool.

00:02:45.540 --> 00:02:47.160
And for me, it's exactly the opposite, right?

00:02:47.160 --> 00:02:50.240
Like, I spend a lot of my days writing web apps and APIs and things.

00:02:50.240 --> 00:02:53.640
And then to see the data science stuff, it really reminds me, like,

00:02:53.640 --> 00:02:59.520
there's a really different way to work and other things to optimize than, you know, scalable web apps.

00:02:59.520 --> 00:03:00.480
Yeah, yeah.

00:03:00.480 --> 00:03:00.900
For sure.

00:03:00.900 --> 00:03:02.960
So, totally recommended to go.

00:03:02.960 --> 00:03:05.900
And, like, you know, there was, like, some talk on Twitter, like,

00:03:05.900 --> 00:03:10.060
don't always try to do the hallway track if you can, because, like, speakers,

00:03:10.060 --> 00:03:12.380
sometimes speakers would like people in their audience.

00:03:12.580 --> 00:03:16.020
But I tried to go to the talks that I can.

00:03:16.020 --> 00:03:21.380
But then there were a few, like, education-related hallway track groupings or meetups.

00:03:21.380 --> 00:03:23.380
And that's what I attended to.

00:03:23.380 --> 00:03:26.520
So, it was just nice seeing other, like, Python educators,

00:03:26.520 --> 00:03:30.020
which I also went to SciPy a couple weeks later.

00:03:30.020 --> 00:03:32.620
So, it was some of the people I saw again for the second time.

00:03:32.620 --> 00:03:33.420
And I was like, oh, cool.

00:03:33.640 --> 00:03:35.380
Yeah, I definitely love going to Python.

00:03:35.380 --> 00:03:37.500
As people know, I talk about it all the time.

00:03:37.500 --> 00:03:39.200
And it's a really great experience.

00:03:39.200 --> 00:03:42.440
And what I think is interesting is a lot of people feel like they have to be experts to go.

00:03:42.440 --> 00:03:45.260
I met a lot of people who are fairly beginner in their career.

00:03:45.260 --> 00:03:47.780
And it was really valuable to them to be there.

00:03:47.780 --> 00:03:49.640
So, I just want to throw that out there for people.

00:03:49.640 --> 00:03:50.600
Yeah, super welcoming.

00:03:50.600 --> 00:03:54.280
I mean, that's sort of the reason why I stuck around with Python.

00:03:54.480 --> 00:03:57.100
I'm also, like, pretty active in the R community as well.

00:03:57.100 --> 00:04:01.280
And between Python and R, a lot of people join for whatever reason.

00:04:01.280 --> 00:04:05.040
But again, like, the saying, as the saying goes, like, they stay because of the community.

00:04:05.040 --> 00:04:09.400
And everyone's, like, just super nice and helpful and super beginner-friendly.

00:04:09.400 --> 00:04:10.400
Absolutely, absolutely.

00:04:10.400 --> 00:04:13.760
All right, well, I haven't got a chance to ask you the opening questions.

00:04:13.760 --> 00:04:14.800
Let's start there.

00:04:14.800 --> 00:04:21.020
So, before we get into all the techniques and tips and stuff you have for data scientists

00:04:21.020 --> 00:04:27.640
to bring in more structured programming stuff to make their data science techniques and tools better,

00:04:27.640 --> 00:04:29.860
let's talk about you.

00:04:29.860 --> 00:04:31.280
How did you get into programming in Python?

00:04:31.280 --> 00:04:34.900
So, I was pretty much always surrounded by computers as a kid.

00:04:34.900 --> 00:04:41.320
I always had, like, the hand-me-down computer when I was a kid from, like, my parents when they were working.

00:04:41.320 --> 00:04:45.280
I guess it sort of does help that my dad is a software engineer.

00:04:45.280 --> 00:04:47.440
But it wasn't really, like, a thing when I was at home.

00:04:47.440 --> 00:04:49.920
Other than, like, hey, dad does things on computers.

00:04:49.920 --> 00:04:50.760
That's kind of cool.

00:04:51.300 --> 00:04:53.080
I was always tinkering around with computers.

00:04:53.080 --> 00:04:57.240
So, like, I do remember, like, the first thing I would do every time, like, I open a new app is, like,

00:04:57.240 --> 00:05:00.600
hey, let's go to edit and preferences and just see, like, what I can change.

00:05:00.600 --> 00:05:01.840
And it was sort of just tinkering.

00:05:01.840 --> 00:05:03.800
I grew up in New York City.

00:05:03.800 --> 00:05:04.920
I'm from Queens.

00:05:04.920 --> 00:05:10.420
And I went to one of the specialized math and science high schools in New York City.

00:05:10.780 --> 00:05:20.760
And so, for us, sophomore year, it was actually mandated that every student take one semester of computer science and one semester of technical drawing or drafting.

00:05:20.760 --> 00:05:21.460
That's pretty cool.

00:05:21.820 --> 00:05:24.920
I think drafting is less valuable than people imagined it.

00:05:24.920 --> 00:05:28.580
Because I remember I had a drafting class as well and I don't really see it.

00:05:28.580 --> 00:05:32.600
But the software thinking and tools and ideas are certainly is.

00:05:32.600 --> 00:05:34.160
What language was that in?

00:05:34.160 --> 00:05:35.100
Pen and paper.

00:05:35.280 --> 00:05:36.660
And then CAD towards the end.

00:05:36.660 --> 00:05:37.080
Yeah.

00:05:37.080 --> 00:05:44.400
So, yeah, we were, we were, like, in, like, a room and we were drawing, like, isometrics by pen or pencil and ruler.

00:05:44.400 --> 00:05:46.120
Those big slanted tables.

00:05:46.120 --> 00:05:48.420
And the programming one, what was that?

00:05:48.420 --> 00:05:49.840
What technologies did you all cover?

00:05:49.840 --> 00:05:51.780
It was, like, it was only towards the end.

00:05:51.780 --> 00:05:54.480
And it was, like, in some CAD program that I don't remember.

00:05:54.480 --> 00:05:55.140
Oh, okay.

00:05:55.140 --> 00:05:55.600
Yeah.

00:05:55.600 --> 00:05:56.600
And it's, like, interesting.

00:05:56.600 --> 00:05:59.140
Like, that was a thing I never, I thought it was super cool.

00:05:59.140 --> 00:06:05.180
And then, like, now that 3D printing is a thing, it's sort of, like, oh, wait, I used to kind of, I've done this once.

00:06:05.180 --> 00:06:07.700
But, like, I just haven't done it in many years.

00:06:07.700 --> 00:06:09.460
So, it's, like, kind of interesting.

00:06:09.460 --> 00:06:09.880
Yeah.

00:06:09.880 --> 00:06:10.440
How interesting.

00:06:10.440 --> 00:06:11.140
Yeah.

00:06:11.140 --> 00:06:11.500
Yeah.

00:06:11.500 --> 00:06:11.960
How cool.

00:06:11.960 --> 00:06:13.420
Well, that's a great introduction.

00:06:13.420 --> 00:06:16.800
And then did you study computer science in college?

00:06:16.800 --> 00:06:17.980
So, I didn't.

00:06:17.980 --> 00:06:21.220
And that was part of it was I didn't notice until I was in college.

00:06:21.220 --> 00:06:26.960
But when we had to take computer science in high school, it was sort of, man, all of these other people.

00:06:27.060 --> 00:06:29.280
And by, like, other people, it's, like, just a handful.

00:06:29.280 --> 00:06:31.420
It was, like, man, they're really good at this.

00:06:31.420 --> 00:06:37.000
There's no way I'm going to be able to study this in the future or, like, for a career and whatnot.

00:06:37.000 --> 00:06:39.780
We did the one semester of computer science.

00:06:39.780 --> 00:06:46.200
I didn't go for the AP or anything because, like, originally I was going to, like, go down and be a medical doctor.

00:06:46.200 --> 00:06:47.520
That was my original plan.

00:06:47.520 --> 00:06:53.060
So, then in college, I ended up, how do I make my, like, medical education as strong as possible?

00:06:53.060 --> 00:06:57.640
Let's do, like, neuroscience and sort of, like, a bio-heavy program.

00:06:57.640 --> 00:07:02.240
But, like, that's where I sort of took my first set of statistics courses.

00:07:02.240 --> 00:07:05.660
And I was, oh, yeah, like, we hear about mean and standard deviation.

00:07:05.660 --> 00:07:13.300
But to finally understand it in the context of, oh, yeah, here's the exam scores for the previous exam.

00:07:13.480 --> 00:07:15.320
Like, how do you actually rank?

00:07:15.320 --> 00:07:20.600
Just something like that, like, get some meaningful understanding of, like, where do you rank in the class?

00:07:20.600 --> 00:07:22.560
And it's, like, oh, maybe this is how the curve is going to be.

00:07:22.560 --> 00:07:26.720
Or, like, did I didn't do very well in the exam, but, like, I'm actually kind of okay.

00:07:26.720 --> 00:07:28.320
So, that was cool.

00:07:28.320 --> 00:07:35.840
And then because I ended up switching into neuroscience my second year, I had to stay, like, a fifth year in college.

00:07:36.080 --> 00:07:40.800
And so, like, my last two years was, like, oh, you only need four classes to do, like, a computer science minor.

00:07:40.800 --> 00:07:42.700
So, I was, like, yeah, I've done this in high school.

00:07:42.700 --> 00:07:44.580
Let's, like, just pick this up for fun.

00:07:44.580 --> 00:07:55.480
And then so, it was that first intro computer science class when we got to, like, the actual Python programming portion where I was, like, wait, this is actually not as terrible.

00:07:55.660 --> 00:08:00.680
And I would see the other students, like, in which case, like, they would be freshmen and I would have been, like, already a junior.

00:08:00.680 --> 00:08:03.600
And I would see the freshmen, like, they've never seen this before.

00:08:03.600 --> 00:08:06.580
Their struggles were essentially my struggles back in high school.

00:08:06.580 --> 00:08:10.180
And then I realized, oh, it's literally because, like, I saw it before.

00:08:10.180 --> 00:08:17.400
And, like, even though, like, not much of it got retained, it was just thinking about things procedurally, just doing it once.

00:08:17.400 --> 00:08:22.120
Now I can actually think about, like, syntax errors versus, like, doing everything at once.

00:08:22.500 --> 00:08:25.900
That's sort of when I was, like, huh, maybe I could have done this as, like, a career choice.

00:08:25.900 --> 00:08:28.460
But, nope, nope.

00:08:28.460 --> 00:08:30.200
Let's keep going down the medicine route.

00:08:30.200 --> 00:08:37.420
So, I ended up doing a master's in public health and epidemiology just to stack on more research skills.

00:08:37.420 --> 00:08:44.840
The thought being was, hey, research in medicine was super cool, but I'm pretty sure if I ever start medical school, I'm never going to learn this stuff again.

00:08:44.840 --> 00:08:47.140
So, let's just learn everything and then go to medical school.

00:08:47.780 --> 00:08:53.960
So, I did my master's in epidemiology, and that's when I took my first, like, intro to data science course.

00:08:53.960 --> 00:08:58.180
And that is probably the most life-changing moment in my life.

00:08:58.180 --> 00:09:03.520
When I was doing my master's, I was already just learning about all of these other basic statistical techniques.

00:09:03.520 --> 00:09:05.580
I've never heard of logistic regression before.

00:09:05.580 --> 00:09:09.180
And that's, like, the type of analysis you do when you have a binary outcome.

00:09:09.180 --> 00:09:10.980
So, for us, it was like, did this person die?

00:09:10.980 --> 00:09:11.880
Yes or no.

00:09:11.880 --> 00:09:13.180
Or did this person get cancer?

00:09:13.180 --> 00:09:13.740
Yes or no.

00:09:14.240 --> 00:09:15.980
And I've never seen that before.

00:09:15.980 --> 00:09:17.720
And it was just like, wow, this is amazing.

00:09:17.720 --> 00:09:21.440
And then I take my data science class and I was like, what is this random forest thing?

00:09:21.440 --> 00:09:22.220
This is amazing.

00:09:22.220 --> 00:09:25.320
Or, like, what is this, like, ridge and lasso regression?

00:09:25.320 --> 00:09:30.260
And, like, I can just, like, condense, like, thousands of variables into, like, something meaningful.

00:09:30.260 --> 00:09:31.280
Like, that's super cool.

00:09:32.000 --> 00:09:36.940
And so, that sort of started this whole trajectory down to where I am now.

00:09:36.940 --> 00:09:46.360
Because it wasn't until, like, that data science course I, during that semester, because there was so much learning to do, the instructors set up a software carpentry workshop.

00:09:46.360 --> 00:09:49.200
And so, I was an attendee for software carpentry.

00:09:49.200 --> 00:09:54.940
I think software carpentry is a really cool project for folks with the background exactly like you described.

00:09:54.940 --> 00:10:01.780
I actually had Jonah Duckles on the show way back in episode 93 talking about software carpentry.

00:10:01.780 --> 00:10:03.860
So, it's been a really long time since I've spoken about it.

00:10:03.860 --> 00:10:08.580
Maybe just tell the listeners out there what software carpentry workshop is about.

00:10:08.580 --> 00:10:13.820
Because it'd be good for a lot of folks who are in the data science and sort of science in the programming space.

00:10:13.940 --> 00:10:18.160
So, yeah, it's sort of expanded over the past, like, couple years.

00:10:18.160 --> 00:10:25.160
But software carpentry and their sister program, Data Carpentry, they're housed under this one umbrella called the Carpentries.

00:10:25.160 --> 00:10:28.020
And, essentially, they're this nonprofit organization.

00:10:28.020 --> 00:10:37.280
And their goal is simply to teach researchers or scientists the skills that they need for, in the sense of software carpentry, like, programming skills.

00:10:37.280 --> 00:10:40.580
And then, in the case of Data Carpentry, like, working with data.

00:10:40.580 --> 00:10:41.540
So, like, data skills.

00:10:42.080 --> 00:10:43.960
And the two really just go hand in hand.

00:10:43.960 --> 00:10:45.680
So, you'll mix and match.

00:10:45.680 --> 00:10:46.980
They have a lot of overlap.

00:10:46.980 --> 00:10:52.860
And, essentially, there's these two-day workshops where they cover Bash for the shell.

00:10:52.860 --> 00:10:56.900
And the whole premise of that is to, like, show you about, like, what is a working directory?

00:10:56.900 --> 00:11:00.940
And programs do one thing and one thing really well.

00:11:00.940 --> 00:11:03.720
And you can pipe them into one another to chain things together.

00:11:03.720 --> 00:11:06.140
So, that's, like, what you're supposed to take away from Bash.

00:11:06.460 --> 00:11:12.760
And then they go through Git for version control, which it's really hard to get an understanding of Git in three hours.

00:11:12.760 --> 00:11:19.840
But it's just to show you that, like, there are better ways than naming your files final, final, final, et cetera, et cetera.

00:11:19.840 --> 00:11:21.980
Putting the date on the end.

00:11:21.980 --> 00:11:23.000
No, like, really, final.

00:11:23.120 --> 00:11:24.280
Yeah, and putting the date.

00:11:24.280 --> 00:11:29.980
And then there's a section on Python or some of, like, R or any of the other programming languages.

00:11:29.980 --> 00:11:34.620
And it used to be that they also had a fourth section on SQL.

00:11:34.620 --> 00:11:40.060
But then, usually, SQL gets bumped out for, like, a longer Python or R session.

00:11:40.740 --> 00:11:43.220
So, it's a two-day workshop that covers those skills.

00:11:43.220 --> 00:11:51.160
And it's really to give, like, researchers a primer because we go into science not thinking that we're going to program.

00:11:51.160 --> 00:11:55.220
And so, like, a lot of this stuff is just like, oh, I picked it up on my own.

00:11:55.220 --> 00:11:58.740
And it's just a bunch of stuff hobbled together.

00:11:58.740 --> 00:11:59.860
And that's how we learned it.

00:11:59.860 --> 00:12:04.040
And actually, that's how, like, a lot of people in data science, like, that's how they learn programming.

00:12:04.040 --> 00:12:05.940
And then this is, like, the first time, like.

00:12:05.940 --> 00:12:09.860
I feel like, yeah, I feel like this is actually really common, as you're saying.

00:12:09.860 --> 00:12:15.060
And I think it's also a little bit why Python is growing a lot in the data science space.

00:12:15.060 --> 00:12:22.640
Is it's, like, what can I do that's an easy step to do just enough computation to solve my problem so I can go back to what I actually care about?

00:12:22.640 --> 00:12:23.720
Because I don't want to be a programmer.

00:12:23.720 --> 00:12:26.480
I want to be a biologist or a doctor or whatever.

00:12:26.480 --> 00:12:30.760
But then you slowly find yourself six months later with, like, a lot of scripts.

00:12:31.640 --> 00:12:35.580
And you're running code and you're using pandas or NumPy.

00:12:35.580 --> 00:12:38.240
And you're like, well, I have no qualification for this.

00:12:38.240 --> 00:12:43.060
But here I am, like, in it somehow, even though I swore I would never do this because I hated math or something like that, right?

00:12:43.060 --> 00:12:43.360
Yeah.

00:12:43.360 --> 00:12:46.700
So that's the whole premise of the carpentries is, like, okay, let's take one step back.

00:12:46.700 --> 00:12:48.760
You learn how to do this on your own.

00:12:48.760 --> 00:12:53.860
And let's, like, refresh, like, the actual basics and, like, kind of, like, steer you in the correct way.

00:12:53.860 --> 00:12:57.120
That's the general lowdown of what the carpentries are.

00:12:57.120 --> 00:12:57.600
That's cool.

00:12:57.600 --> 00:13:00.160
And you started as a student, but you became an instructor, right?

00:13:00.360 --> 00:13:03.620
I was a student, like, fall of 2013.

00:13:03.620 --> 00:13:08.700
And then it was, like, just at the cusp of, wait, I can actually teach this stuff.

00:13:08.700 --> 00:13:11.200
It wasn't, like, that much a leap and bounce.

00:13:11.200 --> 00:13:13.400
Like, I already knew a little bit about Python programming.

00:13:13.400 --> 00:13:15.360
So, and then the Bash stuff.

00:13:15.360 --> 00:13:20.100
I was, like, one of those people in college that was, like, I'm just going to install Linux and see what happens.

00:13:20.100 --> 00:13:22.880
Deal with, like, problems that come from that.

00:13:22.880 --> 00:13:29.180
I've been saying to myself, like, it's the year of the Linux desktop since, like, 2010 or something.

00:13:29.700 --> 00:13:30.260
It's almost here.

00:13:30.260 --> 00:13:31.660
It's almost here.

00:13:31.660 --> 00:13:35.800
So I ended up signing up to, like, go help out.

00:13:35.800 --> 00:13:42.720
You end up realizing that, like, for a lot of newcomers, a lot of the problems that they have aren't actually that complicated.

00:13:42.720 --> 00:13:48.720
And then just to go into, like, education theory a little bit, it's, they don't have a lot of nodes to make connections with.

00:13:49.100 --> 00:13:53.280
And so a lot of their problems is also, like, just they'd made a typing mistake, right?

00:13:53.280 --> 00:13:55.780
Like, they're just not used to hitting tab to tab complete things.

00:13:55.780 --> 00:13:57.620
So, like, everything is mainly a typo.

00:13:57.940 --> 00:14:01.600
So I started off helping out a few workshops.

00:14:01.600 --> 00:14:13.400
And then I matriculated into, like, their next, like, instructor class where I was, like, certified to be an instructor where it was mainly, like, getting familiar with the material and, like, learning how to teach the material.

00:14:13.400 --> 00:14:13.960
That's cool.

00:14:14.160 --> 00:14:14.380
Yeah.

00:14:14.380 --> 00:14:15.840
And then I was an instructor.

00:14:15.840 --> 00:14:21.400
And my first couple of years as an instructor, that was, like, right on the border of, I was still in grads.

00:14:21.400 --> 00:14:23.440
I was, like, finishing up my master's program.

00:14:23.440 --> 00:14:26.540
And also, like, I had a job.

00:14:26.540 --> 00:14:31.280
But I ended up working so much during my job that my boss was pretty much, like, please go home.

00:14:31.280 --> 00:14:34.000
And so I spent a lot of time going home.

00:14:34.000 --> 00:14:36.440
But it was really just, like, go teach, like, other workshops.

00:14:36.440 --> 00:14:42.740
And it was, like, super nice being in the New York City area because, like, going to a university or any place was pretty much local for me.

00:14:43.160 --> 00:14:45.280
So I got a lot of teaching experience out of that.

00:14:45.280 --> 00:14:50.780
And I didn't know at the time, but I say it now, like, teaching is, like, one of the best ways to learn something.

00:14:50.780 --> 00:14:59.520
So Bash and Git and Python and later on R, like, I just got more familiar with it just because I was teaching it all the time.

00:14:59.520 --> 00:15:06.780
And then, you know, once you have some foundation, like, learning the next small bit of information is, it becomes easier and easier.

00:15:06.780 --> 00:15:09.320
And then it just snowballs into something.

00:15:09.320 --> 00:15:10.040
That's cool.

00:15:10.040 --> 00:15:10.380
Yeah.

00:15:10.480 --> 00:15:20.720
And then, like, all of that teaching knowledge ended up being, like, the foundation for, like, the book that I ended up writing or was tasked to write or call pandas for everyone.

00:15:20.720 --> 00:15:24.700
I mean, it's really, like, an honor that, like, I got recommended to write this thing.

00:15:24.700 --> 00:15:26.680
So I should frame it in that sense.

00:15:26.800 --> 00:15:28.220
I've done a lot of training as well.

00:15:28.220 --> 00:15:39.640
And I feel like once you kind of go through a couple of cycles of that, you just get so good at learning something with enough depth to present it that it becomes, like, this really great power.

00:15:39.640 --> 00:15:41.180
And it's kind of addicting, right?

00:15:41.200 --> 00:15:42.700
You're like, oh, what's the next thing I can learn?

00:15:42.700 --> 00:15:44.620
What's the next research project I can go on?

00:15:44.620 --> 00:15:45.520
And, yeah.

00:15:45.520 --> 00:15:54.820
So it sounds like you did the software carpentry thing and it kind of somehow sucked you down this pandas for everyone hole of writing this book, which is Addison Wesley, which is pretty cool.

00:15:55.060 --> 00:15:59.280
Even writing the book, like, now you're just like, oh, I just can't write, like, really janky code anymore.

00:15:59.280 --> 00:16:03.480
Like, this actually needs to be, like, quote, unquote, like, the better way of doing things.

00:16:03.480 --> 00:16:13.860
So, like, there was, like, still, even though I was, like, writing a book and I was supposed to be the expert in this, like, a lot of it was also, like, I should probably read this part of the documentation just to make sure.

00:16:13.860 --> 00:16:16.200
Because, like, I also learned this, like, on my own.

00:16:16.200 --> 00:16:25.920
Right. Well, that's the thing about the difference of practicing as a programmer or as a data scientist versus an author or an instructor, right?

00:16:25.920 --> 00:16:28.600
Like, as a practicing person, you have a problem.

00:16:28.600 --> 00:16:30.600
You're like, I need to figure out how to make pandas do this.

00:16:30.600 --> 00:16:32.000
Like, it doesn't matter how it happens.

00:16:32.000 --> 00:16:33.900
But if you can make it happen, you're done.

00:16:33.900 --> 00:16:35.380
Like, that's the end of the research.

00:16:35.380 --> 00:16:35.900
You're done.

00:16:35.900 --> 00:16:37.260
This part is solved.

00:16:37.260 --> 00:16:38.060
What's the next problem?

00:16:38.060 --> 00:16:40.840
But as an instructor, like, well, but there's these other two ways.

00:16:40.840 --> 00:16:44.300
And but if somebody says, well, why not this way versus that way?

00:16:44.300 --> 00:16:44.920
What's the difference?

00:16:44.920 --> 00:16:51.800
All of a sudden, like, all these cases that would you would never go down, like, you have to start going down those now, which is I think is awesome, actually.

00:16:51.800 --> 00:16:53.220
But it's definitely a different way of thinking.

00:16:53.220 --> 00:16:56.580
It's super cool because, like, now it becomes its own, like, learning path.

00:16:56.580 --> 00:16:59.640
Like, you see other people have problems and you see how they think about it.

00:16:59.640 --> 00:17:02.680
And, like, it sort of adapts how you present material.

00:17:02.680 --> 00:17:11.960
For me, when I was originally, when I first started off teaching workshops out of the book, I pretty much went in the order that I presented the chapters in.

00:17:12.060 --> 00:17:20.120
And then more and more recently, like, I realized, like, wait, like, tidy data principles is actually, like, one of the most important things in, like, data science and data cleaning.

00:17:20.120 --> 00:17:24.600
After we load our first data set, I pretty much just jumped to, like, that chapter.

00:17:24.600 --> 00:17:29.560
Because if you can really understand that, everything else becomes way easier, quote unquote easier.

00:17:29.560 --> 00:17:30.220
Yeah, sure.

00:17:30.220 --> 00:17:34.700
Well, if you're trying to do operations on bad data and it keeps crashing, like, that's no fun.

00:17:35.800 --> 00:17:37.380
Like, why does it say none?

00:17:37.380 --> 00:17:38.100
Is that invalid?

00:17:38.100 --> 00:17:39.420
You know, it doesn't have this attribute.

00:17:39.420 --> 00:17:40.200
I don't understand.

00:17:40.200 --> 00:17:42.160
Like, well, let's talk about that.

00:17:43.960 --> 00:17:47.480
This portion of Talk Python To Me is brought to you by Indeed Prime.

00:17:47.480 --> 00:17:50.040
Are you putting your Python skills to good use?

00:17:50.040 --> 00:17:55.420
Find your dream role with Indeed Prime and start doing more of what you love every day.

00:17:55.420 --> 00:18:04.520
Whether you're a developer, data scientist, or anything in between, one application puts you in front of hundreds of companies like PayPal and VRBO in over 90 cities.

00:18:04.960 --> 00:18:13.620
Indeed Prime showcases your experience and tech skills to match you with great fit roles that meet and exceed your salary, location, and career goals.

00:18:13.620 --> 00:18:21.960
And when you start a one-on-one conversation with one of their career coaches, you'll get resume reviews and personalized advice to help you get what you deserve.

00:18:21.960 --> 00:18:28.500
So, if filling out countless job applications isn't your thing, let top tech companies apply to you.

00:18:28.500 --> 00:18:32.960
Join Indeed Prime for free at talkpython.fm/indeed.

00:18:32.960 --> 00:18:35.720
That's talkpython.fm/indeed.

00:18:35.720 --> 00:18:42.140
The reason I wanted to talk a little bit about software carpentry, other than just like you have been doing in his school,

00:18:42.140 --> 00:18:51.480
is I think it's a really good segue into this larger topic of how do you take the average data scientist and the work that they're doing

00:18:51.480 --> 00:18:58.780
and help bring in these more computer science, maybe not even computer science, let's say software engineering principles,

00:18:58.780 --> 00:19:01.820
to help them basically be more effective, right?

00:19:01.820 --> 00:19:03.220
So, maybe we start at the beginning.

00:19:03.220 --> 00:19:05.060
We've got some idea.

00:19:05.060 --> 00:19:10.920
We probably found out we can open up a Jupyter notebook, load something into Pandas,

00:19:10.920 --> 00:19:15.040
and poke around with it with a Matplotlib or something, right?

00:19:15.040 --> 00:19:16.000
Maybe that's it, right?

00:19:16.000 --> 00:19:22.000
Maybe we've seen a lot of Matlab code as well, where it's like, well, I got this does this thing,

00:19:22.000 --> 00:19:23.460
but it's like there's no functions.

00:19:23.920 --> 00:19:26.560
Maybe there's loops, maybe not, right?

00:19:26.560 --> 00:19:28.400
It's just like all crammed in there.

00:19:28.400 --> 00:19:30.760
And those are PhDs writing that.

00:19:30.760 --> 00:19:34.940
So, like really brilliant people, but they just don't have the software engineering skills.

00:19:34.940 --> 00:19:37.060
So, where do we start with that?

00:19:37.400 --> 00:19:43.260
There's a few papers I would direct people to sort of get a sense of where I'm coming from.

00:19:43.260 --> 00:19:48.960
So, there's this one paper by William Noble called A Quick Guide to Organizing Computational Biology Projects.

00:19:48.960 --> 00:19:55.580
And that's sort of the premise of how, I guess, I would present, how do we introduce software skills?

00:19:56.340 --> 00:19:59.440
And in that paper, he literally talks about you should have a folder structure.

00:19:59.440 --> 00:20:03.700
And maybe this is one way you should set up your folders for your analysis projects.

00:20:03.700 --> 00:20:05.940
And I'll talk a little bit about that in a bit.

00:20:05.940 --> 00:20:06.740
But yeah.

00:20:06.740 --> 00:20:07.120
Yeah.

00:20:07.120 --> 00:20:10.260
So, it's called A Quick Guide for Organizing Computational Biology Projects.

00:20:10.260 --> 00:20:14.680
And, you know, it's probably focused on biologists, but I'm sure that it's like pretty generally applicable.

00:20:14.680 --> 00:20:15.140
Yeah.

00:20:15.140 --> 00:20:15.580
Yeah.

00:20:15.580 --> 00:20:21.080
Other than like maybe the sequence.py file, like replace that name with whatever you need.

00:20:21.080 --> 00:20:21.760
Right.

00:20:21.760 --> 00:20:24.960
It's Hubble.py or whatever.

00:20:24.960 --> 00:20:25.700
Yeah.

00:20:25.880 --> 00:20:33.760
And the other two papers, the first author is by Greg Wilson, who restarted Starforce Carpentry, like back in the 2000s.

00:20:33.760 --> 00:20:38.640
And he wrote two papers, one in 2014 called like Best Practices for Scientific Computing.

00:20:38.640 --> 00:20:43.820
And then in 2017, the paper is called Good Enough Practices in Scientific Computing.

00:20:43.820 --> 00:20:48.480
If you just look at the papers, it almost seems like, hey, we're presenting like the ideal case.

00:20:48.480 --> 00:20:51.320
And then we almost realize like that's impossible in the real world.

00:20:51.320 --> 00:20:53.160
But they're both like pretty good papers.

00:20:53.160 --> 00:20:55.100
And they talk about like different things.

00:20:55.100 --> 00:20:55.300
Right.

00:20:55.300 --> 00:21:00.400
What would we have if we had like the perfect adaptation of software engineering to this world?

00:21:00.400 --> 00:21:05.040
Like, okay, well, what can we reasonably ask people to do that will make their life better?

00:21:05.040 --> 00:21:05.760
It sounds like.

00:21:05.760 --> 00:21:06.140
Yeah.

00:21:06.480 --> 00:21:15.780
And the way I approach it is like, just like when I teach data science skills, I pretty much make a beeline to tidy data and tidy data principles.

00:21:15.780 --> 00:21:20.280
In this case, it's almost like a beeline towards project organization.

00:21:20.280 --> 00:21:25.580
Just having some kind of structure to your analysis project.

00:21:25.580 --> 00:21:33.360
That will snowball into all of the cool tools that you probably heard of and don't know how people end up there.

00:21:33.500 --> 00:21:42.980
But if you take slow steps, I found that project organization is the fundamental thing where it's sort of like the gateway to everything else.

00:21:42.980 --> 00:21:43.380
Right.

00:21:43.480 --> 00:21:47.620
Because a lot of what you need, it sounds like, is code organization.

00:21:47.620 --> 00:21:48.520
Right.

00:21:48.520 --> 00:21:55.560
It's like the architecture and functions, classes, different modules, the concept of I'm going to pass data to this thing and make it reusable.

00:21:56.060 --> 00:22:05.520
All of that stuff really seems to be like natural follow-ons of like, well, how do we organize this project by function or by purpose?

00:22:05.520 --> 00:22:07.200
And like, just really think through that, right?

00:22:07.200 --> 00:22:07.540
Yeah.

00:22:07.540 --> 00:22:15.160
And it doesn't even have to be as complicated as, oh, we're doing like proper software engineering and like we need to create a Python package.

00:22:15.160 --> 00:22:17.780
Like that can all be deferred to much later.

00:22:18.000 --> 00:22:21.000
Because usually what ends up happening, you mentioned like, hey, I'm a scientist.

00:22:21.000 --> 00:22:22.740
I found out about Jupyter Notebooks.

00:22:22.740 --> 00:22:24.420
It's a really cool tool.

00:22:24.420 --> 00:22:27.120
Taking pictures of black holes out, like using them.

00:22:27.120 --> 00:22:29.440
So, yeah, you have all these tools.

00:22:29.440 --> 00:22:35.940
And like the scenario is like, hey, it's great that you're using a programming language to work with data.

00:22:35.940 --> 00:22:39.440
Excel is a great GUI for data, but it has its limitations.

00:22:39.440 --> 00:22:40.620
Cool.

00:22:40.620 --> 00:22:42.680
You are now using a programming language.

00:22:42.680 --> 00:22:44.680
Where can we go from there?

00:22:44.680 --> 00:22:51.240
And like when you are in that beginning state, just to make everything work, like you dump everything in one folder.

00:22:51.240 --> 00:22:52.860
You have like your Jupyter Notebooks.

00:22:52.860 --> 00:22:55.220
You have all your scripts, all your data.

00:22:55.220 --> 00:22:56.240
Your data files.

00:22:56.240 --> 00:22:56.480
Yeah.

00:22:56.480 --> 00:22:56.800
Yeah.

00:22:56.800 --> 00:23:00.000
If I say load this, I just want to say the file name.

00:23:00.000 --> 00:23:05.460
I don't want to have to think about like where that's relative to the other on some like server or something like that, right?

00:23:05.460 --> 00:23:05.700
Yeah.

00:23:05.700 --> 00:23:09.860
And then like as an academic, you might have like a Word doc in there or maybe a LaTeX file.

00:23:09.860 --> 00:23:17.680
And then you compile that thing and it very quickly becomes this folder with hundreds of files and you can't find anything.

00:23:17.680 --> 00:23:24.280
And that's when you just start end up, you know, maybe the word final comes into like the beginning of the file name just so like you can find things, right?

00:23:24.280 --> 00:23:25.160
Yeah.

00:23:25.160 --> 00:23:26.560
I was going to say it already sounds bad.

00:23:26.560 --> 00:23:32.640
And then if you start trying to do version control by like having multiple files named the same thing, then you're really pushing your luck.

00:23:32.740 --> 00:23:32.980
Yeah.

00:23:32.980 --> 00:23:38.880
So the most important thing, I think like if you're at that point, where can you go next, right?

00:23:38.880 --> 00:23:40.760
It's always like trying to do things incrementally.

00:23:40.760 --> 00:23:43.540
Like how do you make your life like 10% better each time?

00:23:43.540 --> 00:23:52.120
And then it's like a nice way, especially if you're like brand new grad student or you're in science, but like you've never really learned programming.

00:23:52.120 --> 00:23:53.760
Like where can you go from there?

00:23:53.960 --> 00:24:01.000
It's useful to have some kind of guide or path that you can follow or think about to like make yourself better and do these things more efficiently.

00:24:01.000 --> 00:24:01.380
Yeah.

00:24:01.380 --> 00:24:05.700
So let's talk about some of the programming things that you can think about.

00:24:05.700 --> 00:24:10.700
One of the ones that you have is like try to make your code easy to read.

00:24:10.700 --> 00:24:11.540
Oh, yes.

00:24:11.760 --> 00:24:15.980
So one of the things I talk about in programming is like make things easier to read.

00:24:15.980 --> 00:24:17.240
Do things in steps.

00:24:17.240 --> 00:24:23.140
Don't try to like write one for loop that have a whole bunch of like side effects going on, right?

00:24:23.140 --> 00:24:27.420
Like things should just be incremental just to take like a cue from like education.

00:24:27.420 --> 00:24:34.980
Like we as human beings can only carry, I think the number is like four plus or minus three objects in our mind at the same time, like roughly seven.

00:24:34.980 --> 00:24:37.320
You should pretty much follow that too.

00:24:37.400 --> 00:24:47.060
When you're programming, you shouldn't have to have, I mean yourself or like potentially another reader try to carry like 10 different things going on at the same time.

00:24:47.060 --> 00:24:48.620
It's just not helpful for.

00:24:48.620 --> 00:24:51.340
Like maybe an example is I'm trying to go through a loop.

00:24:51.340 --> 00:24:52.740
I'm really trying to do three things.

00:24:52.740 --> 00:24:59.100
Like as I get the data and I compute something with the first step and then I do some other filtering and I do another thing.

00:24:59.100 --> 00:25:03.640
I could try to cram that into like one giant loop or maybe it should be three separate little loops.

00:25:04.380 --> 00:25:10.120
One that like cleans the data, one that like does that computation, another that then filters it, right?

00:25:10.120 --> 00:25:13.320
Three loops sounds like a better step than one giant loop trying to do it all.

00:25:13.320 --> 00:25:13.640
Yeah.

00:25:13.640 --> 00:25:18.200
Or you can more in like education framework, like, or you can like group things together.

00:25:18.200 --> 00:25:21.260
And in programming, the way we group things together is like write functions.

00:25:21.260 --> 00:25:30.780
So then you end up with one giant loop and it's really just making three function calls, but that's easier to keep track of than like, let's say we didn't write the function.

00:25:30.780 --> 00:25:38.180
Now we have like three different things like scattered in our code and you end up with a loop that's 150 lines long.

00:25:38.440 --> 00:25:47.020
And that's like scary because like I see a loop and like before I even look at this thing, like I'm already like, oh man, we are in for a ride.

00:25:47.020 --> 00:25:47.520
Right.

00:25:47.720 --> 00:25:54.460
So let me just give my perspective from like the software development, web dev, you know, more application side of things.

00:25:54.460 --> 00:26:00.320
It's like, if I see a function that's more than 10 lines long, it starts to make me nervous.

00:26:00.960 --> 00:26:14.120
I'm like, there is something going on here that's probably bad unless there's like a lot of error handling and like response, like even 10 is a lot in the typical scientific computing bits that I've at least seen a while ago.

00:26:14.120 --> 00:26:15.820
There was more than 10 lines.

00:26:15.820 --> 00:26:16.780
There's more than 10 lines.

00:26:16.780 --> 00:26:23.520
And so like Jenny, Brian, like from the R world has like this talk about like code smells.

00:26:23.520 --> 00:26:27.260
And it's like, that's like one of those like code smells of like, Hey, why does it look like this?

00:26:27.260 --> 00:26:39.360
Or like, at least when you're working with data or in the PI data stack, usually you shouldn't have to write for loops in the sense of like, if you're trying to operate on a data frame, they should be an apply call to a function.

00:26:39.360 --> 00:26:44.460
Even like sometimes when I see loops, it's like, yes, I will write them just because something broke.

00:26:44.460 --> 00:26:51.920
And I'm just trying to figure out like where my data frame, like I have a bad value, but like the final result ends up being like an apply call or something.

00:26:51.920 --> 00:26:52.200
Yeah.

00:26:52.200 --> 00:26:56.980
It's interesting because a lot of the libraries, NumPy, Pandas and whatnot, can they do the looping?

00:26:57.040 --> 00:27:00.240
They do it much faster and more efficient than you will in Python.

00:27:00.240 --> 00:27:19.320
One of the cool things that I like teach during the data science part is like when we go over like applying functions, if you're doing numerical computations, like just the NumPy decorator for like vectorize or the number decorator for vectorize, like just wrap the decorator around your computation function.

00:27:19.500 --> 00:27:24.060
It like pretty much for free gives you order of magnitude speed improvements.

00:27:24.060 --> 00:27:28.860
And so it's like, it's way better than just you trying to like optimize this thing yourself.

00:27:28.860 --> 00:27:29.060
Right.

00:27:29.060 --> 00:27:30.960
And that's like one of the other programming things.

00:27:30.960 --> 00:27:34.240
It's premature optimization is like the bane of all evil or whatever.

00:27:34.240 --> 00:27:37.160
Just write the thing you want, especially if they're like loops.

00:27:37.380 --> 00:27:42.500
Python has many mechanisms to like help you with that and make it faster pretty much for free.

00:27:42.500 --> 00:27:43.480
That's definitely cool.

00:27:43.480 --> 00:27:45.220
I love this idea of code smells.

00:27:45.220 --> 00:27:46.300
I'm fascinated by it.

00:27:46.300 --> 00:27:46.960
I want to come back to it.

00:27:46.960 --> 00:27:52.500
But another thing I want to throw in that I kind of, I feel like is in this realm is like the idea of reusability.

00:27:53.320 --> 00:27:57.120
You can write code so that it's easily reusable or that it's not so much.

00:27:57.120 --> 00:28:04.760
So like I could write a function, but maybe I have a bunch of global variables that I'm still using and it makes the function like it moves the code away.

00:28:04.760 --> 00:28:09.800
So I understand that it's like it's more compact and more readable, but it doesn't necessarily make it reusable.

00:28:09.800 --> 00:28:21.540
So thinking about like how do I parameterize these things and make them something that I can use in other situations or once you solve this problem in this way, like I never have to think about this again.

00:28:21.540 --> 00:28:26.340
I just now use this in the other part and that was rough, but that was Friday and I don't have to think about it ever again.

00:28:26.340 --> 00:28:28.960
Like that's a pretty good principle, I think, here as well.

00:28:28.960 --> 00:28:29.260
Yeah.

00:28:29.260 --> 00:28:33.680
And even when you're writing your functions, you can write your function for your use case now.

00:28:33.940 --> 00:28:43.800
And, you know, for example, it's like a function that is a regular expression parser for like a US telephone number, which is, if you try to write one of those, it's like way more complicated than ever it needs to be.

00:28:43.800 --> 00:28:48.760
But it's like final exam in like regular expression 101 or something.

00:28:48.760 --> 00:28:50.780
Like it's really like way worse than it should be.

00:28:50.780 --> 00:28:51.260
Yeah.

00:28:51.260 --> 00:28:52.700
You'll write your function with that in.

00:28:52.700 --> 00:29:02.700
And like one of the things I end up doing is even if I have hard coded things within the function and then I realize later on like, oh, wait, I pretty much need to run that function again.

00:29:02.840 --> 00:29:06.740
But instead of like the second index, I need like the fourth index or whatever.

00:29:06.740 --> 00:29:15.000
You can make backwards compatible functions or code by like saying like, oh, I'm just going to create a default parameter in my function.

00:29:15.000 --> 00:29:16.940
It's going to default to the one that already works.

00:29:16.940 --> 00:29:20.900
But then now I can just reuse that function later on and like just change that value.

00:29:20.900 --> 00:29:26.780
Simple things like that, that you don't have to rewrite the function just for your second use case.

00:29:26.880 --> 00:29:35.460
I like, I talk about like, if you ever hit control C on your computer, you better be paying attention when and how many times you're hitting control V, right?

00:29:35.460 --> 00:29:39.200
And if it's like more than three times, you're probably doing something wrong.

00:29:39.200 --> 00:29:40.220
Yeah, for sure.

00:29:40.220 --> 00:29:40.820
For sure.

00:29:41.680 --> 00:29:46.220
One of the things I think would be nice, you talked about premature optimization and all these performance stuff.

00:29:46.220 --> 00:29:52.300
What is your recommendation around like how you structure your code?

00:29:52.300 --> 00:30:03.260
So a lot of times I imagine that the data science stuff has pretty much like there's a Jupyter notebook and most of the code, like the supporting functions are kind of the beginning.

00:30:03.260 --> 00:30:06.160
And then later on, like they're kind of using them and so on.

00:30:06.160 --> 00:30:12.880
When do you tell folks to break out like separate Python modules that you could load into your notebooks?

00:30:12.880 --> 00:30:27.440
And like, what's the, how do you think about like different module files versus notebooks and things like you can apply refactoring tools really easily to a bunch of files using PyCharm or things like VS Code, not so easily in Jupyter, right?

00:30:27.440 --> 00:30:29.260
So where's the balance there?

00:30:29.260 --> 00:30:36.660
Yeah, so the thing with Jupyter notebooks is, yes, there was like a talk at JupyterCon about why Jupyter notebooks are bad.

00:30:36.660 --> 00:30:39.860
And I have this love hate relationship with Jupyter notebooks.

00:30:39.860 --> 00:30:50.400
But one of the things I can say, so Rachel Tappman from Kaggle, she gave like an Our Ladies Meetup talk in 2018 about like putting together a data science portfolio.

00:30:51.260 --> 00:31:04.540
And one of the things in there is like the Jupyter notebook is great, but most of the time you probably are just interested in like the figures or tables that's being generated, especially if you're taking this into a meeting, right?

00:31:04.540 --> 00:31:12.580
Like no one wants to like scroll forever to get to the bottom of the notebook because the first like three quarters is cleaning code.

00:31:12.660 --> 00:31:12.900
Right.

00:31:12.900 --> 00:31:21.660
I sort of like got into this sort of workflow of like I'll use the Jupyter notebook to like test things out in like data cleaning pipeline.

00:31:21.660 --> 00:31:25.800
But the actual data cleaning stuff all go into like Python scripts.

00:31:25.800 --> 00:31:33.540
So like what ends up at the end of the day, what happens in the Jupyter notebook is like pretty much load the libraries I want, load the data I want.

00:31:33.540 --> 00:31:39.700
Maybe there's like a few functions that's specific to like the figures I need and then just the figures and tables I need.

00:31:39.700 --> 00:31:51.100
So my Jupyter notebooks are pretty small and that down the line in terms of like other software engineering practices that just makes the diffs in through Git just way more manageable if I start making changes.

00:31:51.100 --> 00:32:04.000
So if you end up with massive Jupyter notebooks that are a lot of it is just data cleaning code, you would think about like moving that out to other notebooks or other files just so you have more files.

00:32:04.000 --> 00:32:16.940
I'm in the camp of pretty much in a lot of academic or scientific use cases, maybe not in like physics when they're working with like sensor data, but file IO is not that big of a bottleneck.

00:32:16.940 --> 00:32:24.160
So like I will have more scripts and more files that just write out data just to have another script and file read it back in.

00:32:24.240 --> 00:32:28.960
But that just breaks up my thought processes into smaller manageable pieces.

00:32:28.960 --> 00:32:29.280
That's interesting.

00:32:29.280 --> 00:32:31.780
It's like a little bit of a cache as well, right?

00:32:31.780 --> 00:32:39.120
Like you can take the step N and go to N plus one and like iterate on how that happens without rerunning all the stuff, right?

00:32:39.120 --> 00:32:40.940
Because you just reload that file that you save.

00:32:40.940 --> 00:32:41.980
Yeah, yeah, exactly.

00:32:41.980 --> 00:32:49.580
So like this goes down into like project template world where like I'll have a data folder in our data folder, you know, we'll have like an original data folder.

00:32:49.580 --> 00:32:54.340
That is the data that we download stuff in, never make changes to your raw data.

00:32:54.340 --> 00:32:57.000
And then everything else gets modified with a script.

00:32:57.420 --> 00:33:01.580
I'll have like, for example, a script that reads in one of my original data sets.

00:33:01.580 --> 00:33:03.680
I'll do my first set of processing.

00:33:03.680 --> 00:33:05.840
Like maybe it's like, oh, fixing missing values.

00:33:05.840 --> 00:33:13.020
And then I'll immediately write it out to somewhere in like under data and processing because it's now a process data set.

00:33:13.020 --> 00:33:14.840
And I want to distinguish between data sets.

00:33:14.840 --> 00:33:21.540
I can, I should just pretty much lock as read only versus things that I could like potentially modify and delete later on.

00:33:21.800 --> 00:33:25.880
I'll have a whole series of these scripts that pretty much just like you'll see it.

00:33:25.880 --> 00:33:33.500
I rarely these days have scripts that are like more than a hundred lines long because it's pretty much read in, do this one task, write it out.

00:33:33.500 --> 00:33:38.320
And especially if you have like one step that just takes a really long time.

00:33:38.320 --> 00:33:50.100
Yeah, it serves as pretty much as a cache where you just save out your temporary results and then you can deal with it later without like accidentally rerunning the part of your code that you didn't mean to run because now you're stuck for an hour.

00:33:50.420 --> 00:33:53.500
And that's sort of like what happens with Jupyter Notebooks as well.

00:33:53.500 --> 00:33:58.720
When we first started programming, like when I first started programming, it was just like, I just need this stuff to run.

00:33:58.720 --> 00:34:04.580
So I'll run cell one and then jump to cell 10 and I'll run cell one again and then jump to cell 15.

00:34:04.580 --> 00:34:07.340
And then I can like scroll all the way down and get my plot.

00:34:07.340 --> 00:34:07.540
Right.

00:34:07.540 --> 00:34:09.300
And then it's like, how am I supposed to run?

00:34:09.300 --> 00:34:10.980
How do you document something like that?

00:34:10.980 --> 00:34:11.180
Right.

00:34:11.180 --> 00:34:18.160
And that's sort of one of the drawbacks with the Jupyter Notebook is, yeah, the execution order isn't guaranteed in what was written.

00:34:18.160 --> 00:34:19.720
It's a little bit like a go-to.

00:34:20.260 --> 00:34:20.400
Yeah.

00:34:20.400 --> 00:34:21.740
It's pretty much like a go-to.

00:34:21.740 --> 00:34:22.060
Yeah.

00:34:22.060 --> 00:34:22.900
Which is kind of bad.

00:34:22.900 --> 00:34:24.840
Except it's not even like documented, right?

00:34:24.840 --> 00:34:25.080
Yeah.

00:34:25.080 --> 00:34:26.660
At least it doesn't even say go to 20.

00:34:26.660 --> 00:34:28.460
It's just like they went to 20.

00:34:28.460 --> 00:34:28.680
Yeah.

00:34:28.680 --> 00:34:31.340
And then when you execute it, it turns to 21, right?

00:34:31.340 --> 00:34:34.460
So like it doesn't even, like you don't even know what 20 is, right?

00:34:34.940 --> 00:34:42.440
So if you end up in a situation where you're running bits and pieces of a code all over the place, that's a sign of like, wait, let's fix this now.

00:34:42.680 --> 00:34:45.600
It's pretty cheap to create a new file.

00:34:45.600 --> 00:34:48.860
And let's do all the data cleaning or the parts I need for this figure.

00:34:48.860 --> 00:34:50.720
Maybe that can just be in one thing.

00:34:51.040 --> 00:34:56.800
And then more and more as you find pieces that need to be reused, you'll, oh, maybe I can turn this into a function.

00:34:56.800 --> 00:34:58.740
Then you'll put that as a module.

00:34:58.740 --> 00:35:01.660
And I would say like even if it's a module, just leave it in.

00:35:01.660 --> 00:35:08.340
If the folder structure is pretty much you have a data folder and an analysis folder and an output folder where output is like your figures and stuff.

00:35:08.800 --> 00:35:10.120
At first, it's okay.

00:35:10.120 --> 00:35:12.660
You can have your modules in your analysis folder.

00:35:12.660 --> 00:35:17.440
And so you can still say import something and it'll still import properly in that sense.

00:35:17.440 --> 00:35:28.060
You don't have to like just go and make a Python package right away because at least in what I've seen is sometimes your analysis, it's not really going to be reused across projects.

00:35:28.060 --> 00:35:31.240
You don't need the overhead of writing a Python package.

00:35:31.240 --> 00:35:46.760
It's when you, for example, if you're querying, if you're doing some study on like code in GitHub, for example, and you write your own GitHub querying API call stuff, and then you realize this is part of one giant grant with many different analyses that need to happen.

00:35:46.760 --> 00:35:51.340
Maybe like your GitHub querying code will turn into a package because you're actually reusing it.

00:35:51.340 --> 00:35:54.820
You don't have to turn everything into a Python project.

00:35:54.820 --> 00:35:57.440
You don't have to do that to do it like quote unquote correctly.

00:35:57.440 --> 00:36:09.900
Yeah, and I feel like the value from going from like just some huge notebook or some huge script file and then moving that into modules that have functions you can import and run and whatever, that's like 85% of the way, right?

00:36:09.900 --> 00:36:11.960
Whether or not you can pip install the thing, it doesn't matter.

00:36:11.960 --> 00:36:16.380
You know, there's a lot of overhead to make something super reusable to make it documented.

00:36:16.380 --> 00:36:23.140
Like maybe if you're in academics, maybe that's a cool project for like a senior undergraduate person.

00:36:23.140 --> 00:36:23.980
Like, hey, you know what?

00:36:23.980 --> 00:36:24.980
You know Python.

00:36:24.980 --> 00:36:27.640
Why don't we take this and turn this into an open source project?

00:36:27.640 --> 00:36:28.920
And that can be your project, right?

00:36:28.920 --> 00:36:33.300
Like, I'm not sure it's a great research time and energy in general.

00:36:33.300 --> 00:36:33.600
Yeah.

00:36:33.600 --> 00:36:42.360
Well, so more and more, there's very recently PyOpenSci is an organization that sprung up and it's trying to mimic our OpenSci.

00:36:42.360 --> 00:36:53.440
And it's essentially like supposed to be a repository of Python packages made towards making science better for some scientific use case.

00:36:53.440 --> 00:37:08.480
And all of those are going to be reviewed by somebody and it fast tracks you for if you want to write a paper based off of that software package, it'll fast track you into Joss, which is a journal of open source software.

00:37:08.480 --> 00:37:08.880
Yeah.

00:37:08.880 --> 00:37:10.280
And I had them on the show as well.

00:37:10.280 --> 00:37:10.820
Yeah.

00:37:10.820 --> 00:37:15.840
So now you have at least like the incentives are more or less lined up, right?

00:37:15.840 --> 00:37:20.840
Because before, like if you were just maintaining a software package, you know, what are your academic incentives?

00:37:20.840 --> 00:37:24.060
Because a lot of that is still around publishing and grants.

00:37:24.060 --> 00:37:31.800
So at least now there's the incentives are now lined up where like even though you are writing a software package, you can now write a paper about it.

00:37:31.800 --> 00:37:32.020
Yeah.

00:37:32.020 --> 00:37:35.100
It may generate a paper which might help you with your tenure and so on.

00:37:35.100 --> 00:37:38.000
But I guess let me take a step back really quick on my statement.

00:37:38.000 --> 00:37:46.260
Like it might not help you in your academic career directly to spend the software engineering time, but it may help you significantly in your research.

00:37:46.260 --> 00:37:51.900
If you can publish something and then you get other researchers to start using it, right?

00:37:51.900 --> 00:37:54.560
It becomes a package that you have more contributors to, right?

00:37:54.560 --> 00:37:57.340
Maybe you have one student you could fund part time.

00:37:57.340 --> 00:37:59.860
Now all of a sudden there's 20 institutions like all working.

00:37:59.960 --> 00:38:07.140
Like that could be a huge benefit, but I think a lot of stuff is so specialized, so tied to your data and your particular problem.

00:38:07.140 --> 00:38:10.740
Like you say, your first thought shouldn't be how do I open source this as a package?

00:38:10.740 --> 00:38:13.300
It's like, how do I just like make this a decent software project?

00:38:13.300 --> 00:38:13.620
Yeah.

00:38:13.620 --> 00:38:16.980
And that's a pretty lofty first goal too.

00:38:16.980 --> 00:38:18.900
Like how do I make this work properly for myself?

00:38:18.900 --> 00:38:19.080
Right.

00:38:19.100 --> 00:38:25.500
Because then that you go into the route of like, okay, I should write tests for this just to make sure it's like at least behaving correctly.

00:38:25.500 --> 00:38:34.680
There's a bunch of incentives as well for just having an open source project and trying to get other people to play with it because you'll build out the functionality for the thing you built.

00:38:34.820 --> 00:38:38.920
And as functionality expands, you'll sort of get more and more people in.

00:38:38.920 --> 00:38:44.040
And it sort of ties back to like the Python community is great.

00:38:44.040 --> 00:38:53.500
And so like now you are embracing the broader Python community and now you have more and more resources or people you've met to help you with your own project.

00:38:53.740 --> 00:39:01.760
If you're at like PyCon or SciPy, you can have your own sprint for your software project just to have other people try this out.

00:39:01.760 --> 00:39:08.580
You end up building your own community off of your little software project, which is, it makes you feel good.

00:39:08.580 --> 00:39:10.580
And it's still also advancing science.

00:39:10.580 --> 00:39:12.640
And a lot of science is also communication.

00:39:12.640 --> 00:39:15.380
And you built this stuff to help other people.

00:39:15.380 --> 00:39:19.720
So like you might as well try to make it easier for other people to help you as well.

00:39:19.720 --> 00:39:20.020
Yeah.

00:39:20.020 --> 00:39:22.320
It could definitely help your career as well.

00:39:22.320 --> 00:39:31.580
I mean, people like Wes McKinney, Jake VanderPlas, Travis Oliphant, like folks like that, like they're legitimate big names in the whole Python space in general.

00:39:31.580 --> 00:39:34.880
And a lot of that came from, you know, these academic projects and whatnot.

00:39:34.880 --> 00:39:35.540
So that's pretty cool.

00:39:35.540 --> 00:39:40.200
This portion of Talk Python To Me is brought to you by Rollbar.

00:39:40.200 --> 00:39:41.620
Got a question for you.

00:39:41.620 --> 00:39:44.340
Have you been outsourcing your bug discovery to your users?

00:39:44.340 --> 00:39:46.980
Have you been making them send you bug reports?

00:39:46.980 --> 00:39:48.680
You know, there's two problems with that.

00:39:48.680 --> 00:39:50.660
You can't discover all the bugs this way.

00:39:50.800 --> 00:39:53.200
And some users don't bother reporting bugs at all.

00:39:53.200 --> 00:39:55.100
They just leave, sometimes forever.

00:39:55.100 --> 00:39:58.920
The best software teams practice proactive error monitoring.

00:39:58.920 --> 00:40:07.300
They detect all the errors in their production apps and services in real time and debug important errors in minutes or hours, sometimes before users even notice.

00:40:07.300 --> 00:40:12.400
Teams from companies like Twilio, Instacart, and CircleCI use Rollbar to do this.

00:40:12.640 --> 00:40:18.480
With Rollbar, you get a real-time feed of all the errors so you know exactly what's broken in production.

00:40:18.480 --> 00:40:25.660
And Rollbar automatically collects all the relevant data and metadata you need to debug the errors so you don't have to sift through logs.

00:40:25.660 --> 00:40:29.620
If you aren't using Rollbar yet, they have a special offer for you, and it's really awesome.

00:40:30.020 --> 00:40:46.160
Sign up and install Rollbar at talkpython.fm/Rollbar, and Rollbar will send you a $100 gift card to use at the Open Collective, where you can donate to any of the 900-plus projects listed under the Open Source Collective or to the Women Who Code organization.

00:40:46.160 --> 00:40:49.780
Get notified of errors in real time and make a difference in Open Source.

00:40:50.040 --> 00:40:52.720
Visit talkpython.fm/Rollbar today.

00:40:52.720 --> 00:41:01.000
Before we move off, I don't want to drop this idea of code smells because, first of all, I love this concept.

00:41:01.000 --> 00:41:09.340
It's just such a good visualization of what can be wrong with software, but not broken with software.

00:41:09.340 --> 00:41:14.940
Because a lot of times you think of, well, my code now works, but what should I do?

00:41:14.940 --> 00:41:17.840
And I think the code smells is a very practical thing.

00:41:17.920 --> 00:41:23.160
Just for folks listening, like, code smells, the idea is the code is working.

00:41:23.160 --> 00:41:29.400
It's not broken, but when you look at it, you try to read it, like, your nose literally could kind of curl up.

00:41:29.400 --> 00:41:31.520
You're like, ew, there's something wrong with this.

00:41:31.520 --> 00:41:34.000
I guess it works, but I guess it's not good.

00:41:34.000 --> 00:41:35.580
It's really not good, right?

00:41:35.580 --> 00:41:37.600
Like a 300-line function, not good.

00:41:37.600 --> 00:41:40.680
Like, it works, but there's something wrong.

00:41:40.680 --> 00:41:46.420
And I knew this mostly from Martin Fowler's work back in 1999 when he wrote refactoring.

00:41:46.420 --> 00:41:50.440
And this was sort of the introduction to, like, how do you know when to refactor?

00:41:50.440 --> 00:41:54.080
Well, you look for the places that make your nose turn up.

00:41:54.080 --> 00:41:56.560
You go, ew, what do we do with this, right?

00:41:56.560 --> 00:41:58.320
Like, oh, there's a 300-line function.

00:41:58.320 --> 00:41:59.680
That's bad.

00:41:59.680 --> 00:42:00.960
What can we do about that?

00:42:01.000 --> 00:42:03.180
Or here's a function taking 20 parameters.

00:42:03.180 --> 00:42:04.660
That's really horrible.

00:42:04.660 --> 00:42:07.620
You know, it's really easy to switch this integer for that integer.

00:42:07.620 --> 00:42:09.240
And how do you know when that happens?

00:42:09.240 --> 00:42:10.640
So what could you do to make that better?

00:42:10.640 --> 00:42:11.840
And there's just a bunch of them.

00:42:11.840 --> 00:42:17.460
But I only know this through the sort of software engineering side of things.

00:42:17.460 --> 00:42:24.220
And this presentation that you talked about here, which was Jenny Bryan, right?

00:42:24.220 --> 00:42:28.620
She has some really interesting tips from the data science perspective, right?

00:42:28.620 --> 00:42:28.960
Yeah.

00:42:29.080 --> 00:42:33.140
Yeah, so the first one is do not comment or uncomment sections of your code to alter behavior

00:42:33.140 --> 00:42:35.200
because you want to try different stuff out.

00:42:35.200 --> 00:42:37.680
Yeah, and that's, like, a very common thing, right?

00:42:37.680 --> 00:42:43.880
Like, the easiest case where that happens is if you are in a collaboration environment,

00:42:43.880 --> 00:42:45.480
you have five people.

00:42:45.480 --> 00:42:52.560
You have five comments of data loading because everyone hardcoded, like, a data path, right?

00:42:52.560 --> 00:42:58.460
And so, like, there's literally, like, you comment in your code just to, like, load the data set

00:42:58.460 --> 00:43:00.660
across, like, depending on who you are, right?

00:43:00.660 --> 00:43:05.240
And then, like, you end up, like, if you end up using, like, some kind of version control system,

00:43:05.240 --> 00:43:10.760
like, the vast majority of your commits are just, like, it's my turn that ran it.

00:43:10.760 --> 00:43:15.880
And you just have this one bit of these couple of lines that are just, like, committing.

00:43:15.880 --> 00:43:16.520
Just cycling.

00:43:16.520 --> 00:43:17.760
Just cycling back and forth.

00:43:17.840 --> 00:43:19.220
Yeah, so, I mean, what is the fix, right?

00:43:19.220 --> 00:43:24.260
The fix would be to do something where you have this proper structure, as you already talked about,

00:43:24.260 --> 00:43:29.540
and then you use something like os.path or pathlib, and you compute the relative path over to that,

00:43:29.540 --> 00:43:32.000
and then you generate an absolute path.

00:43:32.000 --> 00:43:37.600
That would work for everybody as long as they all check out the same general structure, which sounds like Git.

00:43:37.620 --> 00:43:45.200
Yeah, and they dealt with this in the R world with these two packages called Rprojroot, like, for the root of an R project,

00:43:45.200 --> 00:43:52.060
and here, here as in, like, find this file using here as, like, the root path or something.

00:43:52.800 --> 00:43:55.600
And that's sort of, like, my contribution to all of this.

00:43:55.600 --> 00:44:02.700
I tried to pretty much, I wrote a package called PyPrideRoot that tries to mimic, like, the same functionality as well,

00:44:02.700 --> 00:44:07.480
because it works if you are working with scripts and stuff.

00:44:07.480 --> 00:44:12.580
But the second you have, like, some kind of folder structure where you have a Jupyter notebook,

00:44:12.580 --> 00:44:17.200
you'll sort of realize that, like, the Jupyter notebook doesn't care that you have a folder structure.

00:44:17.200 --> 00:44:21.340
Like, the second you're in it, like, the working directory is now wherever the Jupyter notebook is,

00:44:21.580 --> 00:44:25.320
not whatever folder structure you've, like, very carefully pieced together.

00:44:25.320 --> 00:44:28.080
And so this was, like, an attempt.

00:44:28.080 --> 00:44:30.180
It's not a very complicated function.

00:44:30.180 --> 00:44:32.920
It literally takes, like, oh, what is your working directory?

00:44:32.920 --> 00:44:40.120
And I'll recursively go up by its parent and checking for, like, special files like .git or a .here file.

00:44:40.120 --> 00:44:45.800
And then I'll pre-pen that to whatever path, just so, like, you can now use relative paths in a Jupyter notebook,

00:44:45.800 --> 00:44:47.480
just like you would in a script.

00:44:47.480 --> 00:44:51.560
So you can avoid that problem as well, the commenting in and out.

00:44:51.560 --> 00:44:52.440
Yeah, yeah, that's cool.

00:44:52.440 --> 00:44:54.860
And I'll definitely link to that project that you built.

00:44:54.860 --> 00:44:59.400
Tip two, use if and else in moderation, which seems pretty good.

00:44:59.400 --> 00:45:00.960
Number three is pretty straightforward.

00:45:00.960 --> 00:45:01.940
Use functions.

00:45:01.940 --> 00:45:03.880
I mean, just do.

00:45:03.880 --> 00:45:05.060
It's a good idea.

00:45:05.060 --> 00:45:05.720
You should do this.

00:45:05.780 --> 00:45:12.620
Yes, and, like, even when you're writing a function, it's okay to have a very complex function.

00:45:12.620 --> 00:45:16.820
And even complex functions don't need to be written all in one go, right?

00:45:16.820 --> 00:45:21.160
Like, you can break up your function, even though it does, like, a very complicated task.

00:45:21.160 --> 00:45:23.520
There's probably small subtasks.

00:45:23.520 --> 00:45:26.500
And your function can call other helper functions.

00:45:26.500 --> 00:45:30.040
It's not just like, oh, this is a really complicated thing.

00:45:30.040 --> 00:45:31.920
Let me just write a function for it.

00:45:31.920 --> 00:45:35.340
As you're writing the function for it, like, that's one of the other code smells as well.

00:45:35.340 --> 00:45:38.400
Like, if I have a hundred line function, like, that's kind of scary.

00:45:38.400 --> 00:45:40.440
You couldn't break this down into smaller pieces?

00:45:40.440 --> 00:45:41.700
Like, that's kind of weird.

00:45:41.840 --> 00:45:49.620
And so, like, having helper functions that feed into, like, a larger function is also how you fix that code smell.

00:45:49.620 --> 00:45:50.200
Yeah, absolutely.

00:45:50.200 --> 00:45:54.000
And obviously, that makes testing way easier because you test little bits.

00:45:54.000 --> 00:45:57.420
And then, you know, you test the kind of orchestration of them, and you're good.

00:45:57.420 --> 00:46:01.960
Another one that I'm a huge fan of, it's like a serious pet peeve of mine,

00:46:01.960 --> 00:46:07.520
is to have quick returns near the top or guarding clauses or guard clauses.

00:46:07.840 --> 00:46:13.000
If you've got a function that's, like, indented, and then it's got a loop, and then it's got an if,

00:46:13.000 --> 00:46:17.900
and then another if, and then another if, and it's just, like, way to the right.

00:46:17.900 --> 00:46:20.080
If you're scrolling to the right, you're doing it wrong.

00:46:20.080 --> 00:46:21.160
Yeah, yeah.

00:46:21.160 --> 00:46:26.280
And during PyCon, I actually, like, just bought your entire encyclopedia of training.

00:46:26.280 --> 00:46:28.440
And I forgot which one.

00:46:28.440 --> 00:46:31.740
I think it was the how to write your Python code, like, an experience.

00:46:31.740 --> 00:46:32.900
Pythonic code or something.

00:46:32.900 --> 00:46:33.920
Yeah, yeah, that one, huh?

00:46:33.920 --> 00:46:34.200
Yeah.

00:46:34.200 --> 00:46:35.720
So, like, I remember that chapter.

00:46:36.180 --> 00:46:38.240
Yeah, like, don't write nested if statements.

00:46:38.240 --> 00:46:42.020
Like, essentially, write them inside out so, like, it's flat.

00:46:42.020 --> 00:46:42.780
Yeah, exactly.

00:46:42.780 --> 00:46:43.780
Do them backwards.

00:46:43.780 --> 00:46:44.200
Yeah.

00:46:44.200 --> 00:46:44.780
Yeah.

00:46:44.980 --> 00:46:48.400
So, that was something that was just, like, that's what you should do.

00:46:48.400 --> 00:46:50.920
It makes it so clear.

00:46:50.920 --> 00:46:54.000
And, like, it's not very commonly taught, I don't believe.

00:46:54.000 --> 00:46:55.280
So, these are called guarding clauses.

00:46:55.280 --> 00:47:00.040
And the idea is instead of testing for a good condition and then another good condition and

00:47:00.040 --> 00:47:04.320
another good condition and then doing the thing, which puts everything way on the inside, you

00:47:04.320 --> 00:47:07.400
test for all the bad conditions first and you just bail out.

00:47:07.960 --> 00:47:11.940
And then what you're left with is a non-indented simple bit of code, which is what you're actually

00:47:11.940 --> 00:47:12.240
after.

00:47:12.240 --> 00:47:14.760
So, it's really clear what you're testing against.

00:47:14.760 --> 00:47:17.600
And then once you're past that, here's the simple thing we do.

00:47:17.600 --> 00:47:18.200
I love it.

00:47:18.200 --> 00:47:19.720
So, that was one of her tips as well.

00:47:19.720 --> 00:47:20.420
It's a nice one.

00:47:20.420 --> 00:47:21.640
Yeah.

00:47:21.800 --> 00:47:23.940
She's got some great little examples there.

00:47:23.940 --> 00:47:26.180
Some stuff on object orienting and so on.

00:47:26.180 --> 00:47:27.320
But, yeah, these are really good.

00:47:27.320 --> 00:47:29.920
I, you know, switch, which doesn't apply as much to Python.

00:47:29.920 --> 00:47:35.140
I actually wrote a switch language extension for Python using the context management with

00:47:35.140 --> 00:47:35.420
block.

00:47:35.420 --> 00:47:36.080
That's pretty awesome.

00:47:36.080 --> 00:47:38.560
But I'm not going to get into that because that's a whole different debate.

00:47:38.560 --> 00:47:41.300
But I do think this idea of code smells is really interesting.

00:47:41.300 --> 00:47:45.260
And you should think about them for data science because I'm sure there are different.

00:47:45.260 --> 00:47:49.460
It sounds like, it looks like there are different data standout smells that are more common

00:47:49.460 --> 00:47:51.280
than, say, standard software engineering.

00:47:51.380 --> 00:47:54.240
If you're doing database programming or whatever, you get like a different style there.

00:47:54.240 --> 00:47:54.460
Yeah.

00:47:54.460 --> 00:48:00.340
And just for like other programming related things and how you can like structure your projects,

00:48:00.340 --> 00:48:03.920
Jenny Bryan also has this talk about like, how do you name your files?

00:48:03.920 --> 00:48:08.940
It's kind of interesting because like I, if you think about these common problems long enough,

00:48:08.940 --> 00:48:12.540
everyone pretty much just converges to like the same set of solutions.

00:48:12.540 --> 00:48:17.660
I remember like coming up with, yeah, I should just name things this way or like set up my

00:48:17.660 --> 00:48:18.320
folder this way.

00:48:18.320 --> 00:48:21.280
And then like all of a sudden, Jenny Bryan like gives a talk at like,

00:48:21.360 --> 00:48:22.140
big R conference.

00:48:22.140 --> 00:48:25.840
Like, wait, that was like, I feel validated that like I didn't come up with something like

00:48:25.840 --> 00:48:26.520
nonsensical.

00:48:26.520 --> 00:48:31.280
Other people as well, like they write packages sort of like a cookie cutter, just like set

00:48:31.280 --> 00:48:31.780
up projects.

00:48:31.780 --> 00:48:33.180
And it's pretty much like the same way.

00:48:33.180 --> 00:48:36.000
And one of them is like, oh, how do you name your files?

00:48:36.000 --> 00:48:36.320
Right.

00:48:36.320 --> 00:48:40.860
Like, and especially in analytics, there's clearly an order you should run this stuff

00:48:40.860 --> 00:48:41.120
in.

00:48:41.120 --> 00:48:47.000
So one of the ways of like, how do you name your files is pre-pen a number to them.

00:48:47.000 --> 00:48:47.320
Right.

00:48:47.320 --> 00:48:51.160
So like you can say like one dash and then like the script and that's the order you write

00:48:51.160 --> 00:48:51.500
it in.

00:48:51.500 --> 00:48:54.040
If you want to do better, you say zero one.

00:48:54.200 --> 00:48:56.880
So like 10 and one doesn't get sorted in properly.

00:48:56.880 --> 00:48:57.380
Yeah.

00:48:57.380 --> 00:49:02.580
And then if you really want to go one step further, I started this habit of like having

00:49:02.580 --> 00:49:03.500
a three digit number.

00:49:03.500 --> 00:49:08.800
So like zero one zero, and that gives you a buffer room to like insert something in

00:49:08.800 --> 00:49:09.220
the middle.

00:49:09.220 --> 00:49:12.480
Or if you like forget something or like you realize that.

00:49:12.480 --> 00:49:14.580
That's like the 10, 20, 30 in basic.

00:49:14.580 --> 00:49:15.000
Yeah.

00:49:15.000 --> 00:49:17.580
Like, what if you got to put a line in between that?

00:49:17.580 --> 00:49:18.660
You got to go to 30 still.

00:49:18.660 --> 00:49:20.160
Well, you could do 19.

00:49:20.160 --> 00:49:20.500
Yeah.

00:49:20.500 --> 00:49:20.800
Whatever.

00:49:20.800 --> 00:49:21.100
Yeah.

00:49:21.100 --> 00:49:26.720
And I found that out because like, that's how sort of some of the files in Linux in the

00:49:26.720 --> 00:49:29.540
order of like how it loads up like services or something.

00:49:29.540 --> 00:49:31.860
It's like defined in like those three digit numbers.

00:49:32.320 --> 00:49:34.340
And I was like, oh, this is interesting.

00:49:34.340 --> 00:49:35.220
I should do that.

00:49:35.220 --> 00:49:37.900
It just saves me from like renumbering like a whole bunch of stuff.

00:49:37.900 --> 00:49:38.080
Yeah.

00:49:38.080 --> 00:49:38.440
That's cool.

00:49:38.440 --> 00:49:40.900
I mean, just thinking about the structure is quite interesting.

00:49:40.900 --> 00:49:45.440
At the end of the day, even though you have all this structure for your analytics project,

00:49:45.440 --> 00:49:51.720
because everything is like nice and in some kind of order, if you do, for example, want

00:49:51.720 --> 00:49:54.780
to create a Python package, like it's already there for you, right?

00:49:54.780 --> 00:49:56.860
Like you can create another folder.

00:49:56.860 --> 00:49:58.020
That's the name of your module.

00:49:58.020 --> 00:49:59.660
Put a setup that PY file.

00:50:00.360 --> 00:50:02.700
You could have the ability to set that up.

00:50:02.700 --> 00:50:04.960
And now you can like pip install dash E.

00:50:04.960 --> 00:50:08.820
And then anytime you edit that file, like your analysis will still work.

00:50:08.820 --> 00:50:10.100
And that's pretty cool.

00:50:10.100 --> 00:50:14.960
The other thing with project structure related stuff, like if you have things numbered at

00:50:14.960 --> 00:50:18.920
the end of the day, everything comes down to like a DAG compute system.

00:50:19.040 --> 00:50:23.180
And so like, because you have your stuff in order and there's properly defined inputs and

00:50:23.180 --> 00:50:28.360
outputs, you can use like a make file or like a simple script as like a poor man's make file.

00:50:28.360 --> 00:50:33.720
But then you end up in like the situation like, oh, that's where Luigi and Airflow come into

00:50:33.720 --> 00:50:34.020
play.

00:50:34.020 --> 00:50:35.940
They're pretty much just DAG executors.

00:50:36.220 --> 00:50:41.120
Like I said, at the very beginning, setting up your project is pretty much like the gateway

00:50:41.120 --> 00:50:43.820
drug into like all of this other cool technology.

00:50:43.820 --> 00:50:48.980
Cause like you've, you would have set everything up in such a way that you then use those tools

00:50:48.980 --> 00:50:51.220
when you hit that point where you need it.

00:50:51.220 --> 00:50:55.960
And it's like a nice way to like slowly improve, do self-improvement stuff.

00:50:55.960 --> 00:51:00.100
And then you also like end up using all the cool stuff that you see at like these big conferences

00:51:00.100 --> 00:51:00.680
as well.

00:51:00.880 --> 00:51:02.200
Yeah, that's, that's really cool.

00:51:02.200 --> 00:51:05.860
And of course the structure gets you just that much closer to trying it out.

00:51:05.860 --> 00:51:10.460
Now, what do you think about Papermill and some of these concepts?

00:51:10.460 --> 00:51:11.420
Are you familiar with Papermill?

00:51:11.420 --> 00:51:11.620
Yeah.

00:51:11.620 --> 00:51:14.080
Papermill is, I think that's the Netflix.

00:51:14.080 --> 00:51:14.780
Yes.

00:51:14.780 --> 00:51:19.540
It lets you basically turn a Jupyter notebook into something that can receive inputs and then

00:51:19.540 --> 00:51:22.940
have outputs almost like a function or a module or something like that.

00:51:22.940 --> 00:51:25.180
So I personally haven't used it.

00:51:25.180 --> 00:51:30.500
That's mainly because when I was started working, like Papermill wasn't really a thing at the

00:51:30.500 --> 00:51:30.700
point.

00:51:30.700 --> 00:51:35.640
So like I had migrated out into like, let's just make everything a Python script because

00:51:35.640 --> 00:51:40.040
that has no dependency and we can just execute things that way.

00:51:40.040 --> 00:51:44.280
And then the notebook itself just becomes like, Hey, this is the report.

00:51:44.280 --> 00:51:49.480
In some sense, I can see if I, for me, I guess like the next time I start an analysis project,

00:51:49.480 --> 00:51:53.300
like I probably will use Papermill just because it's like, Oh, it's this cool technology.

00:51:53.300 --> 00:51:57.980
And I've like set up my folder structures in such a way where like I can now use it.

00:51:58.040 --> 00:51:58.240
Right.

00:51:58.240 --> 00:52:01.320
So I've heard of it, but I personally haven't used it yet.

00:52:01.320 --> 00:52:01.560
Yeah.

00:52:01.560 --> 00:52:03.920
I haven't used it either, but it sounds pretty interesting.

00:52:03.920 --> 00:52:07.800
Like it sounds like Netflix, like you said, is doing really interesting stuff to me.

00:52:07.800 --> 00:52:12.740
One of the things that sounded special, it made me go, okay, well maybe that is worth considering,

00:52:12.740 --> 00:52:15.060
even though it's like not necessarily my style, right?

00:52:15.060 --> 00:52:20.800
is if you have a big, long sort of pipeline of operations and each one is its own Jupyter

00:52:20.800 --> 00:52:21.120
notebook.

00:52:21.120 --> 00:52:28.020
If it fails, you can save, you basically keep the notebook as it was computed laying around.

00:52:28.020 --> 00:52:32.040
So you can just open it up and you have basically a history of what happened and then what failed,

00:52:32.040 --> 00:52:34.020
which sounds like a pretty interesting way.

00:52:34.060 --> 00:52:38.640
Cause if you switch it to scripts, which I'm all for, but you end up with, you know,

00:52:38.640 --> 00:52:41.440
it exited without, with like not code zero.

00:52:41.440 --> 00:52:42.940
Oh, that's bad.

00:52:42.940 --> 00:52:43.260
Right.

00:52:43.260 --> 00:52:44.200
Like, what does that mean?

00:52:44.200 --> 00:52:46.920
Like, I forgot, I don't even have logging or any of these things, right?

00:52:46.920 --> 00:52:47.480
Like what happened?

00:52:47.480 --> 00:52:48.500
Like, why did it not work?

00:52:48.500 --> 00:52:52.300
So I do think there's some interesting stuff happening around there, but I do also

00:52:52.300 --> 00:52:59.780
feel like the software engineering tools you have apply really well to modules, right?

00:52:59.780 --> 00:53:01.940
Like it's easy to run that through pytest.

00:53:01.940 --> 00:53:07.160
It's easy to run that through a profiler, the refactoring tools work on those.

00:53:07.160 --> 00:53:13.200
Not that you can't do some of that stuff with notebooks, but it's easier to use them on files.

00:53:13.200 --> 00:53:13.600
Yeah.

00:53:13.600 --> 00:53:18.440
And especially if you're checking things into version control, that's sort of like the one

00:53:18.440 --> 00:53:18.740
thing.

00:53:18.740 --> 00:53:24.140
My main gripe with the notebooks is like, every time I make a change, like I have no

00:53:24.140 --> 00:53:28.180
idea what's going on in diff and it's just like, yeah, just add and commit.

00:53:28.180 --> 00:53:29.560
Like, I think it's right.

00:53:29.560 --> 00:53:31.280
Let's see.

00:53:31.280 --> 00:53:33.420
Do you accept their changes or your changes?

00:53:33.420 --> 00:53:34.340
my changes.

00:53:34.340 --> 00:53:38.840
Or like, or like if I just want to open the notebook.

00:53:38.840 --> 00:53:43.680
So there's this program called enter act, which at least is like a, a desktop version.

00:53:43.680 --> 00:53:48.320
So I don't have to like open up fire a server and then open a notebook.

00:53:48.380 --> 00:53:48.820
That way.

00:53:48.820 --> 00:53:52.140
But yeah, like sometimes like, I just want to double click this thing just to see it.

00:53:52.140 --> 00:53:56.440
I don't like want to open up a terminal and like launch everything just to see something.

00:53:56.440 --> 00:54:00.860
So it was like little things like that, where I was like, I'll try to do as much as I can

00:54:00.860 --> 00:54:01.320
in a script.

00:54:01.320 --> 00:54:03.780
And then like everything else goes into a notebook.

00:54:03.780 --> 00:54:08.500
And then in the notebook, I still save out the things I want just so like I have an easier

00:54:08.500 --> 00:54:13.340
way to like access figures or tables without having to like look at the entire notebook.

00:54:13.340 --> 00:54:13.580
Yeah.

00:54:13.580 --> 00:54:16.140
I guess that is one of the challenges is the whole diff thing.

00:54:16.140 --> 00:54:16.860
Maybe we could talk.

00:54:17.000 --> 00:54:19.740
We're kind of getting long on time, but there's a lot of interesting stuff to cover.

00:54:19.740 --> 00:54:20.920
So I'll ask you a few more questions.

00:54:20.920 --> 00:54:23.020
Let's think a little bit about collaboration.

00:54:23.020 --> 00:54:32.020
Like you talked about the anti-pattern of having like Sarah's path, Dan's path, Michael's path,

00:54:32.020 --> 00:54:36.400
whatever, like, and just commenting them out which one is active at the moment.

00:54:36.400 --> 00:54:40.820
But there's probably some other stuff for collaboration, like are you using Git?

00:54:40.820 --> 00:54:46.000
Are you using some online shared notebook that's kind of like Google Docs?

00:54:46.000 --> 00:54:48.680
Like what are your thoughts around that kind of stuff?

00:54:48.680 --> 00:54:53.740
So Google has something called like the Co-Laboratory Notebook, which is essentially like Google Docs,

00:54:53.740 --> 00:54:56.020
but gives you a Jupyter Notebook system.

00:54:56.020 --> 00:55:02.200
That's pretty cool in the sense that like, yeah, we won't have this commenting out of like random

00:55:02.200 --> 00:55:04.940
lines because everyone's really just working on the same place.

00:55:04.940 --> 00:55:06.560
Like that's really nice for collaboration.

00:55:06.560 --> 00:55:10.680
I still think that you need some form of version control.

00:55:10.680 --> 00:55:15.060
Like that is, I think like at this day and age, like it's pretty much required, especially

00:55:15.060 --> 00:55:18.440
when programs start to get more and more complex.

00:55:18.440 --> 00:55:20.620
Like you need a way to fall back on.

00:55:20.620 --> 00:55:26.440
The nicest feature I use in Git is like I write something, everything is broken and I just say

00:55:26.440 --> 00:55:30.580
Git reset and I just pretend I never did that and I just start over.

00:55:30.580 --> 00:55:32.720
Yes, exactly.

00:55:32.720 --> 00:55:34.000
Like that was a really bad idea.

00:55:34.000 --> 00:55:34.800
Please revert that.

00:55:34.800 --> 00:55:35.060
Okay.

00:55:35.060 --> 00:55:35.980
Now we're good.

00:55:35.980 --> 00:55:38.760
And it lets you be more exploratory.

00:55:38.760 --> 00:55:41.860
It lets you be more aggressive and trying to change it.

00:55:41.860 --> 00:55:43.880
Like this might not work, but if it works, it's going to be awesome.

00:55:43.880 --> 00:55:45.180
And try it.

00:55:45.180 --> 00:55:46.240
Actually, that didn't work.

00:55:46.240 --> 00:55:46.960
Revert.

00:55:46.960 --> 00:55:49.040
Or, you know, maybe it's a little more forethought.

00:55:49.040 --> 00:55:50.900
You create a feature branch to explore it.

00:55:50.900 --> 00:55:52.140
You do it there.

00:55:52.140 --> 00:55:53.540
And they're like, forget that.

00:55:53.540 --> 00:55:54.260
That was a bad branch.

00:55:54.260 --> 00:55:55.140
We're just going back here.

00:55:55.140 --> 00:55:56.160
Like, let's not do that.

00:55:56.160 --> 00:55:56.380
Right.

00:55:56.380 --> 00:55:56.700
Yeah.

00:55:56.700 --> 00:55:57.620
But it's a really great feature.

00:55:57.720 --> 00:55:57.840
Yeah.

00:55:57.840 --> 00:55:57.880
Yeah.

00:55:57.880 --> 00:56:03.100
And like, just along the lines of like collaboration stuff, like make small incremental changes.

00:56:03.100 --> 00:56:04.840
And that's like the actual stuff.

00:56:04.840 --> 00:56:07.100
That's the code that will actually get reviewed.

00:56:07.100 --> 00:56:07.360
Right.

00:56:07.360 --> 00:56:15.340
Like no one will review a code base where you're like at the end of the paper, the entire like submission relies on this code base.

00:56:15.340 --> 00:56:17.560
And you're like, I need someone to review this thing.

00:56:17.560 --> 00:56:17.920
Right.

00:56:17.920 --> 00:56:21.160
Like there's no way that's going to get a proper review.

00:56:21.160 --> 00:56:26.720
And so just in general for like, doesn't even have to be in like research or science.

00:56:26.720 --> 00:56:29.480
It's a good habit to like make small incremental changes.

00:56:29.480 --> 00:56:32.340
And like, maybe that's what your weekly meeting is about.

00:56:32.340 --> 00:56:34.360
It's just like, this is what I did this week.

00:56:34.980 --> 00:56:40.180
Someone press the green button to merge this in because that will actually be reviewed.

00:56:40.180 --> 00:56:42.720
And then you'll have a discussion around that point.

00:56:42.720 --> 00:56:43.960
Like all of that stuff.

00:56:43.960 --> 00:56:47.000
For me, I personally, I'm not in a managerial position.

00:56:47.000 --> 00:56:51.560
So like, those are the types of meetings I find like productive where I can actually talk about.

00:56:51.560 --> 00:56:52.500
This is why I did.

00:56:52.500 --> 00:56:53.520
This was the implementation.

00:56:53.520 --> 00:56:55.400
This is what I'm thinking about next.

00:56:55.400 --> 00:57:01.420
And then have a conversation around that because it can still be productive and you can still have like talks about longer goals.

00:57:01.420 --> 00:57:08.440
But like you also now have the benefit of like someone else looking at your work to make sure it doesn't have like a bad code smell.

00:57:08.440 --> 00:57:14.560
You know, maybe like you, you, you're off by like a factor of 10 and no one's going to notice that in like 900 lines of code.

00:57:14.560 --> 00:57:19.500
But they will if it's just like 20 lines of code, like a change like that is much easier to find.

00:57:19.620 --> 00:57:21.160
Yeah, that's definitely good advice.

00:57:21.160 --> 00:57:27.000
I definitely recommend working in small little, little bits and changes and, you know, make some small change.

00:57:27.000 --> 00:57:27.740
Do a git commit.

00:57:27.740 --> 00:57:29.200
Make another small change.

00:57:29.200 --> 00:57:30.120
A little git commit, right?

00:57:30.120 --> 00:57:34.480
Like don't wait until the end of the week or like until the end of the paper and like, all right, time to check it in.

00:57:34.480 --> 00:57:36.620
Like, no, not a good idea.

00:57:36.620 --> 00:57:43.780
One of the things I wish like exists more in academia is just having more resources to do pair programming.

00:57:43.780 --> 00:57:51.100
Because usually people are assigned one project and there isn't like two people assigned to the exact same bit, which is what you really need pair programming for.

00:57:51.440 --> 00:58:02.940
When I was co-instructing like the summer program in my previous lab, I would sit down next to students and I would pair program them through some kind of data related work.

00:58:02.940 --> 00:58:08.620
And it's super valuable for them because they actually get to see how I'm thinking about like this problem.

00:58:08.620 --> 00:58:11.400
And I'll say like, you're doing a join of two tables.

00:58:11.400 --> 00:58:16.020
Yeah, make sure that like the keys don't have duplicates if you're expecting duplicates, right?

00:58:16.020 --> 00:58:19.860
That's like one of those things of like, yeah, the code ran, so I'm just going to keep going, right?

00:58:19.860 --> 00:58:28.120
And you don't realize that you just did, you just did a Cartesian product and now you have a million rows and you don't know why, but you're just going to keep going.

00:58:28.120 --> 00:58:29.200
Why is it taking so long?

00:58:29.200 --> 00:58:29.700
Yeah.

00:58:29.700 --> 00:58:32.740
So pair programming, yeah, it's super valuable.

00:58:32.740 --> 00:58:38.860
And even now during my internship, it's I'm on the receiving end of pair programming, but this is more on the software engineering side.

00:58:38.860 --> 00:58:45.420
It's super valuable just to see like, oh yeah, this is how you write good code or like, this is how they're thinking about it.

00:58:45.420 --> 00:58:50.680
And it's even stuff like I talk about, yeah, be careful where you're hitting like control V a bunch of times.

00:58:50.680 --> 00:58:53.760
And it's like, oh yeah, like this is in two different places.

00:58:53.760 --> 00:58:55.340
Like, let's just refactor this out.

00:58:55.340 --> 00:58:56.600
And it's like, oh yeah, I didn't catch that.

00:58:56.600 --> 00:59:04.420
And like, when you refactor it out, you can actually have more guarding clauses just to make this like an even better check.

00:59:04.840 --> 00:59:11.680
That's one of the things I wish, like at least in research, like there was more budget and time for is just pair programming.

00:59:11.680 --> 00:59:16.800
And that just makes collaboration easier because you're now just talking with a person back and forth.

00:59:16.800 --> 00:59:20.240
It just makes that whole process like way nicer and smoother.

00:59:20.240 --> 00:59:20.660
Yeah.

00:59:20.660 --> 00:59:23.000
I mean, we certainly have the tools these days for it, right?

00:59:23.000 --> 00:59:29.440
You talked about Google Colaboratory, which has like live multiple editor features, kind of like Google Docs.

00:59:30.020 --> 00:59:40.300
You've got obviously screen sharing, you've got like VS Codes, ways to like watch somebody else's system on two sets of Visual Studio code.

00:59:40.300 --> 00:59:41.860
And there's some really interesting options.

00:59:41.860 --> 00:59:44.040
But yeah, it's got to, it's like also a cultural thing.

00:59:44.040 --> 00:59:48.960
And also you've got to have people to collaborate with on that part, right?

00:59:48.960 --> 00:59:49.280
Right.

00:59:49.280 --> 00:59:56.480
And in the sense of, hey, maybe like when you, even though you're in this small world and you write your package, like now you have someone to collaborate with, right?

00:59:56.480 --> 01:00:01.560
And that's sort of like socially motivating that you have other people using your stuff.

01:00:01.560 --> 01:00:01.800
Yeah.

01:00:01.800 --> 01:00:11.040
It definitely feels good to have someone looking at it, interacting with what you're building because building software completely in isolation just for yourself.

01:00:11.040 --> 01:00:12.320
It's kind of a weird place to be.

01:00:12.320 --> 01:00:13.740
It's not as much fun as it could be.

01:00:13.740 --> 01:00:13.980
Yeah.

01:00:13.980 --> 01:00:14.600
It's fun.

01:00:14.600 --> 01:00:20.040
Like when you're just in the sense of like, I got to get something like that minimum viable product, like that's fun.

01:00:20.040 --> 01:00:24.740
And then it's just like, as soon as you hit maintenance mode, it's like, who am I maintaining this for?

01:00:26.600 --> 01:00:26.960
Yeah.

01:00:26.960 --> 01:00:35.660
Or just all like a lot of the projects, you know, you're going to be working on it and you kind of get the happy path, mostly working and you feel like you're mostly done.

01:00:35.660 --> 01:00:44.640
But then there's all these little loose ends, the documentation you got to write for the other people involved, all the little tests and the edge cases.

01:00:44.640 --> 01:00:46.340
And just, it can just go on and on and on.

01:00:46.340 --> 01:00:49.700
It feels like, I thought I was done a month ago with this and I'm still working on it.

01:00:49.700 --> 01:00:51.600
How is this not still not done yet?

01:00:51.600 --> 01:00:58.020
Like I've definitely had that feeling in software and I'm sure it's just the same, you know, that was actually in a semi-research context.

01:00:58.020 --> 01:00:59.580
I'm thinking back to it.

01:00:59.580 --> 01:00:59.980
Yeah.

01:00:59.980 --> 01:01:03.060
Final thought on this collaboration bit.

01:01:03.060 --> 01:01:04.260
What do you think about GitHub?

01:01:04.260 --> 01:01:10.440
Like creating either a private or a public repo, using that for your work to share with people?

01:01:10.780 --> 01:01:12.160
I love it right now.

01:01:12.160 --> 01:01:15.200
Like pretty much if I have a thought, I just make a GitHub repo.

01:01:15.200 --> 01:01:21.560
So like my personal GitHub account has a bunch of projects where like they're pretty much empty, but they have a name.

01:01:21.560 --> 01:01:26.500
And it's just because like I thought of something one day and I just made a repo out of it.

01:01:26.500 --> 01:01:28.720
It's even really good for simple stuff.

01:01:28.780 --> 01:01:34.760
Like if you're at a conference and you just want a place to take notes, that doesn't matter what machine you're on.

01:01:34.760 --> 01:01:38.400
I've taken just notes and markdown as a GitHub repository.

01:01:38.400 --> 01:01:42.680
And then like during like a lightning talk, just be like, hey, I just started putting up my notes.

01:01:42.680 --> 01:01:46.380
And then maybe some people will like add, hey, wait, this is my talk.

01:01:46.380 --> 01:01:47.740
Let me put my talk in there.

01:01:47.920 --> 01:01:53.420
And you end up collaborating on like some kind of notes for like a conference, which is pretty cool.

01:01:53.420 --> 01:02:03.820
And for me, like I try to in lines of that 10% improvement, like every time, like originally, like I just made everything in Git just because I needed more practice with it.

01:02:03.820 --> 01:02:07.660
And it was just like a nice safe place for me to like, oh yeah, like add and commit.

01:02:07.660 --> 01:02:11.820
Like if you do it a couple of hundred times, that part doesn't become scary anymore.

01:02:11.820 --> 01:02:13.600
And so that's right.

01:02:13.600 --> 01:02:15.260
It just becomes so natural.

01:02:15.260 --> 01:02:17.800
Like, oh yeah, when I first learned Git, it's like, why am I doing this?

01:02:17.820 --> 01:02:19.040
This is so tedious.

01:02:19.040 --> 01:02:20.680
And then it's like, now it's like, okay, whatever.

01:02:20.680 --> 01:02:24.340
But then like you can do other stuff with Git, which is like super cool.

01:02:24.340 --> 01:02:33.420
So GitHub is like a great way to practice using Git and then also gives you the ability to practice or get ready for collaboration.

01:02:33.420 --> 01:02:34.040
Right.

01:02:34.040 --> 01:02:45.460
So even for me, even if I'm working on personal projects, sometimes like I will do branches for myself, push branches to GitHub by myself, and I will submit pull requests to myself.

01:02:47.460 --> 01:02:49.580
Just to document it and make it really clear.

01:02:49.580 --> 01:02:51.140
Like this is the reason for it here.

01:02:51.140 --> 01:02:52.460
The files that changed and all that.

01:02:52.460 --> 01:02:52.640
Right.

01:02:52.680 --> 01:02:54.480
And like, I was doing that for a couple of years.

01:02:54.480 --> 01:03:04.020
And like now, like during my internship, like that has become so second nature that like I can actually do Git things and it doesn't hinder collaborating in like the real world.

01:03:04.020 --> 01:03:04.240
Yeah.

01:03:04.240 --> 01:03:07.300
So it was a lot of like just practice that like, I just thought it was cool.

01:03:07.300 --> 01:03:12.920
Like, I didn't realize until now that was like, wait, like this is actually just like years of practicing on my own.

01:03:13.260 --> 01:03:20.960
And so like, in that sense, like, and like Microsoft essentially saved GitHub and like, it's just as good as ever.

01:03:20.960 --> 01:03:24.940
So like, yeah, plus plus one for GitHub all the way.

01:03:24.940 --> 01:03:25.940
Yeah.

01:03:25.940 --> 01:03:26.260
Awesome.

01:03:26.260 --> 01:03:27.240
I totally agree.

01:03:27.240 --> 01:03:27.840
I totally agree.

01:03:27.840 --> 01:03:28.580
Okay.

01:03:28.780 --> 01:03:30.500
This is really interesting.

01:03:30.500 --> 01:03:32.840
I think there's a lot of concrete advice here.

01:03:32.840 --> 01:03:34.600
I'll link to the papers.

01:03:34.600 --> 01:03:37.540
I'll link to your PyRoot project.

01:03:37.540 --> 01:03:39.900
The code smells thing, all that.

01:03:39.900 --> 01:03:45.720
We'll put all this up there and people can come back and definitely dig into the details if that's useful for them.

01:03:46.080 --> 01:03:51.360
So before we get to the final bit of the show, though, I've got to ask you the two questions, Dan.

01:03:51.360 --> 01:03:55.060
First of all, if you're going to write some Python code, what editor do you use?

01:03:55.060 --> 01:04:01.720
So I used to use Emacs with LPy and now I am now a VS Code convert.

01:04:01.720 --> 01:04:03.140
They've brought you over.

01:04:03.140 --> 01:04:08.040
You know, I would say like the last four shows that I've had, everyone has said VS Code, which is pretty interesting.

01:04:08.040 --> 01:04:08.360
Yeah.

01:04:08.620 --> 01:04:21.660
I was pretty reluctant until like I had to write some Python code and I was on, I switched over to my Windows machine and I was like, I don't have any way to edit code right now.

01:04:21.660 --> 01:04:23.000
Let's just try this thing.

01:04:23.000 --> 01:04:25.700
And, you know, it worked.

01:04:25.700 --> 01:04:27.620
And so like I was pretty happy with it.

01:04:27.620 --> 01:04:29.860
So I sort of just hung around.

01:04:29.860 --> 01:04:36.980
What's actually really cool is the screen sharing ability in VS Code that does pair programming.

01:04:36.980 --> 01:04:37.440
Yes.

01:04:37.700 --> 01:04:39.840
That live, I think it's called Live Share.

01:04:39.840 --> 01:04:42.460
I've never had a good chance to use it, but I've seen it and it looks amazing.

01:04:42.460 --> 01:04:47.180
Yeah, I've used it with one of the other interns and it's like, this is really cool.

01:04:47.180 --> 01:04:49.840
And they also have like a voice communication mechanism.

01:04:49.840 --> 01:04:57.780
So like yet another way to like do voice chat, but at least the screen, like the live coding part, like that was super cool.

01:04:57.780 --> 01:04:58.360
Very nice.

01:04:58.360 --> 01:04:58.600
Yeah.

01:04:58.600 --> 01:04:59.500
Okay, great.

01:04:59.500 --> 01:05:01.480
Definitely a good answer for the editor.

01:05:01.480 --> 01:05:02.980
Packages, some notable ones.

01:05:02.980 --> 01:05:08.800
The package that notable that I haven't heard on the show yet is one called Pie Janitor by Eric Ma.

01:05:08.800 --> 01:05:11.920
And he works at Novartis.

01:05:11.920 --> 01:05:18.660
And this is pretty much his consolidation of pretty common data cleaning stuff in pandas.

01:05:19.060 --> 01:05:27.860
And that ties to another package by Zachary Saylor called Pandas Flavor, which is a wrapper around your ability to extend pandas.

01:05:28.200 --> 01:05:36.020
And the benefit of that is, you know, if you want pandas to have a method that you don't already have, like you might think like, oh, let me create another class.

01:05:36.020 --> 01:05:38.520
I'll inherit pandas and I'll release a package.

01:05:38.800 --> 01:05:42.200
But no one's really going to use that because it's not a pandas data frame object.

01:05:42.200 --> 01:05:44.760
It's like some weird class that you created yourself.

01:05:44.760 --> 01:05:57.160
And so like this is sort of like a mechanism for you to inject your own methods into a pandas data frame object, but still have a pandas data frame object without having to re-extend the class.

01:05:57.160 --> 01:05:58.520
So it's super cool.

01:05:58.520 --> 01:05:59.520
Yeah, that's really great.

01:05:59.520 --> 01:06:02.500
And yeah, the Pie Janitor, I really like that one.

01:06:02.500 --> 01:06:18.400
It takes a whole bunch of imperative data frame operations and turns it into a really nice fluent API like data frame dot from dictionary dot remove columns dot drop not a number drop, you know, rename call and just boom, just flows it all together.

01:06:18.400 --> 01:06:19.120
It's really nice.

01:06:19.120 --> 01:06:23.660
I haven't covered on the show, but we did talk about it over on Python bytes that podcast.

01:06:23.660 --> 01:06:25.180
So, yeah, it's definitely a cool one.

01:06:25.180 --> 01:06:26.300
It's been on my radar as well.

01:06:26.300 --> 01:06:27.160
Nice.

01:06:27.160 --> 01:06:27.560
All right.

01:06:27.560 --> 01:06:28.920
Well, final call to action.

01:06:28.920 --> 01:06:31.640
People who are out there, maybe they're in science, data science.

01:06:31.920 --> 01:06:33.440
It's something like that.

01:06:33.440 --> 01:06:42.480
And they want to make their code take, you know, that 10% step you're talking about towards the more proper engineering structured world.

01:06:42.480 --> 01:06:43.160
What do they do?

01:06:43.160 --> 01:06:46.680
For me, like I was lucky enough to be in New York City, which is a big city.

01:06:46.680 --> 01:06:53.820
So it was always like local meetups were always like a thing that were very busy and you learn a lot from there.

01:06:53.820 --> 01:07:01.560
But even if you don't live in a very big city, you can either start one yourself because chances are you are not alone.

01:07:01.560 --> 01:07:04.360
And the Python community is super supportive.

01:07:04.360 --> 01:07:10.400
You can always if you say something on Twitter, someone will give you the ways of how to start something.

01:07:10.800 --> 01:07:16.340
And if you're at a university, you can always have meetings in like a classroom or something.

01:07:16.340 --> 01:07:17.640
So don't worry.

01:07:17.640 --> 01:07:17.900
Right.

01:07:17.900 --> 01:07:20.100
Maybe it has an interdisciplinary, right?

01:07:20.100 --> 01:07:21.920
Like maybe there's not that many people in your department.

01:07:21.920 --> 01:07:26.600
But if you go across, you could probably find a decent number of folks you want to attend.

01:07:26.600 --> 01:07:26.900
Yeah.

01:07:26.900 --> 01:07:33.980
And so meetups are a great way to like learn or meet other people or at least just like ask questions about stuff.

01:07:33.980 --> 01:07:46.500
And if you can make it to like any of the Python conferences or like attend a sprint like that is probably like going to a sprint was like the fastest way that I've became a better Python programmer.

01:07:46.500 --> 01:08:02.140
Or even if it was something as like editing a piece of documentation, like just seeing the mechanism of how other people collaborate on such a large scale and then still seeing your work like in one of these major projects like that's super motivating and like cool.

01:08:02.140 --> 01:08:02.980
Yeah, that's really cool.

01:08:02.980 --> 01:08:04.100
Yeah, it's a great opportunity.

01:08:04.100 --> 01:08:12.860
And it's also a great opportunity to, you know, rub shoulders with really prominent people in something that you're working with, right?

01:08:12.900 --> 01:08:23.960
The maintainers of this probably important project who are there and, you know, what better chance to get to know them a little bit than to sit down and like add a feature with them or spend a day in the room with them.

01:08:23.960 --> 01:08:24.700
Something like that, right?

01:08:24.700 --> 01:08:32.640
That really can build some connections that, you know, especially if you're in a small town somewhere and not meeting them in person, that could be a challenge.

01:08:32.640 --> 01:08:36.720
Yeah, and a lot of people stay within Python because of the community.

01:08:36.720 --> 01:08:43.580
So like, I guess my final call to action comes from Greg Wilson in his book called Teaching Tech Together.

01:08:43.580 --> 01:08:48.660
He talks about the rules of teaching how to program or like building community.

01:08:48.660 --> 01:08:52.600
And the first rule is be kind, all else is details.

01:08:52.940 --> 01:08:53.500
Yeah, that's great.

01:08:53.500 --> 01:08:54.780
Be kind to all else's details.

01:08:54.780 --> 01:08:55.420
I agree.

01:08:55.420 --> 01:08:58.900
It's definitely right up there is one of the most important ones.

01:08:58.900 --> 01:09:00.980
All right, Dan, thank you for being on the show.

01:09:00.980 --> 01:09:03.480
It's been really great to talk about these ideas with you.

01:09:03.480 --> 01:09:05.780
I think there's a lot of good advice people can take away.

01:09:05.780 --> 01:09:07.760
Yeah, it's been great talking with you, Michael, as well.

01:09:07.760 --> 01:09:08.040
You bet.

01:09:08.040 --> 01:09:08.340
Bye.

01:09:09.340 --> 01:09:11.840
This has been another episode of Talk Python To Me.

01:09:11.840 --> 01:09:16.960
Our guest on this episode was Daniel Chen and has been brought to you by Indeed and Rollbar.

01:09:16.960 --> 01:09:24.720
With Indeed Prime, one application puts you in front of hundreds of companies like PayPal and VRBO in over 90 cities.

01:09:24.720 --> 01:09:28.180
Get started at talkpython.fm/Indeed.

01:09:28.180 --> 01:09:30.960
Rollbar takes the pain out of errors.

01:09:30.960 --> 01:09:38.260
They give you the context and insight you need to quickly locate and fix errors that might have gone unnoticed until users complain, of course.

01:09:38.860 --> 01:09:44.700
Track a ridiculous number of errors for free as Talk Python To Me listeners at talkpython.fm/rollbar.

01:09:44.700 --> 01:09:46.800
Want to level up your Python?

01:09:46.800 --> 01:09:51.640
If you're just getting started, try my Python Jumpstart by Building 10 Apps course.

01:09:51.640 --> 01:09:59.800
Or if you're looking for something more advanced, check out our new async course that digs into all the different types of async programming you can do in Python.

01:09:59.800 --> 01:10:04.460
And of course, if you're interested in more than one of these, be sure to check out our Everything Bundle.

01:10:04.460 --> 01:10:06.360
It's like a subscription that never expires.

01:10:06.360 --> 01:10:08.640
Be sure to subscribe to the show.

01:10:08.640 --> 01:10:11.140
Open your favorite podcatcher and search for Python.

01:10:11.140 --> 01:10:12.280
We should be right at the top.

01:10:12.280 --> 01:10:21.260
You can also find the iTunes feed at /itunes, the Google Play feed at /play, and the direct RSS feed at /rss on talkpython.fm.

01:10:21.260 --> 01:10:23.360
This is your host, Michael Kennedy.

01:10:23.360 --> 01:10:24.860
Thanks so much for listening.

01:10:24.860 --> 01:10:25.920
I really appreciate it.

01:10:25.920 --> 01:10:27.660
Now get out there and write some Python code.

01:10:27.660 --> 01:10:47.380
I'll see you next time.

01:10:47.380 --> 01:10:48.180
Bye.