#400: Ruff - The Fast, Rust-based Python Linter Transcript

00:00 Our code quality tools, linters, test frameworks, and others, play an important role in keeping our code error-free and conforming to the rules our teams have chosen.

00:08 But when these tools become sluggish and slow down development, we often avoid running them or even turn them off.

00:15 On this episode, we have Charlie Marsh here to introduce Ruff, a fast Python linter written in Rust.

00:21 To give you a sense of what he means by fast, common Python linters can take 30 to 60 seconds to lint the CPython codebase.

00:29 Ruff takes 300 milliseconds.

00:31 I ran it on the 20,000 lines of Python code for our courses web app at Talk Python Training, and it was instantaneous.

00:39 With that kind of speed, it's the kind of tool that can change how you work.

00:43 I hope you're excited to learn about Ruff.

00:45 This is Talk Python to Me, episode 400, recorded January 18, 2023.

00:58 Welcome to Talk Python to Me, a weekly podcast on Python.

01:08 This is your host, Michael Kennedy.

01:09 Follow me on Mastodon, where I'm @mkennedy, and follow the podcast using @talkpython, both on fosstodon.org.

01:17 Be careful with impersonating accounts on other instances.

01:20 There are many.

01:21 Keep up with the show and listen to over seven years of past episodes at talkpython.fm.

01:26 We've started streaming most of our episodes live on YouTube.

01:30 Subscribe to our YouTube channel over at talkpython.fm/youtube to get notified about upcoming shows and be part of that episode.

01:38 This episode is brought to you by Cox Automotive.

01:40 Join their team and use your technical skills to transform the way the world buys, sells, and owns cars.

01:46 Find an exciting position that's right for you at talkpython.fm/cox.

01:50 And it's also brought to you by User Interviews.

01:53 Earn extra income for sharing your software development opinion at User Interviews.

01:58 Head over to talkpython.fm/userinterviews to participate today.

02:03 Before we get to the conversation for this episode, I have a very brief sponsored message from the PyCharm team.

02:09 As you know, I'm a huge fan of PyCharm, the tool and the team, and I'm happy to bring it to you.

02:14 Do you like storytelling?

02:15 Want a job where you can work with the Python community and help people with their professional development?

02:21 PyCharm is doing big things this year, and they have an exciting job opening you should consider.

02:27 Developer Advocate.

02:28 It's a good job doing good work with a great company.

02:31 Learn more and apply at talkpython.fm/PyCharm dash Advocate dash Job.

02:36 The link is in the episode show notes.

02:38 Charlie, welcome to Talk Python to me.

02:41 Thanks so much.

02:42 It's my pleasure to be here.

02:43 Yeah, it's a pleasure to have you.

02:45 I'm going to talk about making Python code fast.

02:47 Built some pretty interesting tools here with Ruff.

02:52 We'll see about integrating Rust into Ruff, and there's a lot of cool tools around this

02:58 sort of code quality side of things, right?

03:01 Like Black took off in ways, I think, even though Lukash didn't maybe imagine it would take off,

03:06 and this is kind of in that same realm, and of course, it plays well with Black along the

03:13 way as well.

03:13 So we'll get to talk all about those things.

03:16 It's going to be tons of fun to dig into them, but before we get to it, let's start with

03:19 your story.

03:20 How did you get into programming in Python and Rust?

03:22 Yeah, yeah.

03:23 So I got into programming, I guess, the summer before I went to college, when I guess I was

03:29 supposed to be doing something fun.

03:30 I taught myself Java, and it was the sort of thing that a lot of people had told me I would

03:36 like, but I never really had a chance to dig into.

03:38 I ended up, I went to school, majored in computer science, so I did a bunch of internships while

03:43 I was in college.

03:44 And then when I, my sort of junior year internship and my first year out of school, I worked at

03:50 a company called Khan Academy, which is like an education technology company.

03:54 And at the time, this is less true now, but at the time they were a really big Python

03:59 child.

03:59 I mean, they had one of the biggest like App Engine deployments.

04:03 It was like Khan Academy and like Snapchat and like a couple other, maybe Spotify or someone.

04:09 Wow, I had no idea.

04:11 I mean, I'm just, yeah, I'm obviously I'm familiar with Khan Academy, K-H-A-N, I'm sure a lot of

04:17 people are.

04:17 Yeah.

04:17 But I didn't realize that one, they were such a GCP customer and that was Python.

04:22 Yes.

04:22 Was that Flask or Django on GCP or what was it?

04:25 It was like Flask on App Engine.

04:27 Yeah.

04:28 We did not use Django.

04:29 Yeah.

04:29 But they actually, I mean, it's a whole nother story, but they did a big Go rewrite after

04:34 I left, which is also super interesting.

04:36 I mean, it's not Python, but it is still very interesting.

04:38 Just like migrating a system from any language to any other language is always pretty interesting.

04:43 But at Khan Academy, yeah, you know, I did some Python.

04:46 I would say that like I mostly did web and we can talk a little bit more about this because

04:52 like really through my career, I've kind of jumped between a lot of different ecosystems.

04:55 And so like at Khan Academy, I did, I think a year of Android.

05:00 And did a little bit of iOS, did a lot of web, did a little bit of Python.

05:03 And then most recently, I worked at a company called Spring Discovery, which is like a computational

05:09 biotech company.

05:10 And everything we were doing was based on computer vision.

05:13 So we'd like take really high resolution pictures of cells and then try to model like the way

05:18 that they change when you add drugs.

05:19 Wow.

05:19 Okay.

05:20 Which is, yeah, it's pretty wild.

05:21 I mean, I came into that with no bio background.

05:23 I joined as like second engineer and build out a lot of our like data and machine learning

05:28 platform.

05:29 That was kind of a crash course in building like a massive system in Python, right?

05:32 We were doing scientific computing.

05:34 So it made a lot of use Python.

05:35 So we ended up using Python for like almost everything.

05:38 Yeah.

05:38 Yeah.

05:39 Python is a really good choice for the web, but it's like the de facto choice for data

05:44 science, right?

05:44 Like it's.

05:45 Yes.

05:45 Yeah, exactly.

05:46 There's not a whole lot of other choices.

05:48 Whereas on the web, there's, you know, five or six other platforms and languages that are

05:52 also contenders for that space.

05:55 Right, right.

05:55 So, you know, we built out what I consider to be like a fairly large Python code base.

06:00 It was like a big monorepo.

06:02 And, you know, I spent four or five years, you know, building that out and like really writing

06:08 a lot of Python every day.

06:09 That's where like a lot of my Python experience comes from.

06:12 And like that experience specifically also informed a lot of, a lot of growth.

06:15 And, you know, I can talk more about sort of how those experiences intertwined, but that's

06:19 kind of my Python background.

06:20 It must have been really fun to explore some of this visualizing cells and interacting with,

06:26 you know, microscopes and.

06:28 That was amazing.

06:29 Yeah.

06:29 All those kinds of things.

06:30 Yeah.

06:30 The coolest part of the job was just that if you bucket it like really crudely, like a

06:35 third of our team were sort of like software engineers or like data scientists, machine

06:38 learning researchers, and then, you know, a third were, you know, wet lab biologists,

06:43 PhDs.

06:45 And so every day I was like working directly with scientists and like building software for

06:49 scientists, which was just like a super cool thing to be able to do as part of a small team.

06:53 Yeah.

06:53 But yeah, wrote a lot of Python.

06:55 That's awesome.

06:56 I worked at a place that was a bunch of PhD researchers doing eye tracking, like not Apple,

07:03 but your actual eyes.

07:05 Like, what are you looking at?

07:06 wrote the software to actually control the device and get all the information off of

07:10 it and then like analyze that and put, you know, generate like reports for the scientists

07:15 and stuff.

07:15 And it's, there's a, there's a special kind of cool for software that touches like reality,

07:21 right?

07:21 Yeah.

07:22 I mean, it adds a lot of complication.

07:24 Of course it does.

07:25 It's very cool.

07:26 You don't get exact numbers.

07:28 You want to test it like, you got to approximate all the things, but still, yeah, that sounds

07:33 like a super fun job.

07:34 How did it inform rough?

07:36 Like what are some of the experiences that?

07:37 Yeah, totally.

07:38 Just so people know rough is like a super fast linter.

07:42 Just give us the quick elevator pitch.

07:43 So they, they, they know what the heck rough is.

07:45 We'll dive more into it later.

07:46 Sure.

07:47 Yeah.

07:47 Yeah.

07:48 So rough is what I call an extremely fast Python linter.

07:51 If you've used linters and Python in the past, you've probably used something like play gate

07:56 or pilot, or maybe like PI code style, you know, maybe use other tools to help manage your code

08:01 style, like black or like auto peppy, all this stuff that deals with effectively code style.

08:07 So rough is a linter.

08:09 It looks at your source code and it tries to report back and tell you a sort of flag and

08:14 tell you about issues with it.

08:15 And those could be stylistic issues.

08:17 Like this variable name is not great, or they could be, you know, like logical issues.

08:22 Like you reference a variable here and that isn't defined.

08:25 Right.

08:25 I think there's a lot that distinguishes rough, but like the thing that I think caught

08:28 a lot of people's attention was just how fast it is.

08:30 So I sort of hate benchmarks, like, cause no matter how much time you like put into a benchmark,

08:35 like it's always wrong from like a certain perspective and like people always complain about

08:39 them, but I think rough is comfortably like somewhere in 10 to a hundred times faster.

08:44 Some people even say like a thousand times faster on their setup.

08:47 So it can be like really, really dramatically faster.

08:49 And it can also do a lot of auto fixing, which is kind of a distinctive feature that I don't

08:55 know that many other tools.

08:57 And there are certainly tools that do code transformation in Python, but like Pylon and

09:02 Flake 8 and such don't do this.

09:04 So for lots of the errors that Ruff will flag, it can actually fix the code for you and fix

09:09 the issue for you, which is a big time saver.

09:11 I don't know if this is one of them, but just to give people an idea, like one of the errors

09:14 or warnings rather that it'll give you is you have an F string, but you're not leveraging

09:20 its F string capabilities, right?

09:22 I'll just say F quote, and then it has just static text, a little.

09:25 Literal string.

09:26 So it doesn't need the F, right?

09:27 Theoretically, you could say auto fix and just it drops the F for you.

09:31 It's like, yeah, you don't need that, right?

09:33 Yeah.

09:33 And, you know, we can also like go the other direction.

09:36 So like if you're doing a bunch of like percent string formatting or using like the dot format

09:42 style helpers, we're actually, we can fix at least one of those right now.

09:47 And strings are really complicated, but like we can actually rewrite, I think the dot format

09:51 calls into strings, which is cool.

09:54 Fantastic.

09:54 You know, another thing we can do is we can kind of keep your code like modern in a way.

09:58 We've taken a lot of inspiration from other tools like Pi Upgrade 2 has like so much cool

10:03 stuff and functionality in it if you haven't seen it.

10:06 And we take some of the rules from Pi Upgrade and basically re-implement them in Rust and package

10:11 them into a single interface and a single tool.

10:14 So you just install one thing and you get, you know, 370 something rules.

10:20 I can assert that it is indeed fast.

10:23 We'll dive into it a little bit later, but I ran it on Talk Python training, the courses

10:29 website that I have.

10:30 And I think that's got 20,000 lines of Python and it's instant.

10:34 It's like you press enter and then out, you know, the stuff is just printing out.

10:38 These are the things that found, which is pretty awesome.

10:41 So back to your data science work.

10:44 Yeah, yeah.

10:44 How are you, what you're doing influence your creation of RUF?

10:47 Like why not just use Flake 8 or something like that?

10:51 You know, like I said, I'd like worked in a lot of other ecosystems where, with sort of

10:55 like stricter languages, right?

10:57 Like Java and whatever else.

10:59 And it's not to say those ecosystems are better, but I often come at the stuff from a perspective

11:03 of wanting a lot of like static analysis tooling.

11:06 So we had like a really heavily typed code base.

11:08 We used, like we used mypy, you know, we used Black, we used Flake 8, we used Eyesore, we

11:15 used, I don't know, like doc formatter.

11:18 There's like a lot of different stuff that we used.

11:19 Yeah.

11:19 You know, really my job was like, I was the maintainer of this big system.

11:22 And like most of, like a lot of people in the team were effectively like clients of that

11:27 code base.

11:27 So they were kind of like using the stuff that, you know, I and some others were building

11:31 as like a library.

11:32 And so I had to maintain like this really big system.

11:35 And like the more static tools you have, you know, in my opinion, it just like greatly

11:40 increases the leverage that you can have as a maintainer.

11:43 So we use a lot of tooling.

11:44 And I think like a couple of things happened.

11:47 One, you know, I was spending a lot of my time in Python, but I was also doing a lot of

11:51 web.

11:51 And so, you know, like one week I'd be doing Python, the next I'd be going and doing like

11:56 React and TypeScript and whatever else.

11:58 And so people complain a lot about web tooling and like, you know, there are certainly lots

12:04 of valid complaints about it.

12:05 But there's been a lot of interesting innovation, I think, in that space.

12:08 And I was certainly influenced by some of it, just seeing some of the tools that came out

12:13 and how fast they were.

12:14 Sure.

12:14 And the fact that a lot of those tools were not being written in JavaScript sort of intentionally

12:18 to make them super performant was pretty interesting to me.

12:21 So like, I don't know, I don't want to like name drop a bunch of tools, but you know, in

12:24 the web ecosystem, more tooling is being written in like Go, Rust, even some other sort of more

12:30 out there languages like Zig.

12:31 And to me, that was kind of an interesting thing.

12:33 It was like, well, does the Python tooling have to be written in Python?

12:36 Like what does, like, why does that have to be the case?

12:39 And like, what does it buy you and what do you lose?

12:41 So that was one thing.

12:42 The other was we actually started to do a lot of Rust Python hybrid stuff in our own code base.

12:48 You know, our code base was fully Python.

12:50 And over time, we took some of the like really performance critical pieces and started to

12:56 rewrite them in Rust and expose them to the Python side using some of these like really great

13:03 Rust Python hybrid tools that, you know, maybe we'll get into later.

13:06 So like one example is we were dealing with image data.

13:09 So like a common thing we had to do is just like download a lot of image files, like really,

13:14 really fast.

13:14 Right.

13:15 And so, you know, we wrote that in Python.

13:17 We wrote like a bunch of versions that use all sorts of like multiprocessing, threading,

13:21 and like we tried everything.

13:22 And then eventually we like rewrote it in Rust.

13:24 And so we had a really simple, you know, Rust module that would just download files really

13:29 fast.

13:29 And then we exposed that as a Python API.

13:31 So like from the Python code, it just looked like Python, but the actual code that was running

13:36 and the way it was, you know, built and executed all went through Rust.

13:40 Right.

13:41 So you have these big data, big, let's say big computation problems where you're like, you

13:46 know, this could be better.

13:47 The same time you're looking at this happening on the web side, you know, web pack or whatever

13:51 it is.

13:52 And it's like, this is way better.

13:54 We can do this in Python, but we aren't.

13:55 So let's start, right?

13:56 Probably you got a little experience there and you're like, what else can I build with Rust

13:59 now that I know how?

14:00 Yeah.

14:01 I was seeing, you know, the way that this innovation was happening in the web.

14:04 I was seeing specifically the Rust Python like opportunity from what we were doing.

14:09 And then I was, you know, frankly, like feeling a fair amount of frustration just with the

14:14 performance of a lot of the existing tools.

14:16 Like, I don't know.

14:17 I'm always really careful with how I talk about this stuff because I hate being, I'm

14:20 really like not a hater.

14:21 And like, I've actually gotten like so much value out of like all of those tools.

14:25 But I was being like, well, should it really take like one to two minutes to let my code,

14:31 like couldn't it be much faster?

14:33 Right.

14:33 Like it's like a big code base.

14:34 And like, as you add more plugins and everything else, it really like changes, like how you

14:39 feel about the tool.

14:40 Right.

14:41 Like it's no longer like something I just run and get feedback.

14:43 It's like, oh, I have to run the linter now.

14:46 Oh, I have to run the blah blah blah.

14:47 And I was sort of craving that.

14:49 Yeah.

14:49 And if you're doing it on a team, my experience at least has been some people in the team continue

14:55 to run it because they find more value than stress.

14:58 And others are like, eh, I'm not running that stupid thing.

15:00 But then their problems get pushed onto the people who still care.

15:04 And so then it creates this kind of like a tension, like Joe's code is always kind of crappy

15:08 because he won't lent it.

15:09 But then like when I check out his code, my stuff starts reporting.

15:13 You know, like those are the weird dynamics that happen from tools that are like super slow

15:18 unit tests or super slow linting or those types of things where certain people stop working

15:23 with them and others don't.

15:24 And then you get these weird tensions.

15:26 So having it fast, there's like real value to that.

15:30 This portion of Talk Python Nemy is brought to you by Cox Automotive.

15:35 With brands like Kelly Blue Book, Autotrader, Dealer.com and more, Cox Automotive flips the

15:42 script on how we buy, sell, own and use our cars.

15:46 And now the team at Cox Automotive is looking for software engineers, data scientists, scrum

15:52 masters and other tech experts to help create meaningful change in the industry.

15:57 Do you want to be part of a collaborative workplace that values your time and work life balance?

16:02 Consider joining Cox Automotive.

16:04 Visit talkpython.fm/cox today.

16:08 Thank you to Cox Automotive for sponsoring the show.

16:13 I left spring and like mid to late August.

16:17 I started working on Ruff along with like a couple other projects and I never built a linter

16:22 before.

16:22 It was sick.

16:23 So like I didn't, yeah, I didn't really know what I was doing, but it felt maybe this is

16:28 like a little bit of hubris or not, I guess, because like it seems to be working.

16:33 But like I was kind of like, I'll start with a linter because it feels like a tractable

16:36 like scope of problem.

16:38 I was like, type checker.

16:39 I know that's hard and like I don't really know much about it.

16:42 A linter.

16:42 I feel like I can learn like some of the concepts.

16:45 I have a good understanding of like of like how it might work.

16:48 To me, when I look at it, I think linting people's code, like abstract syntax trees and

16:52 stuff.

16:52 I'm like, that doesn't sound like a problem to me.

16:54 But it's good that you saw it that way because you did get through it.

16:57 Yeah.

16:58 Well, the first version I released was like very limited.

17:01 It was sort of, I sort of needed to get it out because I wasn't sure that people would

17:05 actually really care that much about like what I was doing.

17:07 Yeah.

17:08 Because like I've described to you a lot of like pain points.

17:11 They're also sort of like opinion, right?

17:12 But like, you know, I wasn't like a fast forward linter.

17:15 Like who cares?

17:16 Was sort of, I wasn't sure.

17:17 Yeah.

17:18 So, you know, when I released it, I was like, I want to prove it's possible.

17:22 I want to like have reasonable confidence that the performance won't get worse or much

17:26 worse as I expand it.

17:28 But I only supported like, I don't know, maybe like 20 rules, which was fine with me because

17:32 I was really trying to prove a point more than, you know, get people to adopt it even.

17:37 Right.

17:37 Well, and a lot of times go into that experience, even if very few people used it, you learn fantastic

17:43 things, right?

17:44 Yeah.

17:45 I learned so much.

17:46 And I read a ton of code.

17:48 Like I read all, I've read like a lot of, I've read like a lot of source code now.

17:54 So I've read like all of like, you know, PyFlakes, like high doc style, high code style, like, you

18:01 know, because I've spent time like trying to re-implement a lot of those, a lot of those rules

18:04 and like understand how they do things.

18:06 And you just learn a lot from like reading and trying stuff.

18:09 Yeah.

18:09 Rust is sort of like a whole nother thing where, you know, I did some Rust at Spring,

18:13 but I would say that building Rust was really like my Rust learning experience.

18:17 I think I had to kind of try to build something from scratch to like really understand the language.

18:23 Sure.

18:23 Everything I did in Spring.

18:24 You can do all the tutorials.

18:26 Yeah.

18:26 And I was like, I was trying to get in and out as quickly as possible.

18:29 No, no.

18:30 I just like, you know, I didn't do the initial Rust migration at Spring.

18:33 I worked with another like super talented engineer who did that.

18:37 And so every time I went into, you know, and I made a bunch of changes to it, but every

18:41 time I was going in and out, I was like, I just need to like ship this and like move on.

18:45 Like, I'm not going to spend a bunch of time like really understanding everything that's

18:49 going on here.

18:49 I just want to get it to work.

18:50 And so I needed to like sit down and like really like focus and be presented with like

18:56 a blank plate, I think to learn and just like fail a lot and like build stuff that didn't

19:00 compile.

19:01 Right.

19:01 And figure out how to resolve those problems.

19:03 It's honestly, it's a tough, it is a tough learning curve.

19:05 I'm sure that it is.

19:07 I started out in C++ and so I, I know there are harder languages and they're easier languages.

19:12 So I've never written C++.

19:15 I'm never.

19:17 You're not missing that much.

19:18 I would say.

19:18 Eddie out there says rough is so awesome.

19:20 I actually changed all the switch statements in my Python 310 code back to if else blocks

19:25 so I could switch to rough, which is pretty awesome.

19:28 That's super nice and slightly embarrassing because we don't support Matt.

19:32 We don't support Matt.

19:32 We don't support the match.

19:34 That's the one big language feature that we don't support yet.

19:38 You could just kind of comment it out.

19:39 Like it just, we're going to just ignore whatever's in the switch.

19:41 It must be fine.

19:42 On to the next part.

19:43 Yeah, exactly.

19:44 Right.

19:44 Yeah.

19:45 Yeah.

19:46 Yeah.

19:46 I mean, not for your code, but for linting.

19:48 Let's start our dive in with this comment from Tyler.

19:51 He says, what are the differences between linters like rough and flake gate?

19:55 Those versus code stylers or formatters like black.

19:58 And maybe rough lives in the middle of that spectrum.

20:00 I'm not sure.

20:01 Yeah.

20:02 There's certainly like, like, like you said, it's like a spectrum, right?

20:07 So I guess there's a couple of different ways to think about it.

20:09 And it, with those tools specifically, like some of the responsibilities have like in my

20:14 mind, at least shifted a bit over time.

20:15 So for example, like flake gate, it does a lot of, it doesn't change your code at all.

20:20 And it'll just tell you about issues.

20:22 And those issues could be style issues.

20:24 Like you have extra white space here or.

20:27 Yeah.

20:27 I don't know.

20:28 What's another big example.

20:29 This line is too long.

20:30 Yeah.

20:30 Yeah.

20:31 But it could also be like a logical issue, which is something that black wouldn't touch.

20:35 So it might be like this import is unused.

20:38 So on the one end you have like, you know, different collections of rules and maybe not really

20:43 changing the code, just kind of telling you about it.

20:45 Black is different in that black just reformats your code.

20:48 So it does, it's not actually aware of like the logic in the code.

20:52 Like it doesn't know that an imports unused.

20:54 It doesn't know that function is never called.

20:57 Yeah.

20:57 It just takes, you know, that abstract.

20:59 Well, not, not exactly the abstract syntax tree, but it basically preserves your code as

21:03 is and just tries to restyle it.

21:05 And so, you know, it's interesting because like if you use black, you probably don't need

21:09 a lot of style linting.

21:11 Like if you're using black to format your code, a lot of the rules.

21:14 Like a part that relevant anymore.

21:16 They'll say something like you should have followed the rules of this, but black automatically

21:21 follows those rules for you.

21:22 Right.

21:22 So they're kind of, you don't need to correct the line length because black will wrap it

21:25 because that's what black does.

21:27 Right.

21:28 Yeah.

21:28 Or even like, you know, if you have like X equals one with like no spaces between the

21:33 X and the equals and the equals and the one, like the linter will yell at you for that.

21:37 But black fixes that for you.

21:39 So a lot of people will turn off those rules.

21:41 You know, this is why I say the responsibilities have shifted a bit over time.

21:45 Yeah.

21:45 I think you're right.

21:46 That rough, like it kind of sits in the middle.

21:48 We do less stylistic linting for this exact reason.

21:51 Like I use black a lot.

21:53 Black is like immensely popular.

21:55 So I actually like strong, like heavily deprioritized, like all of the stylistic rules that are made

22:01 redundant by black.

22:02 Okay.

22:03 The difference is we do do a lot of that automatic fixing.

22:05 So like we'll, we can remove unused imports for you.

22:08 We can even do like a lot of more complicated rewrites.

22:11 Like if you're using, if you're using like a dictionary constructor, like the actual, you

22:16 know, D I C T parentheses arguments, we can actually rewrite as a literal.

22:21 If you have the rules enabled and you want to do that, you know, there's like a bunch of

22:25 rules around like comprehensions and literals, and we can do those rewrites too.

22:28 So, you know, it does, it can do like style and reformatting, but it's somewhere in the

22:33 middle.

22:33 Yeah.

22:33 The fact that it makes some changes, but it doesn't go completely all in like black is

22:38 kind of why I think it lives a little bit in the middle there.

22:40 Yeah.

22:40 One of the things that I think might give people a lot of, brought a lot of faith or a willingness

22:46 to give it a try is there's a lot of projects out here that are large projects that people

22:52 would know that have adopted, actively adopted and started using rough.

22:58 I saw a quote from Sebastian Ramirez from FastAPI right now in the audience.

23:03 We have Olfek says the next release of hatch, which I hope to do this weekend will have

23:10 all the new projects generated with config for a Linting environment that uses rough, which

23:14 I think is pretty excellent.

23:15 Can we speak to speak to some of that, like about these, these projects adopting it and

23:20 how that's helped you?

23:21 Yeah, totally.

23:22 Yeah.

23:22 I never expected that like this, maybe that like goes without saying, I think you'd have

23:27 to be like super arrogant to release something and expect a bunch of these people to use it.

23:31 Yeah.

23:31 It's been kind of wild.

23:32 You know, initially there were a couple big projects that I would call like really early

23:37 adopters and they helped shape the project a lot.

23:42 like Pydantic was a very early adopter.

23:44 Zulip was a very early adopter.

23:46 FastAPI was actually like a pretty early adopter too.

23:49 Okay.

23:50 And then like Pandas and Airflow and stuff, those kind of are more recent.

23:55 Okay.

23:55 It's been a very interesting, like to reflect on how that's happened because on the one hand,

24:01 it's like sort of ridiculous that all these products would migrate over because I'm on,

24:06 I don't even really have like a proper versioning system right now.

24:09 Like I'm on V00225 or something.

24:12 I basically release like every day and I consider like, I don't really make, we've made like

24:18 very few breaking changes, but like in theory we could make breaking change.

24:22 I don't want to, but like if you have Scribe December, like we could make changes like that.

24:27 You know, there's not really like docs, like the docs are just like this huge read me.

24:30 It's kind of wild, right?

24:32 We don't support like all the Python 3.10 features, right?

24:35 We don't support match statements.

24:36 So on the one hand, it's kind of wild that all these projects would move over.

24:39 I think there are a couple of things that help a lot.

24:42 So one is that the entire time I've been like very, very focused on compatibility with existing

24:48 tools.

24:48 And that comes at a lot of cost.

24:51 Like if there are things that I like don't agree with, right?

24:54 Or whatever else, like I sometimes still support them or implement them.

24:57 And, you know, but the upside is like, it's very easy for people to migrate.

25:01 Right.

25:02 It might change nothing potentially.

25:04 Right.

25:05 It just, it just goes faster.

25:06 Hopefully.

25:06 And over time, like, you know, we don't have like a third party plugin system.

25:10 We've just like implemented everything like in Rust, like as part of Rust.

25:15 I viewed that as like, I'm just going to look at the most popular plugins and the most important

25:20 rules.

25:20 And I'm just going to figure out like, what's a blocker for whom?

25:24 And like, what do I need to implement in order to like unblock people and like make sure

25:28 they can use it?

25:28 Is there a way for people to let you know which of those are really important to them?

25:32 Like, for example, if there's some big project out there, it's like, we would love to move

25:36 to rough, but there's this one, one plugin we're waiting on.

25:41 Yeah.

25:41 I mostly hear about it in issues.

25:43 And there are a couple issues where it's like kind of a big GitHub issue.

25:49 It'll be like implement this plugin.

25:51 And a lot of people will come and like give it a thumbs up or whatever.

25:54 We have pretty good coverage now, actually.

25:57 Like, I mean, there is a long tail.

25:59 Like there's a big ecosystem of like flaky plugins.

26:01 You know, sometimes someone will come along and ask for a plugin that I haven't really like

26:06 seen before.

26:06 And then it's just a calculus of like, what do I want to prioritize?

26:09 How much work is it?

26:11 Yeah, exactly.

26:12 Is this a super easy thing to do?

26:15 Like, oh, the line is too long.

26:17 That's easy.

26:17 Or is it, you know, rewrite this?

26:19 Yeah.

26:20 This generator in this other interesting way.

26:23 Yeah.

26:23 That happened with, well, a lot of people want like, I don't know how to pronounce this library,

26:27 but like Dargland, like D-Argland, Dargland.

26:30 I don't know.

26:31 Anyway.

26:31 I would go with that as well.

26:32 Yeah.

26:32 Yeah.

26:33 Dargland.

26:33 Let's go with that.

26:34 It's very popular, but I've looked at the source.

26:35 It's pretty complicated.

26:36 So like, I need to, you know, that's not something I can just like turn around in a day.

26:40 But like, you know, I hear like what people want.

26:43 And I mean, some of those migrations though, I didn't even know about them until they'd happened.

26:47 Yeah.

26:48 Yeah.

26:48 Of course.

26:48 People come, they submit issues.

26:50 I look at their profiles.

26:51 I see what they're working on.

26:52 I get a sense for what the projects are.

26:53 Right.

26:54 So sometimes I know, and then I'm kind of fixing things knowing that hopefully it's unblocking

26:59 them in different ways.

27:00 But I think, like I said, I think the compatibility piece is like really huge.

27:03 And I actually think that like black and eyesword and like the popularity of those things has

27:08 made compatibility for me like a little bit easier because I can just like adhere to what

27:12 they do.

27:13 And then hopefully there's like no code changes.

27:16 Yeah.

27:16 They centralize around what like black is the truth.

27:19 And then we'll, then we'll stop.

27:20 We'll debate beyond that.

27:21 But like, we're just going to accept black because it's no fun to go commas on the end

27:27 of the line.

27:27 No commas on the trailing thing in the list.

27:29 You know, like that, those kinds of debates just go on and on.

27:32 Yeah.

27:33 I know.

27:33 I think black kind of solves that by just making a decision for you and not everyone's

27:38 totally happy with every choice, but it stops the debate.

27:41 And that actually probably wins the day for most of them.

27:43 One thing that's interesting here that I'm noticing about these projects in the large open

27:48 source projects that have adopted rough, not all of them, surely, but many of them like

27:54 Pydantic and Polars and so on themselves are adopting Rust.

28:00 Do you see a synergy of those?

28:02 Like, well, we're rewriting Pydantic core and Rust like Samuel Colvin is, or the folks behind

28:07 Polars are like, that's the core of it is about Rust.

28:10 And that's the RS on the end, I think.

28:13 So there might be a little extra draw there, right?

28:15 I think it's just more like, like spiritual alignment.

28:19 Yes, exactly.

28:20 Because it doesn't matter.

28:21 No, it doesn't.

28:23 But spiritually.

28:24 But there's, yeah, there's a little bit of a bug.

28:27 Sorry, bug in the sense of like, like once people start writing some stuff in Rust and

28:32 improving their Python, they're kind of like more open to like, you know, trying out other

28:37 things that are written in Rust and so on.

28:39 So it's, I mean, that community has like been very interesting to sort of watch and be a

28:44 part of the like Rust Python community.

28:46 It still feels like pretty nascent, but there are, the tooling is actually like really great.

28:52 And, you know, we could talk about that more, but like, you know, there was definitely like

28:57 a bunch of initial setup for actually like distributing Rust, PyPI and making pip installable.

29:03 But now that that's over, it all like just works.

29:05 And I mean, I wouldn't say it's like requires no effort, but it's actually pretty amazing.

29:10 Like how, like the degree of integration and how much you can do now with these tools like

29:15 Py03 and, and MatcherIn and some others.

29:18 That was one of the things that was really nice surprise for me as well as how easy it is to

29:22 install, right?

29:23 Like, unless you told me, unless it wasn't part of, you know, the elevator pitch, you know,

29:29 the little one sentence subtitle on the project, like you would know from a interacting perspective,

29:34 which is kind of exactly what you want.

29:36 I also think that it's really, there's some value to having these popular projects adopt

29:40 Ruff because Martin and the audience, for example, says, I found out about Ruff by looking to see

29:45 what FastAPI does, because presumably it's a ton of respect for FastAPI and Sebastian, his choices.

29:52 So yeah, it's got to sort of create the snowball effect as well.

29:56 So this portion of Talk Python to me is brought to you by user interviews.

30:02 As a developer, how often do you find yourself talking back to products and services that you use?

30:08 Sometimes it may be frustration over how it's working poorly.

30:11 And if they just did such and such, it would work better.

30:15 And it's easy to do.

30:17 Other times it might be delight.

30:19 Wow, they auto-filled that section for me.

30:21 How did they even do that?

30:22 Wonderful.

30:23 Thanks.

30:23 While this verbalization might be great to get the thoughts out of your head,

30:27 did you know that you can earn money for your feedback on real products?

30:31 User interviews connects researchers with professionals that want to participate in research studies.

30:36 There is a high demand for developers to share their opinions on products being created for developers.

30:42 Aside from the extra cash, you'll talk to people building products in your space.

30:47 You will not only learn about new tools being created, but you'll also shape the future of the products that we all use.

30:53 It's completely free to sign up and you can apply to your first study in under five minutes.

30:58 The average study pays over $60.

31:00 However, many studies specifically interested in developers pay several hundreds of dollars for a one-on-one interview.

31:07 Are you ready to earn extra income from sharing your expert opinion?

31:11 Head over to talkpython.fm/user interviews to participate today.

31:16 The link is in your podcast player show notes.

31:19 Thank you to user interviews for supporting the show.

31:24 One thing that's a little bit overlooked about why that adoption is useful is that it actually feeds back into the project in so many great ways.

31:34 So, you know, for example, DAG sir adopted ROUGH and they have a really big Python monorepo with 50 or 100 separate Python modules or packages in there.

31:48 And they filed like a bunch of issues about around how can we make ROUGH better for monorepos.

31:54 And so now they can run ROUGH once and it runs over all their separate projects that can all have their own configuration.

32:00 And that's like a really, really powerful thing.

32:03 And something that like if I was just sort of sitting around like building ROUGH, like I don't know that I would have known to prioritize that or what it should look like.

32:11 And so like actually like working with people who want to adopt ROUGHs and like figuring out what stops using it provides a ton of value to the project and just prioritization.

32:21 Like a lot of the iSort stuff, I pushed on it a lot when I started because ROUGH does import sorting.

32:25 So you can use it if you want in lieu of iSort.

32:28 And we implement a lot of the same configuration options, not all the same.

32:33 iSort just is like really configurable.

32:35 So we went, we sort of implement them as they get requested.

32:37 But anyway, I prioritize that stuff because a lot of people wanted it and they were coming to the issues and being like, hey, this would really help me adopt it, blah, blah, blah.

32:46 So just like getting the feedback and like hearing what's blocking people, like that itself is like so valuable for the project.

32:52 Right.

32:52 And you have a lot of people involved in these larger projects.

32:54 So they, and they have specific polished needs, right?

32:59 Like there's a lot of people who work on pandas or FastAPI and those are mature code bases.

33:05 It's not just a pet project, right?

33:07 Yeah.

33:07 Yeah.

33:08 Those are very mature.

33:08 Yeah.

33:09 Yeah, exactly.

33:10 That's good.

33:11 But also it's a challenge.

33:12 One other thing I would like to point out, you know, you started off by introducing rough saying, maybe people will be interested.

33:18 I don't really know if they will.

33:20 Maybe they will.

33:20 That'd be cool.

33:21 You know, 6.5, 6.4 thousand stars.

33:24 That's a, it's a non-trivial amount of interest.

33:27 That's a lot of love.

33:28 Yeah.

33:29 Yeah.

33:30 Yeah.

33:30 And like, I don't know.

33:32 One thing that I, I guess I might feel weird, like highlighting all the big projects.

33:36 Cause one thing that I really appreciate is just like anyone who's willing to try a tool

33:41 like this.

33:41 Sure.

33:42 Like moving tools and trying new tools is like a big decision.

33:45 I just like really appreciate everyone who says nice things about it or everyone who's

33:49 like willing to give a try, give it a try, like no matter what they're working on.

33:53 Yeah.

33:53 It's not a platform to say that out loud.

33:55 So hopefully people hear it.

33:57 Yeah, absolutely.

33:58 So the other thing I want to highlight about the repo here, not just the stars, but when

34:03 I look at a project and I want to sort of assess, I really want to adopt this thing.

34:08 Do I want to make this part of my dependencies or things or layers that I got to live with

34:13 here?

34:13 And if it goes sideways, I got to change something is, you know, how old is the project?

34:19 How active is the project?

34:20 And so on.

34:21 So look, we started this podcast 40 minutes ago.

34:26 The last commit was 52 minutes ago.

34:28 Oh yeah.

34:30 Sorry.

34:30 Yeah.

34:31 It's kind of.

34:32 That's awesome.

34:32 I mean, you're like, but you're looking here and it's then it's, it's yesterday.

34:36 It's, it's last week.

34:37 I want to point out there's just a lot of activity, 78 contributors.

34:41 It's a very lively project and not just something that's cool, but you know, there's a lot of interest

34:45 and energy in it, which I think is a very strong signal that it's something people should

34:51 be comfortable adopting.

34:52 Thanks.

34:52 I appreciate you calling that out.

34:54 I think I guess companies one, that's actually something that when I talked before about

34:58 why would these big projects adopt this thing that like seems really experimental, that's

35:03 actually something that comes up a lot is, and I'm not in the conversation, but obviously

35:07 I'm sleuthing a little bit on the PRs.

35:09 Right.

35:09 And they're saying, well, it doesn't support X, but it's like really actively developed.

35:14 And I'm like very confident that they will support that, you know, at some point.

35:18 Yeah.

35:19 And so like, just like having something that is clearly very actively worked on, I think

35:24 is quite like appealing and comforting to people.

35:26 Yeah.

35:26 I actually like do feel bad saying this, but like one of my favorite, not favorite, but

35:31 it kind of an interaction that happens on the repo is someone will file an issue and they'll

35:38 say, this is actually also a bug that exists in the, let's call it the upstream implementation.

35:44 So the existing Python version of that tool also has that bug.

35:48 Right.

35:48 Like Pyland or FlickAids or something like that.

35:50 Yeah.

35:50 It does this, but we think it's wrong.

35:52 We disagree with it or something, right?

35:54 Or there's an issue where, you know, the maintainer is like, oh yeah, this is broken.

35:58 Yeah.

35:58 Yeah.

35:58 Okay.

35:59 It's really not like the reason I tell that thing is like, I, you know, every project is

36:02 different and like every maintainer, like most people, like everyone's doing this like

36:06 for free, right?

36:07 Like I'm not, I'm not here to like call out other maintainers, but I think it's a very satisfying

36:11 interaction for people when, you know, they link to an issue it's been open for, for maybe

36:15 for like years.

36:16 And then I can actually just like fix it in like, like pretty quickly and ship a release

36:21 in the same day.

36:22 And like that, I think that's like a really like powerful thing that resonates with people

36:27 a lot.

36:27 It's just the feeling of like stuff getting shipped and like things getting fixed.

36:32 And yeah, I'm like, you know, I'm working on this full time.

36:35 So like, I'm fortunate to be able to do that.

36:37 And like, I totally get it that like other projects are in different states and stuff.

36:41 But I get a lot of, you know, and people are always super appreciative.

36:45 And so it all just like funnels into, you know, giving me energy.

36:49 Eddie has called you a commit machine.

36:51 Literally every time I update my project dependencies, I'm guaranteed to see new stuff on Ruff, a new

36:56 version of Ruff.

36:57 I appreciate that.

36:58 You know, it's, I do appreciate that.

37:00 Like, yeah, I do try to write a lot of code, but you know, it's obviously not just, not just

37:04 me and like the contributors, the contributors have been awesome.

37:07 And I think, I mean, one thing I'm particularly impressed by is just the number of, number of

37:12 people who, like, I don't even know if I always believe them when they say this, but like

37:16 just the number of people who are like, this is my first time writing Rust or like, I'm

37:21 really not good at Rust or like, blah, blah, blah.

37:23 And hear something amazing.

37:25 Yeah.

37:25 And they just like make like great contribution.

37:27 They're like, Hey, I'm trying to learn Rust.

37:29 And I obviously love that stuff.

37:31 Like for interested in learning Rust, I actually think Ruff is like a great, a great project

37:35 and like a great place for that.

37:37 But I mean, there are people who are working on stuff, especially the stuff that I just

37:41 like, it's like big things I can't like focus all my time on and just like having great

37:45 contributors who are like really, you know, like thoughtful and like prolific.

37:50 Every project is lucky to have people like that.

37:52 How old is the project?

37:54 When did you start it?

37:55 In August.

37:55 Yeah.

37:56 So it's not that old.

37:57 My brain is like melting, but it's somewhere.

37:59 Yeah.

37:59 I think it's like, like somewhere between four and six months.

38:02 Okay.

38:02 Did you say you're working on this full time as like your full time job?

38:06 Yeah.

38:06 Okay.

38:06 And so how's that work?

38:08 Is that GitHub sponsors or are you employed by someone to completely work out?

38:12 What's the story?

38:13 That sounds awesome.

38:13 Still figuring out, you know, I'll probably share more on that soon.

38:17 Okay.

38:17 I don't know if sponsors enabled right now.

38:19 Still figuring out exactly what, like how I want that all to evolve.

38:22 Right.

38:22 I am working on it full time.

38:24 I don't anticipate that changing in any way.

38:25 Like I, I'm really happy in getting so much out of this and yeah, I think there are a lot

38:31 of ways to make it work.

38:32 And if not rough, I want to be working on like other Python tooling.

38:35 So, you know, I think there's like more stuff to come.

38:37 Well, it certainly is neat tooling.

38:39 Let's go, let's go through the GitHub page here a little bit.

38:42 So we've talked about the speed a couple of times.

38:45 We talked about ways in which that happens.

38:47 You know, Rust is an important part, but it sounds like it's not the only reason that it's

38:52 fast.

38:52 You have a chart with the obvious like a little asterisk caveat benchmarks.

38:58 It's reproducible, you know, like all the stuff's in the readme, but yeah.

39:02 That's fast in this situation, but my situation is different in which case, you know, such

39:07 and such.

39:07 Right.

39:07 But for some large code base, we've got rough in 0.3 seconds, autoflake at six, pylint at

39:16 over 60 seconds, just barely.

39:18 This is on the CPython code base from scratch, which is pretty ridiculous.

39:23 And I already told you the numbers.

39:24 I don't have a millisecond number, but it appears to be instant on 20,000 lines of Python code

39:31 that I wrote.

39:32 And it found some issues, which I thought would already be discovered.

39:35 So that's pretty excellent.

39:36 Nice.

39:37 It's nearly Python 3.11 100% compatible, right?

39:41 You said the one major language feature is the match statement, but other than that, it's

39:46 compatible, right?

39:46 Correct.

39:47 Yeah.

39:47 Okay.

39:48 It supports some of the other language features like that are maybe less high profile, like

39:52 the parenthesized width statements and stuff like that.

39:55 But it doesn't support the pattern of structural pattern matching yet.

39:59 Got it.

39:59 So it's got the autofix support near parity with the Flake 8 rule set, which is cool.

40:04 You talked about the way that you keep the performance fast, but still have parity with Flake 8 is

40:11 you have native re-implementations of the Flake 8 plugins.

40:14 That's right.

40:15 Yeah.

40:15 So we re-implement everything in Rust.

40:18 And a lot of the time, that's like me or whoever's working on it, like looking actually at the plugin and

40:24 like the logic it uses and trying to mimic that in a high fidelity way.

40:29 And often we'll like try to use the same test suite and really try to make sure that we're compatible, except in cases where we think there is like an actual sort of factual error.

40:39 Yeah.

40:39 Why rough is fast?

40:41 The plugin stuff, there's a lot of interesting stuff to say about like plugins, because I think Rust is a big part of why rough is fast.

40:50 But there are a couple of other big pieces.

40:53 So we take a lot of care in the code base to like only do necessary work, if that makes sense.

40:59 So, you know, we support 300 something rules, but depending on which exact rules you have enabled, we'll skip a lot of unnecessary work.

41:07 Okay.

41:08 That can help a lot.

41:09 So, you know, it's basically tailored to how much you're using it.

41:12 So like if you're not using it as an import sorter, you know, there's a bunch of stuff that we get.

41:16 It's just how I would think about it.

41:17 But the other like really big one is because everything is sort of implemented in like one system.

41:24 There's a lot of work that would normally be duplicated between a lot of different tools that can actually just be done once.

41:30 So the way to think of that is like if you look at like flake 8 with flake 8 doc strings, there's maybe one example which uses PyDoc style.

41:40 Sorry, I know it's like a huge like a tree of tools.

41:42 You know, if you add that plug in.

41:44 So PyFlakes is going to parse your source code and turn that into an AST.

41:48 And then PyDoc style actually has basically its own parser that will go back over the source code and look at all the tokens and extract the doc string.

41:56 And if you have a bunch of different tools, like some of them will share that abstract syntax tree.

42:01 But a lot of them end up doing work that other tools are also doing.

42:04 Right, because they can't count on necessarily that being done or the plug in system doesn't provide work that's already been done to them.

42:12 So they got to start over anyway. Right.

42:14 Yeah. Or they might want to do things in like a slightly different maybe you have like three different plugins.

42:18 Like I guess one example is a lot of plugins like need some way to tell if a function is like public or private.

42:24 And it's not like super complicated, but you know, it's not like totally trivial.

42:28 Like you need to account for functions that are in inside of classes.

42:32 Like is it a public method or private method to account for functions that are inside of other functions.

42:36 Right. So there's like a lot of rules actually for like determining what's like public and private.

42:41 And, you know, you might have one lint rule that's like all public methods need to have doc string.

42:46 And you might have another rule that's like all public methods need to have type in two things.

42:50 And if you have two separate plugins that are doing that, they both need a way to actually like extract that information.

42:57 And, you know, it's not necessarily the case that like it'll be way faster to do it with one pass and one.

43:03 I think it will. It may not be like or as magnitude.

43:06 But the other thing is you have a lot of like consistency. So like, you know, it's defined in one way.

43:11 You're not going to be at the whims of like small decisions that the tools have to make.

43:16 For us as like maintainers and like building rough, you know, there's a lot of stuff you like don't have to implement multiple times.

43:21 Right.

43:22 I can leverage that and like go and implement other lint rules.

43:24 And I don't have to like figure out from scratch, how do I determine like what public and private in it?

43:28 How do I like extract all the doc strings from a file? Like we already have all that infrastructure.

43:33 So there's kind of like economies of scale, I guess, to like implementing new stuff.

43:38 Like it sort of snowballs a little bit, like the tool gets more powerful, it gets kind of easier to do things.

43:43 I mean, I think you'd be surprised like how much of building a tool.

43:47 I don't know about you personally, but like I was surprised by like how much of building a tool like this is just done.

43:52 It's like settings and like configuration and like how do people express their settings?

43:58 What is the command line interface look like?

44:00 How do you find all the Python files in a directory?

44:03 Like it sounds trivial, but like, you also want to respect like getting more is do you want users to be able to ignore files?

44:09 Like there's a lot of rules.

44:10 And so again, it's like if I look at a new Python tool, you know, that wants to go off and implement something,

44:15 they might have to implement all that stuff themselves.

44:17 But for us, it's like we have all that infrastructure in place.

44:21 And so if we want to build a new fun piece of functionality, it's just sort of defining the rule and we get all this other stuff for free.

44:27 Yeah, you can definitely build on the layers that are in place because it's all together.

44:31 That's cool.

44:31 Yeah.

44:32 All right, let's look through here a little bit.

44:34 Like I said, there's the testimonials from the different projects like Brian from Bokeh or Sebastian from FastAPI and whatnot there.

44:43 But let's start with installing it.

44:45 Like I said, you would know that this wasn't just a Python tool you could use.

44:49 pip install rough. That's the way to go.

44:51 But when I installed it, I didn't pip install.

44:54 I pip x installed it.

44:56 Are you a fan of pip x?

44:57 Yeah, I use pip x.

44:58 Yeah, to me, it seems like rough is something I want to run on different projects.

45:02 And I just kind of want rough in my tool set.

45:05 And so I pip x installed rough.

45:08 What do you think?

45:08 Good idea.

45:09 Is it better to put it, say, like in a per version of per project?

45:12 Like everything.

45:13 I think it probably depends.

45:15 Yeah.

45:15 I mean, I definitely saw a lot of stuff at pip x.

45:17 I think the one thing that I, the one issue with rough, and I guess I recognize wines can be annoying for some people, for sure.

45:25 Like we released a lot.

45:26 And so if projects don't pin their dependencies, then they can start to see like new errors as we add new rules, which I guess in some sense is like a breaking change.

45:36 But I don't really consider it as such.

45:38 It's like, you know, they have the like, let's say they have the like, like a simplify rule set enabled and we add a new rule and that gets turned on for them.

45:47 So anyway, the point is like people, I guess I recommend pinning rough if you're using it for a project, which I mean, with pip x, you're just going to stay on one burger.

45:55 Right.

45:55 So you run pip x, you upgrade dash all or whatever, or upgrade that thing.

45:59 Yeah, exactly.

46:00 It's doing it to a sense.

46:01 It's kind of a pin, but it's pinned globally and it gets upgraded globally and that not, not per project.

46:06 Yeah.

46:06 Yeah.

46:07 Yeah.

46:07 For me, that's kind of okay.

46:08 But if I was on a large team, you know, lots of people who were working on it, you might want to control that more.

46:13 I can see that.

46:14 We have a homebrew distribution and a condo distribution on condo forage.

46:20 And then I did those and then there's everything else was done by other people.

46:25 So I'm like, which is cool, but most of me, I'm like less familiar with them.

46:28 Yeah, sure.

46:29 I mean, I don't know.

46:30 Publishing on homebrew and condo forage was like pretty interesting experience for me.

46:33 I've never done that and I didn't actually really know that.

46:36 much about how those worked.

46:37 I think so.

46:38 Yeah.

46:38 Yeah.

46:39 That was cool.

46:40 Hopefully it's useful for people.

46:41 There's definitely some complexity with like the packaging and release stuff, which I don't

46:46 know if you want to talk about that or not, but you know, because we're building Rust, there's

46:49 like a couple of things that we need to do that are a little different from a standard Python package.

46:53 Maybe just touch on it a bit because what you deliver is a wheel, right?

46:58 But it's got the compiled Rust bit for me on my ARM 64 macOS build, right?

47:04 Yeah, that's right.

47:05 Yeah, that's right.

47:06 So basically it just means we have to do the wheel in some sort of pre-compiled version of

47:11 the package as opposed to just giving you the source code.

47:14 And like if I just gave you the source code and not the wheel, you'd actually need to have

47:18 Rust installed because you'd have to actually build the thing and install it, which is of course

47:23 an extremely unreasonable expectation for the kind of people who are using this for the most part.

47:29 Yeah, but before the wheels really became popular, that was part of the step, right?

47:33 It would build for you.

47:35 Yeah.

47:35 Yeah.

47:35 Or it wouldn't build for you, which was frustrating, but that was part of the step, right?

47:39 Yeah.

47:39 Yeah.

47:39 We just build wheels for all the platforms that I know of.

47:43 It's not something.

47:44 Is it all CI/CD?

47:45 You just push it to some branch and magic happens and then wheels start appearing in places.

47:50 Whenever I cut a release on GitHub, like I just use like GitHub releases, basically.

47:54 I just use GitHub releases.

47:56 It kicks off our release job and then that's all through GitHub action.

47:59 So it just has a big matrix, right?

48:01 Of all the different operating systems.

48:03 We only have to build, thankfully, we only have to build one wheel per operating system.

48:09 We don't have to build one wheel per Python version, which is something that can happen sometimes.

48:13 And then you have like a mass explosion of number of wheels.

48:16 That's all facilitated with a tool called Maturin, which I'm a big fan of, which is,

48:21 it's sort of like, I don't know, like set up tools and a bunch of other stuff bundled together,

48:26 but specifically for shipping mixed Rust Python.

48:29 So I don't actually have to like do anything for the host part to make that work.

48:33 Yeah.

48:33 My pyproject.tomol has, you know, Maturin at the top and I have a Rust project and I call

48:40 Maturin build and it creates a wheel.

48:42 It's all like, I don't know.

48:44 I think it's like super cool.

48:45 That works.

48:45 Yeah, that sounds super smooth and super neat.

48:49 Yeah.

48:49 And like I said, as a consumer of it, also very smooth, right?

48:53 It was still quickly.

48:54 I didn't have to wait on some build.

48:56 I didn't need build tools or config that I don't care about or don't use.

48:59 I'm pretty sure I do have Rust, Rust C as a CLI option I can use here, but you know,

49:05 you don't want to depend on that, right?

49:06 No.

49:07 So you run it on the command line, you say rough and you point at either a file or a directory

49:13 or some pattern of files, right?

49:15 Yep.

49:15 Star type of thing, which is cool.

49:18 And that gives you the output.

49:20 You can also do it with a --watch and just run it constantly, which is pretty cool,

49:25 right?

49:25 It's fast enough that basically as you save it, if there's problems that'll just appear

49:30 in the terminal, right?

49:31 Or wherever you run it.

49:32 That's kind of a funny example because I based, or it's a good example because I,

49:36 I'm pretty sure I took that exactly from probably from TSC, which is the TypeScript compiler.

49:43 Yeah.

49:43 They have a watch setting and I like how it works.

49:46 And so I wanted that for, so it's just a, that's a very literal example of me looking at

49:51 another tool from like another space and being like, oh, that's really nice.

49:55 And I wish you had that other tools like it too.

49:57 Yeah.

49:57 I suppose it's terrible and not terribly difficult to implement, right?

50:00 You take the pattern and you put a watch in the OS watch implementation and you say,

50:06 call me back when something changes and because it's fast, it just reruns, right?

50:09 But it's still, Yeah, it's exactly right.

50:11 It's very handy though, that, that it's there.

50:13 The other thing that's interesting is you can do it as a pre commit hook, which is nice.

50:18 Yeah.

50:19 That works pretty well for people.

50:20 I think a lot of people seem to use it.

50:22 I'm glad that it exists.

50:23 Yeah.

50:24 A little bit of a weird setup because maybe it's actually not that interesting to talk

50:27 about, but it's in like a separate repo for sort of like, I can't even really remember.

50:31 Okay.

50:32 Interesting.

50:34 Now to configure it, it has a, let's say comprehensive CLI set of options.

50:39 I can't remember maybe 15 or so command line options.

50:42 You can pass it, but you can also put, put it into the pyproject.toml with things just

50:48 like tool.rough and then you can put your various settings in there.

50:51 Right.

50:51 So it just sort of integrates along that, or you can have a rough.toml and then put it in there.

50:57 Yeah.

50:57 So that's something that I took basically from hatch.

51:00 So within hatch, you can have a pyproject.toml or a hatch.toml and the hatch.toml kind of like

51:07 tests the configuration.

51:08 Like if you use a rough.toml, you don't need to have like tool.rough at the top and stuff like that.

51:13 Yeah.

51:13 I think one thing that I'm quite happy that we have is like the configuration is very like type safe.

51:21 So if you try to like, you know, so that, that line select equals EF, those are like error,

51:26 like rule codes that are defined in rough.

51:29 And so if you put, if you replaced F with some string that wasn't, you know, a real rule,

51:34 you'll get an error immediately saying that, you know, your pyproject.toml had invalid,

51:38 was invalid in like these ways, blah, blah, blah.

51:40 Interesting.

51:41 Okay.

51:41 And similarly, we create what are called like JSON schemas or the configuration.

51:47 If you're in PyCharm and you open up like a pyproject.toml and you type tool.rough, it'll actually like show you all the options.

51:54 It'll actually surface the documentation we have for the options too.

51:57 Oh, nice.

51:58 Which is pretty cool.

51:59 Yeah.

52:00 I'd like more tools to do that.

52:01 Cause I think it's like super useful.

52:03 I think poetry has it, but not too many others do.

52:06 You can get it in VS Code as well, but you need to install an extension.

52:11 It's called like even better.toml, I think.

52:13 Right.

52:13 Okay.

52:14 But I'm happy with those things.

52:15 I think it makes it like, I think small things like that add up and glad we're able to do them.

52:20 Yeah.

52:21 When I looked at the configuration settings, like the exclude directories and various other

52:25 things, like that's kind of a lot to put on a command line.

52:27 It would be nice if you could just put it and say rough.toml or pyproject.toml.

52:31 And then when you just type rough the thing, it, it sees that and it uses it.

52:35 Right.

52:35 Yeah.

52:36 I've kind of like intentionally like not put a lot of these arguments on the command line.

52:40 Like for that reason, like you can imagine taking everything here and exposing it on the command

52:44 line, but the command line really just deals with like how should the thing execute and not like,

52:49 I don't know, what should the max complexity be?

52:51 Yeah.

52:51 Yeah.

52:52 Yeah.

52:52 Yeah.

52:52 Yeah.

52:52 I think that's fair.

52:54 Like put the common stuff on the command line and then say, and if you need to go crazy,

52:59 here's your --config, pass that in and put it in there.

53:03 Right.

53:03 Yeah.

53:03 Yeah.

53:04 Yeah.

53:04 Yeah.

53:04 One thing that I did want to talk about and you kind of touched on, there's a bunch of the

53:08 command line arguments.

53:09 One of the things I did want to talk about is the editor integration.

53:12 There's a huge bunch of rules here.

53:14 We got to scroll through on the homepage to get, get down to it, which is awesome.

53:18 That are no, no, it's good.

53:19 I mean, these are like all the things that checks that I don't have to think about anymore,

53:22 which is great.

53:23 Eventually though, geez, I forgot it was this long.

53:28 Eventually though it has plugins or maybe that's not quite what I said.

53:33 It has a, an extension of a VS Code and it has ways to basically add it as a command

53:40 to PyCharm, right?

53:41 Which is, those are both awesome editors that are kind of very friendly to thinking about code

53:46 formatting and exposing the errors that this would as well.

53:50 So that's great.

53:50 You might want to just talk about that just real quick.

53:52 Yeah, totally.

53:53 So yeah, we have a VS Code extension.

53:55 That was my first time writing a VS Code extension.

53:57 And actually the, these extensions use, or they conform to something called the language server protocol.

54:03 And effectively that's sort of like a standard, I believe I'll probably get like a bunch of

54:08 the history wrong, but I believe it's Microsoft that like publishes and maintains that.

54:12 It's that.

54:13 The nice thing about that protocol is you can actually use like the lot, like most, almost all

54:19 of the code behind the VS Code extension is actually published as a language server.

54:23 Yeah.

54:24 Rough LSP.

54:25 And what that means is you can actually get like the exact same behaviors in all sorts of different

54:29 editors.

54:30 It's not confined to VS Code.

54:32 So if you use like NeoVim or like sublime text or whatever else, those all support this standard

54:38 protocol and they all have ways to tap into that rough language server protocol.

54:44 So we have sort of like the standalone LSP, which we have instructions for like sublime and,

54:51 and you know, other tools.

54:52 And then we have the actual extension, which is published on the marketplace.

54:56 And those are things I definitely want to like keep supporting.

54:59 And I view them as like official arms, right?

55:02 Of the project, which I think is nice.

55:04 So people should know that like, you know, those are official and like, I pick bugs and

55:08 man, and all that kind of stuff.

55:09 Nice.

55:09 And then for PyCharm, you just add it as an external tool and then you have the rough command

55:15 around or for both of these, if you prefer, you just add it as a pre-commit hook.

55:19 Yeah.

55:19 That's also.

55:19 Yeah.

55:20 That's right.

55:20 Yeah.

55:20 That then like, as you interact with it, it'll run the pre-commit hook as well.

55:24 If you try to check things in.

55:26 Super cool here.

55:27 I think this is a neat project.

55:29 I ran it into my project and I thought, you know, everything is pretty much good to go.

55:36 Where, you know, when I'm using PyCharm, for example, it'll often tell me many of these things.

55:42 Like it'll say there's a local variable that's unused or it'll automatically remove unused imports.

55:47 And yet I'm looking at 331 issues.

55:51 108 are potentially fixable with the --fix option.

55:55 So it sounds like there's still a lot of value to like make that part of my tool chain.

56:00 Yeah.

56:00 I mean, you know, and feel free to have some of your own opinions about which of those rules actually matter.

56:05 Sure.

56:06 And I suspect you can disable them in your config file, right?

56:08 You can say this rule I don't care about.

56:10 Yeah.

56:10 And this is another sort of compatibility thing.

56:12 Like we, API for actually like turning rules on and off is like essentially identical to the flight gate API.

56:19 So that's also useful for people migrating over.

56:21 So the no QA, is it the like comment?

56:24 Yeah.

56:24 Like comment no QA is one option.

56:26 Yeah.

56:26 We respect.

56:27 Yep.

56:27 We respect those too.

56:28 We also respect like, I sorts action comments.

56:31 So yeah, we, we like do a lot of stuff to try and maintain compatibility.

56:36 Nice.

56:36 Maybe another like interesting thing in that vein is that we ship a tool called

56:40 flight gate to roughs, which does sort of what it sounds like.

56:45 So you point it to a flight gate configuration file and it spits out a Tommel for rough,

56:50 you know, that's compatible.

56:51 And it also tells you if there were things in the file that we don't support, which is useful too.

56:55 Like, oh, hey, you're using these rules.

56:57 Those actually aren't implemented yet.

56:58 Yeah.

56:58 That's also maintained as like part of the project.

57:00 So, you know, I recommend checking it out.

57:02 Excellent.

57:03 Well, Charlie, I think we're getting short on time, but not short on topic.

57:06 There's a whole lot more to dive into.

57:08 I think it's, it's really cool project.

57:10 I think it's new.

57:12 Are we gonna do a big WASM section?

57:12 No, I'm kidding.

57:15 I'm kidding.

57:15 We don't have time for that.

57:16 We could do a WASM section.

57:18 We even have some great chat GDP questions in the audience.

57:22 I don't necessarily think that there's time for those, unfortunately, but there is one final question that I want to ask.

57:27 It's kind of inspired by the audience, but I think it's good for everyone.

57:30 Jean Gabriel asks, is there a roadmap or timeline for rough development out there?

57:36 Like what's next basically?

57:38 Like, I guess the short answer is no, like there's not like a published timeline, unfortunately, like most of this stuff exists in my head and in issues.

57:47 Like everything happens on issues.

57:49 Like we're just talking a lot on issues.

57:51 Yeah.

57:51 I'll maybe talk a bit about like the things that I'm thinking about.

57:54 So one is fixing like the structural pattern matching issue.

58:00 And so like getting full like Python compatibility, which it's sort of like one bug, but then unlocks

58:06 us doing what I hope will be like a stable release, even if that's not that much changes.

58:10 And it's just kind of, you know, the marketing of it, but hopefully we'll fix that.

58:14 And then we can do like a stable release.

58:15 The, there are two other kind of big things on my mind.

58:18 So one is this is like a really big project, but not, not rough.

58:23 The thing I'm about to talk about, but, I want to extend rough to do like full code auto

58:27 formatting so that people could use it in lieu of black.

58:31 If they wanted to think it's like a really natural fit for what rough is doing.

58:36 And a lot of people asked about it.

58:37 The way that that will, there's an issue where we talk about this, like on the repo, it's not

58:42 like some big secret reveal, but, the way that I think about that, like, like with all things is

58:47 I want it to be like, sort of incrementally adoptable.

58:50 Like right now you can use rough with ice or.

58:52 Or you can use just rough, or you can use rough to your input sorting, but like not as a linter.

58:57 Like you can kind of pick and choose like what you want to use.

58:59 And I view like the auto formatter is the same way.

59:02 Like I fully expect that people will keep using rough and black, but it's just like the kind of

59:07 functionality that makes a lot of sense, given what we're already doing and like the way that the

59:11 project's evolving.

59:12 So I need to like, and you be careful because like, it's like that blog work.

59:16 And so I'm not like saying exactly when that will be done, but I do want to start working on it.

59:21 And then the other thing, which is maybe like a little bit less exciting to other people.

59:24 And it's like, you know, I think at some point I'd like rough to get to a world where

59:28 it kind of stands alone a little bit more.

59:30 It isn't viewed quite as much as a compatibility as this compatibility layer.

59:34 Like right now, all the rules kind of map back to like, Hey, or other tools.

59:39 And when you're using rough, you're kind of thinking in the mindset of those other tools,

59:43 like what are the rules and how do I configure them at some point in the future?

59:47 I'd actually like, you know, rough to sort of have its own API, like own indexing of the rules.

59:51 And to view that as kind of a compatibility layer that will not go away.

59:55 Like this won't put a burden on users, but I'd like to get to get to a point where like conceptually, a lot of this stuff is like first class and rough.

01:00:02 And it's not viewed as re-implementations of other extensions, which I think will enable us to do a lot of cool things.

01:00:08 But I'm hopeful that I'm doing that in a way that has no disruption to users.

01:00:12 So we'll continue to support the existing API and such.

01:00:14 Yeah.

01:00:15 Yeah, exactly.

01:00:16 Very exciting.

01:00:17 Very exciting.

01:00:18 Well, congratulations on a project that people seem to love.

01:00:21 It's got a lot of traction and definitely doing some cool stuff.

01:00:24 So before we get out of here, though, final two questions.

01:00:27 If you're going to write some Python code, what editor are you using these days?

01:00:30 I use PyCharm for everything.

01:00:32 And that includes Rust.

01:00:33 But I do all my Rust and my Python and PyCharm.

01:00:35 Nice.

01:00:36 Is there a cool plugin that basically adds Rust support to?

01:00:39 Yeah, there's just an official IntelliJ Rust plugin.

01:00:42 Got it.

01:00:43 Nice.

01:00:43 Works really well.

01:00:43 I've used IntelliJ for that platform for Dart and Flutter, but I haven't tried to plug Rust into it.

01:00:50 But yeah, it does a lot of things.

01:00:52 Cool.

01:00:52 And then notable PyPI package, something out there.

01:00:55 You've named a couple that do the Rust integration nicely.

01:00:58 But anything you want to give a shout out to?

01:01:00 Yeah, I really want to give a shout out.

01:01:01 I've already mentioned it a few times.

01:01:02 I think if you're interested in doing Rust Python stuff, you should really look at Matrin.

01:01:06 Matrin.

01:01:07 All right.

01:01:07 M-A-T-U-R-I-N.

01:01:09 It just makes the publishing really easy.

01:01:13 And you still have to learn Rust, which is not easy.

01:01:15 I mean, I do.

01:01:16 It's worthwhile, I think, but it's not easy.

01:01:18 But it makes the publishing Rust code to PyPI, or PyPI, sorry, really easy.

01:01:23 Awesome.

01:01:23 Well, great recommendation.

01:01:25 Thanks for being here.

01:01:26 Final call to action.

01:01:27 People want to get started with Rust.

01:01:28 What do you tell them?

01:01:28 The main thing that I would like to say is if you're interested in contributing,

01:01:32 even if you don't know that much Rust and are interested in learning, I'd love to see you on the repo, and I hope it's a place that people feel really welcome

01:01:42 to come and contribute, even if they are 100% certain if their code is right or anything like that.

01:01:47 So definitely really excited to have more contributors and have more people coming into the project.

01:01:52 And of course, you should also try using it yourself.

01:01:55 All right.

01:01:55 Excellent.

01:01:56 Well, thanks for being here.

01:01:57 It was a great conversation with you.

01:01:59 Yeah, thanks so much for having me.

01:02:00 It was a really cool opportunity.

01:02:02 I appreciate it.

01:02:04 This has been another episode of Talk Python to Me.

01:02:06 Thank you to our sponsors.

01:02:08 Be sure to check out what they're offering.

01:02:10 It really helps support the show.

01:02:11 Join Cox Automotive and use your technical skills to transform the way the world buys,

01:02:17 sells, and owns cars.

01:02:18 Find an exciting position that's right for you at talkpython.fm/cox.

01:02:24 Earn extra income from sharing your software development opinion at user interviews.

01:02:29 Head over to talkpython.fm/userinterviews to participate today.

01:02:34 Want to level up your Python?

01:02:36 We have one of the largest catalogs of Python video courses over at Talk Python.

01:02:40 Our content ranges from true beginners to deeply advanced topics like memory and async.

01:02:45 And best of all, there's not a subscription in sight.

01:02:48 Check it out for yourself at training.talkpython.fm.

01:02:51 Be sure to subscribe to the show, open your favorite podcast app, and search for Python.

01:02:56 We should be right at the top.

01:02:57 You can also find the iTunes feed at /itunes, the Google Play feed at /play, and the direct

01:03:03 RSS feed at /rss on talkpython.fm.

01:03:06 We're live streaming most of our recordings these days.

01:03:09 If you want to be part of the show and have your comments featured on the air,

01:03:13 be sure to subscribe to our YouTube channel at talkpython.fm/youtube.

01:03:18 This is your host, Michael Kennedy.

01:03:19 Thanks so much for listening.

01:03:20 I really appreciate it.

01:03:22 Now get out there and write some Python code.

01:03:23 I'll see you next time.

01:03:44 Thank you.