#362: Hypermodern Python Projects Transcript

00:00 What would a modern Python project look like? Maybe it would use poetry rather than pip directly for its package management. Perhaps its test automation would be controlled by Knox.

00:10 You might automate its release notes with release drafter. The list goes is on and on. And that list is the topic of this episode when me and Claudio Jollovitz as we discuss his Hypermodern Python project and template. This is Talk Python to Me, episode 362, recorded April 6, 2022.

00:43 Welcome to Talk Python to Me, a weekly podcast on Python. This is your host, Michael Kennedy. Follow me on Twitter, where I'm @mkennedy and keep up with the show and listen to past episodes at talkpython.fm. And follow the show on Twitter via @talkpython. We've started streaming most of our episodes live on YouTube, subscribe to our YouTube channel over at talkpython.fm /Youtube to get notified about upcoming shows and be part of that episode. This episode is sponsored by Microsoft. For startups foundershub, check them out at 'talkpython.fm/foundershub' to get early support for your startup, and it's brought to you by compiler from Red Hat. Listen to an episode of their podcast as they demystify the tech industry over at talkpython.fm. Compiler transcripts for this and all of our episodes are brought to you by Assembly. Ai do you need a great automatic speech to text API? Get human level accuracy and just a few lines of code? Visit talkpython.fm/assemblyai. Claudio welcome to Talk Python to Me.

01:45 So happy to be here.

01:46 I'm so happy to have you here. It's great to be talking to you. This is one of those episodes that's going to be so fun because what it's going to turn out to be, I'm pretty sure is diving into a ton of little tools. And I can tell you just doing a little bit of research and putting together some show notes for this. Like, oh, there's that thing. And oh, look, this too. Oh, I didn't know about this. So you've assembled this conglomeration of tools and techniques that you're putting under the Hyper Modern banner. And I think it's going to be a lot of fun to talk about. So we're going to have a good time looking forward indeed. Same. Now, before we get into that, let's talk about your story. How did you get into programming and over here to Python?

02:25 I think one day my dad that must have been in the 80s came back with and said I bought a computer and I was really excited. I imagine there's going to be a room filled with all these machines and ran down the corridor and turned out to be like some kind of keyboard.

02:43 There was a Commodore 64, and initially we just played all those great eight bit games. And eventually I started programming a little bit in Basic, and I think that's kind of when I really found out how much fun this is. And then I think I was interested in a lot of other non computer things for a long while.

03:05 I went to UNS studied law, as most programmers do.

03:08 Of course.

03:11 But somehow this interest in formal systems always stayed with me, and especially Continental law. German law is very much like a little bit like a calculus tracing back to ancient Rome. And I got interested in logic, and there's a small research community working on applying AI and logic to legal theory, and that was really my gateway drug to get back into programming, really logics. I think I programmed this like a little Flashcard system to help me prepare for the law exams.

03:45 Nice.

03:46 Eventually I decided I want to get really deep into this, and I started studying computer science and pretty much never went back to law after that. So I have a law degree, but working as a software engineer, it's interesting.

04:00 I hadn't really thought about it with law. I have a friend who's a lawyer in software, so I know it happens for sure, but thinking about the way you have to mentally sort of solve the problems and the constraints of legal contracts and laws and stuff and how they apply, that's actually kind of a similar skill to think through solving a computer programming problem with APIs and what the computer can do and stuff. Right.

04:24 And it's such a human way of thinking. So it's really interesting from an AI point of view, because it's not the kind of really clear logical deductions that you have, but there's a lot of everyday knowledge that you need to have and the feasible rules. So it's quite exciting.

04:41 Yeah, very neat. Now, what kind of code and what kind of stuff are you doing these days?

04:45 I've been working mostly on cyber security, so working for a company almost 14 years that's doing cybersecurity as a service. So we're working mostly on C++ services or high performance data intensive services.

05:03 We're using Python mostly to automate the build system testing and releases, but also for prototyping so algorithms. It's like really handy before you implement it in a high performance way.

05:17 Yeah, I think Python is used frequently for that. Let's prototype this and then once we completely decide it works right, then we're going to write it in C++ or Rust. That's not the most common use of Python, but it certainly is one that people said, oh, this is really good because you can prototype so quickly. Sometimes people just decide. And also this will just work fine for what we're doing. Actually, it's plenty fast. Or they decide, you know, maybe not right, maybe they need C++, but it's still a cool use case. Now let's kick off our conversation with some thoughts from a former guest.

05:49 He had a really interesting way of sort of presenting Python to people who are not deep in the Python language and said, basically it's actually when people say Python is great for prototyping, for example. Well, they might be talking about one of three things or some combination of that. It could be when people say, oh, Python is good for this, or Python is like that. They might be talking about the language, or they might be talking about the standard library or more and more these days they're talking about the third party ecosystem with I don't even know how many libraries. I got to look this up because it changes so fast right now at the time of recording 368,000 libraries. So when people mention Python, they often mean one or more of those different things. We're going to talk about hypermodern Python. So I think we should frame it a little bit in the sense of like, well, what is modern about the language or what is kind of modern about the standard library? And obviously the ecosystem is where a lot of it is happening. So from your point of view, what is modern Python before we get the hypermodern?

06:54 Yeah, definitely. We can talk about the language, the standard library, the ecosystem. I'd also add the community. I think that's something that really defines Python and all that tooling that evolved in the ecosystem.

07:11 So about the language, because my story is I think I got into Python. Python was it like Python two.

07:20 Three, you've been through the journey, you've been through the great split, and they're rejoining.

07:25 I pretty much missed a lot of the pain of the Python two, three transition. I've been busy with C++ and then came back to Python. And for me, it was just the enthusiasm of rediscovering all the great, like how expressive Python had become. One of the things that really got me excited about modern Python is type annotations. I just find them so helpful to structure programs, to make APIs clear, to help me think about my code and to keep it maintainable and readable. So I use them pretty much always, and I always run MyPY in strict mode.

08:00 I even use it in small scripts. And it's of course very helpful in large systems.

08:07 Absolutely. I don't know that I've. In fact, I'm pretty sure I've never gone in to end and taken a large system and completely made it 100%. mypy checked for me. I'm with you. I absolutely love the types, and I use them a lot to sort of drive the tooling intelligence. For example, if you got a data access layer, you could talk about what is exchanged at the boundary there so that your Editors are all of a sudden super smart about autocomplete. And I was just doing a massive overhaul to the Python Courses website and it was 110 commits in this PR. It was like ridiculous. But before I checked them all in, I went through and I said, okay, look for all the type warnings. Look for anything that might have become out of sync along the way. And I caught one or two things before I accepted the merged the PR back in. So it's just yeah, I absolutely think that's one of the most important additions.

09:05 And also it's so nice how you can leverage them at runtime as well. Not only that, they allow you to check your code in a way that doesn't require hitting every code path, but you can build data validation on top of it and so many more.

09:21 Yeah, absolutely. We look at libraries like Pydantic and FastAPI that are making interesting runtime use out of it.

09:28 Absolutely.

09:28 And speaking of modern Python, there was that proposed change to make typing more efficient where it wouldn't actually import the things until really needed it. Or I can't remember the exact pet, but it was a way to sort of delay type information imports until you're doing something like mypy and the Pydantic people. And Fasting and FastAPI is like, wait, we need this. This is how our thing works. If you take away the actual meaning other than for verification the string of of the type.

10:00 That makes it really hard for these use cases.

10:03 And there's this other approach where they basically lazy evaluate the types. I think, to avoid this string problem. I don't know if I think it's still an open question of how to proceed this. We should have gotten the string types already and then decided to take some more time to find a good solution for everybody.

10:25 Yeah, I think that was sort of delayed. I can't remember the total outcome, but I think it was like we needed to think about this more and make sure all the use cases are covered. Right. Yeah. Cool. Okay. Well, that's the language. One thing that we could talk about real quick that's pretty timely is this Pep. I just had Brett Cannon and Christian On to talk about Pep 594 removing dead batteries from the standard library, which is pretty interesting. The idea is a lot of these libraries had been added. These core modules have been added to the standard library in like 1992, and they might not still be relevant. For example, CGI is not the most common way to do web apps anymore. We've got Microwsgi and gunicorn and all those things. Right. So does it still make sense to maintain them?

11:13 Yeah, I think the set of libraries to remove them was pretty non controversial because I agree they're just very old, obsolete, and also basically unmaintained. See, Python has, I think like 90 core developers, and they have 1600 open PRS right now. So it's very hard to maintain a huge standard library with so little human resources.

11:39 So I think that was pretty good step. What I really find interesting is the vision behind it, like, where should the standard library go? How are the criteria in the future to include libraries? For example, I think recently we got Tomoldib into the standard library.

12:01 Yeah. To join JSON and CSV and XML and all those. Yes.

12:05 It started as a PyPI library called Tom Lee, and it's been adopted quickly by a lot of the tools out there. And now it will be part of the standard library. So, for example, this is something that is important to solve a bootstrapping problem in the packaging ecosystem, because we have PYPi Project Tommy now. And how is Pep going to pass the Pipe Project Tommy file, for example? How are the other tools going to pass it? So it's very advantageous to have it in the standard Lib, but we probably don't want to have passes for every file format out there.

12:43 In fact, a lot of the ones that were removed in Pep five, nine, four are actually having to do with file format. So you've got like AIFC, which is an audio format. You have audio up, like the sun. Au format. There's a bunch of things like that, right? Yeah. I don't want to dive too much into that because we've done a whole show on it. But I do think it's interesting to think about this as the first step in a modernization of the standard library. Right.

13:10 Yeah.

13:11 And then where it really blows open. And I think an interesting inter exchange sort of influence here is, as I mentioned, the 368,000 external packages that are building on a lot of the new language features that I think are super cool. And how much looking back, if that world existed already, how much smaller would the standard library be? Right. Like, would your Lib ever have to be in there? Well, we got requests. I don't know. Maybe not. It could make a lot of sense for it actually to be there, like this bootstrapping problem you talked about, but maybe it doesn't. Right. I think different choices have been made, but yeah. What are your thoughts on the ecosystem from the PyPi perspective?

13:49 So if we look at the third party libraries, I think for me, modern Python is a lot about expressive types, like address. For me, it's like the best example really, to how to write well structured code using attrs Just got a new API, which is really nice, and I can definitely recommend having a look at it. So it's become very easy to define immutable value objects, essentially that will allow you to basically structure your domain logic in a really expressive way.

14:23 Yeah, I was going to say that it did get a new API recently. I forgot that it kind of inspired data classes, and then it sort of turned the tables on it a bit. Right. And rethought about how some of its stuff was offered as well. Right.

14:34 I love that we have data classes in the standard library because it's like a mini address that you always have at your disposal, even when you don't want to take on third party dependencies. But it's definitely always worth looking at errors. It's very fast and has a lot of features. And it doesn't have this problem that you can only really update it once per year.

14:56 Yeah. And that's a super interesting point. They considered putting requests in the standard library for a while, and requests is under the I can't remember the exact organization, but it's under an official Psf maybe. I think it is just the Psf organization now on GitHub. Right. It's sort of officially Python in a sense. But they decided not to put it into the standard library, not because it didn't fit or it wasn't good enough, but because it would actually slow down the development of requests and constrain it too much. And it's sort of similar here as well. Right. Adders can come out every day with new stuff and data classes yearly. Right.

15:31 This portion of Talk Python to me is brought to you by. Microsoft for Startups Founders Hub Starting a business is hard. By some estimates, over 90% of startups will go out of business in just their first year. With that in mind, Microsoft for Startups set out to understand what startups need to be successful and to create a digital platform to help them overcome those challenges. Microsoft for Startups Founders Hub was born. Founders Hub provides all founders at any stage with free resources to solve their startup challenges. The platform provides technology benefits, access to expert guidance and skilled resources, mentorship and networking connections, and much more. Unlike others in the industry, Microsoft for Startups Founders Hub doesn't require startups to be investor backed or third party validated to participate. Founders Hub is truly open to all, so what do you get if you join them? You speed up your development with free access to GitHub and Microsoft cloud computing resources and the ability to unlock more credits over time to help your startup innovate. Founders Hub is partnering with innovative companies like OpenAI, a global leader in AI research and development, to provide exclusive benefits and discounts through Microsoft for Startups Founders Hub Becoming a founder is no longer about who you know. You'll have access to their mentorship network, giving you a pool of hundreds of mentors across a range of disciplines and areas like idea validation, fundraising, management and coaching, sales and marketing, as well as specific technical stress points. You'll be able to book a one on one meeting with the mentors, many of whom are former founders themselves. Make your idea a reality today with a critical support you'll get from Founder Hub. To join the program, just visit 'talkpython. Fm/founderhub' all one Word the links in your show Notes thank you to Microsoft for supporting the show.

17:22 I think that's an interesting aspect of sort of modern Python as well, the ability to just continually deliver new features and adapt it as needed.

17:32 I also really love cattrs because you mentioned Pydantic before. What I really like about theater, it has kind of a similar so you can serialize and decrease data classes and attrss. The difference between Pydantic and cattrs is that Pydantic uses inheritance to give you this functionality, whereas with cattrs you just have your pure Python classes without any they don't need to inherit from anything. You just have decouple serialization logic from your domain logic. And I think that brings a lot of advantages in structuring software.

18:10 This is the right one. I have on the screen here, the C-A-T-T-R-S.

18:14 Right? Yeah.

18:15 Interesting. So you can do things like put a frozen decorator onto just a regular Python class and hey, it's frozen. You can create an instance of it and you can say unstructured and you get a dictionary, you can structure it back until it would type it, parses it back, which is quite neat. This one's new to me. Like I said, we're going to go through a lot of those different things. Anything else you want to give a quick shout out to and the broader ecosystem before we'll dig into your hyper modern ones that are using as well?

18:43 If you haven't seen Rich and httpx, those are definitely some to check out httpx. Basically, we talked about request. Httpx has a very similar interface, but gives you both async and sync operations. Yeah, that's definitely a wonderful project.

19:00 I use httpx a lot and it's really nice because it's so familiar if you know requests, but if you happen to find yourself doing cool Async stuff, you're not stuck not doing async for one of the most important parts, which is calling services. Right. So you can await doing a get or a post or whatever. Yeah, it's really nice. Not 100% switch to it instead of requests, but it's definitely one of my go to as well. Cool. All right. Well, that's sort of some thoughts on modern Python and where things are going. Then you created this series, which is almost like a little mini course on how should you from your perspective, how should you structure and build modern Python projects and what tools should you bring in? Not just should you use httpx over request, but should you use Knox for testing and things like that? Right. You might say yes for knox, you work on it. Right. So you did this article, this six part series article on it, which I'll definitely link to. But then also you created a cookie cutter template that will allow people to jump into it. I find this to be really helpful. I do this sometimes with my classes. Here's the thing we built at the end, but if you just want to beat your own version of it, here's a cookie cutter to actually just beat it with your settings and your values that you don't have to go through. Rebuilding it from scratch. And cookie cutter has been really influential in that sense, don't you think?

20:25 Absolutely.

20:26 When I wrote the articles, there was example code on GitHub and I saw people forking the repository. And I was like, oh, no, they're all going to end up with this example code that displays Wikipedia articles and you console, am I going to keep all the dependencies up to date? And how do I even do that? And I was like, no, I have to find a better way. And cookie cutter was definitely a good way to set up a project template and also something that was much easier for me to keep up to date. So there's after two years, definitely quite a bit of drift between the article series, which is from January 2020, and the cookie cutter as it is today.

21:10 Yeah.

21:11 It's hard to keep an article or a video or a talk or whatever as a living thing that evolves as we gain more experience and stuff. But the cookie cutter template, that's like software plastic.

21:21 You can make it open source and have contributors that do some work for you, which is really extremely grateful for all the contributions I got there.

21:31 Yeah, that's great. I see a bunch of contributors there. Now first, let's start with just the term hypermodern Python. What's the story with the naming here?

21:39 I actually brought, you know, this only for those that have a camera in front of them. This is hypermodern Python. So this is not Python. This is a hypermodern chess game written in.

21:53 So this is where the name hypermodern really comes from. It was meant a little bit tongue in cheek. And also I was very conscious that how is Python going to look like two years after I've written this article series, the ecosystem evolves so quickly, I decided to stick all these images in the blog that are basically past versions of the future. I think it's called retrofuturism. So they're basically all images from the 1920s and so on about how people in the future may be going to fly to the opera using planes that look like little birds.

22:28 Yeah, it's sort of a steampunk mechanical bird. People are cruising along in here.

22:35 It's a cool picture. It's a cool idea.

22:37 So what is hypermodern? I think basically it's just whatever I was excited about and didn't know about beforehand, I had used more the standard tools like setup tools and pip tools and talks. And coming back to Python, I thought, well, let's just check out all the things that have happened and let's see how maybe that might solve a few of the problems that I had.

23:04 Nice. Okay, so then you put together the article, which is a serious article. It's not just a couple of paragraphs. Right.

23:13 It's quite a bit of writing articles. Yeah. Six articles. And my little or maybe it was your website. It's like eleven minutes reading for this part, that kind of thing. So it goes into pretty good detail with examples. And then you captured it in this cookie cutter because to put it into practice, that's a pretty good way now, using cookie cutter is super easy. I'm sure people are familiar with this, but cookie cutter give it the name. It asks a bunch of questions, right? Correct. I think maybe the right way to explore this would be to talk about the features and the steps of the cookie cutter that it does on the page for the cookie cutter GitHub repo, there's a big section that's features and that sort of talks about the different aspects and angles, dimensions you decided to bring in and the tooling there. So how about we just go through these and sort of dive into them? I think people are going to discover some cool tools here. So first of all, you're going to build something meaningful. You've got to get some libraries of pypi. There's very few projects that have no dependencies that are rich these days, and not just rich the package, but feature rich.

24:19 Let's talk about the first one here.

24:21 Okay.

24:21 I want to get through it. Yeah.

24:23 So Poetry really solves a problem for me because it's basically the one to approach. You have one tool that does everything for you. It will allow you to define metadata for your package, build the package for you so you can publish it on PyPI. It can manage environment for you, install all the dependencies of your project and your project itself. And it can also manage the dependencies itself. So it has a resolution mechanism, and it has a log file, which is, for me, a really important feature, because not only is it good for deploying services in a reproducible and deterministic way, but it's also for running the checks on your code and making sure that the checks run exactly the same locally on your machine in CI and on the machines of your collaborators.

25:17 Yeah. They're not as familiar with the poetry lock file as I should be. But basically it's like pinning your versions in a requirements. Txt. Right. But so often people just write, here's my requirements. I have requests or httpx, and I have Bass API, and I have Sequel, Alchemy or SQL Model. And you just type those out and you pick installr and you're good to go until you want to go back to an old version that might have a bug. That's the one in production. But the bug might be because it has the old library of whatever. Right. And you don't know. Right. So you want to be able to pin those versions and then does the lock file also put the hashes in so they can't be fiddled with?

25:57 It does. Yeah. So it has the hashes and it gives you all basically all the indirect dependencies as well. So much like Pip. Compile would do from the pip tools.

26:09 And that is tremendously useful.

26:12 Yeah, it absolutely is. I'm a big fan of the pip tools and Pip. Compile. We'll get to that later.

26:18 When we get to it, there's a section, a whole section in the cookie cutter about it. All right, so it starts by setting up with poetry, right?

26:25 Absolutely. Yeah. I called poetry the one tool approach, and there has been a lot of work in the packaging community to introduce standards, packaging standards that make it easier for the tools to interoperate.

26:42 And I think this is actually a very good way to approach things in an ecosystem like Pythons to make it possible that you can define your package metadata for one tool like Flit, and then use Poetry to build the package or PDM and so many others. And there has been a lot of great work that has made the packaging ecosystem more diverse.

27:10 Yeah. There was some separation of the architecture of certain responsibilities, right?

27:15 Yes. PEP621, which defined the metadata. There was also an attempt to standardize log files, which I think would be great, but that unfortunately has been rejected for now. So I'm hoping that there will be more attempts in the future.

27:30 But so as great as it is to work with poetry, I definitely hope for the ecosystem to become even more diverse and more standardized, to really give us more flexibility and interoperability.

27:44 Yeah, that is great. Definitely seems like a lot of options are coming out around that these days. And then the next one here is test automation with Knox. What's your relationship with Knox, by the way? Are you the maintainer, or are you just a contributor?

27:59 I'm co maintaining Knox. It was written by the Flowers, and it has a large maintainer team. At the time that I wrote the article series, I was just a fan of Knox, and I started contributing to it, and I kind of ended up co maintaining it.

28:14 I'm contributing more than anyone else. Does that make me revolt? All right, so tell us what the role of Knox plays in this.

28:20 So Knox is a great tool. It's not just test automation, it's really it lets you automate basically all the developer tasks you have. So this might be tests, it might be the other checks you have, like Linting, or it could be building your documentation or building wheels, if that's complicated in any way.

28:43 The great thing about Knox is that it uses Python to let you define the tasks rather than basically having something like a make file where you use the shell or in toxic, you have an any file where you enter your commands at least talks. Three, I think talks four is also going to add Python configuration.

29:04 So it's really inspired by talks, I think. So. If you know, tox allows you to run tests on multiple versions of Python, and it's been around for much longer, it's very mature tool. Nox is inspired by that. It also lets you have this matrix of Python versions or even other things. So you can similar to Pytest, you can parameterize your session functions and pass in, say like a specific dependency that you want to test against in different versions. So nox is really useful if you want to have a single entry point into your project maintenance, running all the tasks that you have and running them the same locally and on CI.

29:50 Yeah, that's great. I hadn't really explored this NOX file thing where you have these different tasks, like a task basically being a function. Like you say like Test or Lint, and you can just put a session decorator on it and just say session install pytestion, run pytest or session. Install Flake eight and run it with the parameters. It's really nice and clean and it's way better than a shell script. I think it keeps you in Python, right, right. Which is probably where you want to be on Python project.

30:19 And it runs on all the platforms. At some point I use make files to automate these things. So the Nontox related things, I guess don't work very well on Windows. So you don't have this problem.

30:32 Yeah, that's for sure. Cool. And then we kind of saw an example of that there. But Lenting with Precommit and Flake eight.

30:38 Right.

30:39 What about these two libraries?

30:40 I love Precommit. Actually, it wasn't in the first draft of the article series. I got some reviewers who commented on that I think only functioned from the Pytest project and antic Chlovak both mention that you have to cover pre commit. And I was really skeptical. I had made bad experiences with these kinds of precommit hooks that run. So you make a Git commit and then it doesn't work because you had some wrong white space in it. And I thought, I want my commits to be really snappy. I just use Nox for that. But after hearing these comments, I thought, I'm going to give it a try. It's a new tool. Maybe it solves these problems much better and it really does. I would really recommend anybody to give precommit a try. Basically, you drop a YAML config in your project that defines the hooks that you want to run. So this might be a hook that formats your code using Black, or that Lint code using Flick8or so much more. There. There's an abundance of precommit hooks out there.

31:50 Oh, wow. Yeah. There's probably 20 pages in the list of pre commit hooks that are at the top. You click on supported hooks and Precommit.com.

31:59 Right. Precommit.

32:01 So it's a GitHub manager. But it's not just the GitHub manager. It's also a Linter framework and a multi language Litter framework. So you can have your hooks written in Ruby, C++ you name it, and it's very easy to use them in a Python project or basically any language project. It works using Git. So basically installs the tools from their Git repository and you can run them as part of your Git commits or all the other hook points that git offers. But you can also run it just NCI on your entire code base. And that's really what I love about it is that it has this fail early philosophy, so you really get very early feedback, but it also works as a gatekeeper for your default branch and make sure that all the commits that go into your main branch are well formed.

33:02 This is interesting. I did have creator of pre commit on the show quite a while ago. I talked about it, but I think maybe some of these are new. I hadn't really appreciated them before. One that's really cool here is check. Json as a pre commit and it checks JSON files for parcel syntax. So basically as part of your commit that says, well, I'm guessing here's a changed JSON file is this can it just basically be loaded with JSON loadfit file, right. And another one is yes, you're supposed to have unit tests, but you might not have unit tests for everything. So check AST just means like can Python parse the files that's sort of like the. Compile, right? In a sense, yes. Those are just like the first couple out of the 20 pages. So I need to come back to this and check Tomo with another sort of similar to the check. Json check. Yaml, check up. Xml and so on.

33:53 Yes, there's this repository called precommit hooks, and that has lots of very small hooks that are tremendously useful. And then you have larger tools that also offer integration with pre commit like Flake8, for example.

34:10 This portion of Talk Python to Me is brought to you by the Compiler Podcast from Red Hat. Just like you, I'm a big fan of Podcasts, and I'm happy to share a new one from a highly respected and open source company. Compiler an original podcast from Red Hat. With more and more of us working from home, it's important to keep our human connection with technology. With Compiler, you'll do just that. The Compiler Podcast unravels industry topics, trends, and things you've always wanted to know about tech through interviews with people who know it best. These conversations include answering big questions like what is technical debt? What are hiring managers actually looking for? And do you have to know how to code to get started in open source? I was a guest on Red Hat's previous podcast, Command Line Heroes, and Compiler follows along in that excellent and polished style we came to expect from that show. I just listened to episode twelve of Compiler. How Should We Handle Failure? I really valued their conversation about making space for developers to fail so that they can learn and grow without fear of making mistakes or taking down the production website. It's a conversation we can all relate to, I'm sure. Listen to an episode of Compiler by visiting Talkpython.fm/compiler. The link is in your Podcast Player show Notes. You can listen to Compiler on Apple Podcast, overcast Spotify Podcast, or anywhere you listen to your podcast. And yes, of course you could subscribe by just searching for it in your podcast player, but do so by following Talkpython.fm/compiler so that they know that you came from Talk Python to me. My thanks to the compiler podcast for keeping this podcast going strong on to continuous integration. I feel like GitHub Actions has really taken hold and the Python space is the way a lot of people are doing stuff there.

35:58 Absolutely. It feels like there was some kind of mass exodus from Traverse CI to GitHub Actions. It's so flexible and it goes way beyond just running tests and like the normal what you normally think of as CI. So you can automate a lot of your developer workflows center around the collaboration with others.

36:21 Nice. The type of stuff I work on doesn't super lend itself well to git hub Actions.

36:26 It probably does somewhat, but it's not something I use that much, but it seems like if you had a package that had maybe complicated builds or something like that, you could even use it to build your wheels and stuff like that, right?

36:38 Sure, yeah.

36:40 The way I like to use it is to have most of the logic in nox, because that means I can always just test it locally and see if everything debug it easily, and then I try to keep the GitHub Actions workflows pretty lightweight and just let them invoke nox. I usually have a matrix that contains the NOC sessions, so this might be the testing or running precommit, or to lend the code or to build the documentation. If that's all valid, then the matrix has the Python versions that I want to test on. If I'm working on a library, it's important to support not just the latest Python version, probably Python3.6 onwards, maybe even the upcoming Python version. And then obviously platform. So always, unless you were only working on one platform, try to have at least Linux and Windows and then maybe macOS as well.

37:40 Yeah, this is neat. People talk about, well, we can't really test this on Windows because I have a Windows machine, or vice versa can't test it on Mac.

37:47 So easy. Yeah.

37:49 So here's your three platforms right here. Right.

37:52 I'm not saying it's easy to deploy it if something goes wrong and you're only working on macOS or Linux, you probably at some point once you have a virtual machine running Windows if you don't have a physical Windows machine. But it's also not hard to get one these days too. But otherwise it's very easy to integrate Windows in your CI.

38:12 Yeah, very cool. And then the next one has to do with documentation here, right there. Sphinx MyST and read the documents and one of the themes. So pretty.

38:23 When I wrote the articles it was just Sphinx and read the docs. MyST hasn't happened yet.

38:31 Maybe let's start with Sphinx because I guess many people will already be familiar with it. It's a Python documentation generator. It's also used for Python's own documentation. So for example, the library docs, it's been around for a long time, and it normally uses a restructured text, which is a very expressive language to write technical documentation. Sometimes too expressive, sometimes too expressive. It's not the lightweight language that we know. Mark down to be and write down is really conquered the world. And when I wrote the documentation chapter, I think I linked to an article by Eric Holcher from reading the docs, comparing the two formats. And that was before MyST happened. And he said there's huge restructured text, it's just so much more expressive and it lets you have cross references and all of these things has the powerful directors. Anyway, now we have MyST allows you to do essentially the same thing in Markdown. There's an extension syntax and you can have directives, you can have cross references. And it's a lot of fun to write documentation in this. So this was a recent addition to the project template to support Markdown documentation.

39:50 And for those who don't know, also, I did have a show recently with the misfolks about Sphinx and MyST and so on. And there I learned that one of the things that's cool is you can inline restructured text. So if you get to a section you're like, this is just marked and want to do this, do a little tiny bit of restructured text instead of living in it.

40:07 You actually need to do that still for your generated API documentation. So Sphinx has an extension called auto Doc that will take all the strings in your code and transform that into API documentation. And that still doesn't have a replacement. What you do is you write your Doc strings using restructured text, maybe using Google style Doc. Strings or NumPy style. Doc. Strings, and then you use the auto. Doc directive to basically quote it in line in your Markdown documentation. So that's a little bit of restructured text there. I think working on filling this gap somewhere going on, and I'm really looking forward to that feature coming, because then everything will be just marked on.

40:54 Yeah, that'll be great for sure. I do want to give a quick shout out to Paul Everett's course that he wrote over at talk phython on Sphinx and this if you're interested, check out this free course that you put together for us over there. So that's worth checking out. Now, I'll put a link to show notes. Now, if you're building a Python package that goes on pipe, not everything people build with Python goes on PyPI, or even it should be structured in the shape of a package potentially. But a lot of them are right. A lot of libraries are. So you talk about automatic automating uploads to PyPI and test Pypi. Want to talk about the story there?

41:28 Yes. So the Python package index there actually has a sibling called Test PyPI, which is just a separate instance, it's very useful to upload your wheels and Estes to test Pi before you actually do a release because you can check them and see if everything is the way you expect it to be installed. The package end to end. And in CI, what I like to do is I have to switch where if it's an actual release, I upload to PyPI. But in all the other cases I just upload to test PyPI. So every commit that goes into the main branch is going to be built and uploaded to Test PyPI.

42:12 You can't change what you put into PyPI. You can add new versions that replace it, but you can't change a version.

42:18 Really the best you can do is yank the release. It's going to be there for those who have pinned the version, but otherwise it will be invisible to those who just want to get the latest release. But no placing.

42:32 Yeah. So it's a nice reminder and automation to set up to remember there's test pypi and automatically sort of have your project know and use it. All right, here's another one. When I talk about stuff I was learning, this one is definitely new to me and this is cool. Automate Release Notes with Release Drafter after this. Cool.

42:51 So Release Drafter takes the titles of your merge PRS and it creates a Draft release. So the release in this sense is the GitHub release. That is something that you can see on the right hand side of the GitHub repo page. You have this releases link and that basically gets you to either a list of tags or you can describe the changes. So it's essentially Release Notes on GitHub.

43:24 Surely there's some kind of weird inception where Release Drafter uses Release Drafter to build this change logs or something.

43:33 So Release Draft is really handy for that. Now actually you can git hub Releases have an auto generate button. So some of this functionality you will actually get even without using the Release Drafter action. So I think the Release Drafter action is somewhat more flexible. What it gives you and basically means you're going to have to add a GitHub actions workflow for it and the configuration file and you can provide a template for your Release Notes and some replacement marker or placeholder where all the PR titles go.

44:15 Thing is cool. And it even has a draft.

44:18 You can see the Draft Release Notes as well.

44:23 You can see it and then you have a button to publish the Draft release. And if you don't have a tag yet, that's also going to add a Git tag to your repository.

44:33 That's really nice.

44:34 Lightweight approach to the Release Notes question.

44:38 So I see that it has what's changed? But I also saw in the Release Drafter had like bug fixes and stuff. Is there a way to teach it? Like these PRS are related to bugs and these are additions and stuff like that?

44:51 Yes. You can do that using labels on your PR. So if it's a bug fix and you have a correct label, then you can put it under a separate section.

45:00 So basically just drive it with GitHub labels. Cool. All right. We got a lot to go through here, so maybe these next two are pretty quick here. Actually, next three here. You talked about this sort of article and the cookie cutter drifting quite a bit apart because so many things have changed. One thing that seems to be pretty stable is that since Black came out, people are all about just Black.

45:19 It hasn't seemed to have gone out of style at all. I think it's a pretty stable one. Right.

45:24 It really makes a huge difference for productivity. And I feel if I'm contributing to somebody else's project, it makes things so much easier to know that there's a consistent style. I don't need to be afraid to destroy somebody's well crafted handcrafted formatted code. I can just run Black, and it's also great. Really helps readability it's. Basically the style becomes invisible. So I love Black, and Black now has better status in January. So we now have official Black finally Waited.

45:57 Well known as hash.

46:00 There's no more debate.

46:02 Yeah, I'm definitely a big fan of Black.

46:05 Indeed. I also give a shout out to Predator as well. And then sort of related, you have import sorting with ISORT.

46:11 I've been using ISORT Python Imports for a long time, which is also a tool by Anthony Swordil, who wrote Precommit.

46:18 Yes, I wanted to give Anthony a shout out and I wasn't 100% sure that I had the name just right in my memory, so I didn't want to misattribute it. But yeah, he also did pre commit, which I had him on the show.

46:27 Right. So these days I actually like to use ISORT, which is what everybody uses since ISORT Five. It's become much nicer to use. It uses the AST more. It has no trouble figuring out what your third party dependencies are. And it has an option. It has these profiles to make it really easy to be compatible with Black style.

46:50 And what I like to do is I like to put each input on a single line, which is actually what Reorder Python Imports does as well. And it greatly reduces the chance of merge conflict.

47:03 Right, cool. Quick question. Stepping back to the release drafter, Michael in the audience says his biggest hurdle is for doing regular and good releases are change logs.

47:17 How does Release Drafter sort of fit into that? Basically, if you structure everything as a PR, it'll capture it.

47:24 Yes. So Release Drafter only drafts the GitHub release for you. If you want to have, say, a page in your Sphinx documentation, you're going to need to pull that out there's. Actually, Eviuachim wrote that's his GitHub handle.

47:42 We should put a link later, but there's a tool that will pull the GitHub release and insert it into your Sphinx documentation, so you can do that.

47:51 There's also Towncryer, which really should be mentioned, and Script, which was written by netbatchilder. Those are tools that allow you to add release notes to your PRS as Snippets or Fragments, and that scales very well. If you have an open source project with many contributors, that's cool. Still figuring out the best way to maybe integrate all of these in some way.

48:17 Sure. You can overdo it for sure. All right. I think just a quick shout out. Speaking of Nedbatchelder, we have Pi test coverage, Pi and Codecav all in there is neat. And then CLI interface with Click. That's an interesting one. A popular one as well.

48:33 Absolutely. So that's from the palace project. So the same family of projects like Flask.

48:39 There's also a nice wrapper for Click called Typer, written by the FastAPI author.

48:44 Yes. When I saw that using Click, I'm like, you're such a fan of types. Maybe typer is also relevant here.

48:50 I just had another look at Typer and I think I do like it. After all, I actually really like it. Initially I was like, well, it's actually not. So I kind of like how Click gives you these decorators and separate your options, help text from the actual function.

49:06 But it is true that typer really reduces duplication because you don't have to repeat the types of your options, they're just type imitations of your parameters. So that's really neat.

49:17 Nice. Continuing with the typing story, we've got two things here. A static type check in with MyPY. Suspect a lot of people who are really into typing know about this, like the CLI run against your code and it'll make sure everything hangs together. If this function is calling that function with that variable. And you said that variable is one of the things that makes sure that all is going to fit together. And if you get that working, then you might be open to having something like my sphinxfor optimizations and so on. Which is also interesting, but one that maybe people haven't heard about is runtime type checking with typeguard, right?

49:47 I think it's really one of the most undervalued projects out there in the typing space. Type Guard is so useful.

49:54 When I first heard about it, I was like, why would you want to run time type check your code? If you have a static type checker, then you need to hit all the code paths.

50:05 Static type checking is great because it can just basically deduce the type correctors.

50:12 Typeguard is really useful.

50:15 For example, if you're interfacing with third party libraries who may or may not have type annotations. If they do, great. But how much do you trust them, right?

50:25 Yeah, just because it says it nothing enforces where it said it returns an int, it could return none, it said optional end, but it didn't.

50:33 The code can absolutely have type correctness. As far as MyPYt is concerned. But that might be just because there are some anti types or it's just kind of loosely Typed because there's no way to be stricter about the actual types. And Type guard will check that. So the way I like to use Typeguard is as a Pytest plug in. So you're basically running your test suite, and if you have complete code coverage, that should give you a good chance to catch any type of interesting.

51:03 So you can turn it on while your tests are running and it will run time, check everything. But then in production, not turn it on.

51:10 Exactly. You basically specify it. So you install it next to Pytest and then you pass an option, I think it's Typed out packages and pass the name of your package. And then Type guard is going to wrap every function in your code and check the parameter types and the return time. So that's really useful. It's also a library. So if you want to explicitly check in production, you can also use Type guard for that.

51:38 Nice. Yeah. It's got the type check decorator, which probably you can just put that on stuff you want to make sure it's checked exactly. Okay. That's a good one to learn. Another one is you create your project, you start building it two years later, who knows? Some feature is added, some other language feature might be not the way to do things. So you talk about automated Python syntax Pyupgrades with upgrade.

52:02 It's another Anthony Satellite tool, so you can run it from Precommit.

52:08 It's going to essentially pass the AST of the abstract syntax tree of your code and look for things that have better ways to express them in newer versions of Python. So basically say you drop Python 3.6 and automatically you get unions like the PyPi.

52:31 Right. Like pipe none versus optional bracket, something like that, right? Yeah.

52:37 These kinds of things or giving you nice set comprehensions cost to the set built in both of these. And it's very nice if you are supporting multiple Python versions and you've been waiting to use this feature and now finally you can drop the last Python version that didn't support it and you get this for free.

52:58 I just yeah, that is interesting. I really thought about that. Some of these are not language features in the sense that people are thinking, oh, well, now I can use async and a weight, which would take like a real important, significant change. But it's just like, oh, now you're able to because of Pep 289 past a generator directly to the min function or Max or some rather than a list comprehension, which then gets processed. Right. That would just be more efficient across the board. So that just happens automatically, right? Cool. Yeah. Very neat one.

53:30 All right.

53:31 Next, security audits there's, Bandit.

53:34 Bandit is a tool that looks at your Python code and figures out if there are any things that may or may not give you a security vulnerability or any kind of security issue. It flags some things like just importing a subprocess and you can know QA it.

53:54 I still find that useful because it just gives me a moment to think about all the implications of spawning other processes from a Python code.

54:04 But it has a lot of checks in your Python code base, so it's very nice to use it to guard yourself against some.

54:13 Right.

54:14 Things like YAML load should be YAML safe load. And I bet there's something there about pickling.

54:21 Yeah, for sure.

54:23 Ottoman.

54:24 All right.

54:26 Yeah, exactly.

54:26 Safety is the other one that basically just checks your dependencies. So there's a curated database of security vulnerabilities, and it's going to see if you have any dependency in a version that was vulnerable.

54:39 Nice. And just stepping back just a moment, Nick Ma points out that hyper is amazing when it comes to the documentation and the help option. Pretty cool. Let's see. Going back to checking the documentation with ex dock test.

54:56 So X Doc. Test is essentially a rewrite of a standard library or utility. It's called Doc. Test. And what does it do? So suppose you have a Doc string with an example code that shows how you are supposed to use a library or a function, and you commonly write these with a Python interpreter prompt to give you like what does the user type import my package? And then underneath you have the output of whatever functions you called and so on. So Doc. Test runs these examples and sees if they produce the expected output. They don't throw raise exceptions and so on. Xdoctest is a rewrite that uses the AST more than regular expressions, which is nice. And it's also a bit more flexible, so it has a few nice features compared to doctors.

55:47 All right, cool. So if you have an example in your documentation, here's a way to automatically make sure it's all good. Also, another audience question, follow up here as he asks, what do you think about services like sync snake? Sorry, S-N-Y-K SN to check dependencies like for security. Right. So you depend on flask depends on it's dangerous. It's dangerous. Who knows? Theoretically could have some issue like that kind of check.

56:16 I'm thinking he's asking I haven't used Snyk yet, so I'm only familiar with safety.

56:21 Nice. Okay, well, getting close to the end then. I have a question about all these taken as a whole. All right. Generating API documentation with auto Doc and Napoleon. Right.

56:31 So we talked about this before. This takes the Doc strings and generates API documentation.

56:36 Right. And this is not a Restful API swagger open API. This is like my Python library's function documentation, right?

56:45 Exactly. Yeah. That's the reference for the functions in your package boxes. Right. And Napoleon is a tool that will add support for Doc strings that are written Google style, for example, or some other conventions for dock strings. So Google style is pretty lightweight.

57:05 Which I'm a fan of that as well.

57:07 Declaring your arguments and returns.

57:10 Okay, nice. Generate command line reference with thinks click. So I'm guessing if you're using Click might be relevant.

57:18 Right. So you already have all your options, help text and command descriptions in Click, so why not just use them to generate the documentation? And that's basically what Six Click does.

57:34 When you build your documentation and you use things like Autodoc or things click, you have to remember to install your own package, and then these tools can just import it and read all the documentation that you have in line in your code, including Click text.

57:50 Yeah, very cool. All right. You've done all the work, documented it, you've tested it, it's good to go. Now you're going to release it, put it on GitHub. And the last one is Manage project labels with GitHub labeler.

58:01 So the idea of labeler is that instead of going through the web interface and then typing in all the color codes and Hex and so on, you can just use a file that you put in your repository to manage all your labels for your PRS and for your issues. That's really helpful. And it makes it really easy if you're collaborating with other people.

58:26 Nice. Yeah, looks great. Okay, well, that's your cookie cutter. And in one fails move in one single CLI cookie cutter command, you get all of these. So when this runs, I should have just run it and played with it. But does it give you an option to say, don't install my py, or does it just kind of give it all to you? I know cookie cutter has a lot of conditional behaviors and stuff. What's the experience of using this to create a project?

58:52 I have personally resisted putting in too many options. It's kind of I try to find to kind of show one way that works and also keep it maintainable. So I don't have a lot of I don't want the combinatorial explosion. There's a little bit of options that it gives you. So you can choose a license, for example. And cookie cutter allows you to hide or show parts of your file tree depending on what the user chose.

59:19 Basically, it's not meant to be all the different packaging tools, all the different ways, like talks nox different CI's, that there was a conscious decision to basically say, okay, here's one way to do it, and I can really curate it and make sure that it all works fair.

59:36 Keep it opinionated and straightforward and whatnot. Right. I suppose if people really wanted to use tox instead of nox and they wanted to use typer instead of Click, they could fork the repo, create their own template. Right. And roll with that. Cool. All right. Well, very interesting. One more quick audience question here from Michael says can labeler as in GitHub labeler, export existing settings. It'd be great to unify labels across repos. Any idea?

01:00:02 Right. I think there is a separate tool to do that. So I don't think that GitHub label does it. But I remember that was actually a community contribution. And I remember that the contributor who added this feature first went and exported the existing configuration from whatever we had in the repository.

01:00:23 Okay. Yeah. Cool. Very nice. All right. Well, we are just tiny bit out of time here. So unfortunately, even though there's a bunch of other stuff I wanted to cover, I don't think we're going to be able to because we covered so many cool things. I did want to just give a quick shout out to your music. And in addition to being a lawyer and a software developer and open source person, you also do like compositional stuff. Right? So you've got on your website, you got a whole section. How many videos here? Like ten different music videos that you put together.

01:00:56 You want to just give a quick shout out to that?

01:00:58 Yeah. So I spent I think ten years working both as a software engineer and touring and recording musician, also as a Ranger. So I arranged some string quartets.

01:01:10 So I did arrangements for Naima Husseini, who's a German indie singer.

01:01:18 Definitely check her out. Jackie. I've been on tour with her really across all of Europe and wonderful Reggie inspired singer. And there are so many more.

01:01:30 Yeah.

01:01:32 I'm very for the musicians I've been able to play with.

01:01:36 Yeah, that's fantastic. I listened to a bunch of them. My favorite is Emma Alice Kurtis and Deutschen Theater. That's the one with Naomi Hussain.

01:01:46 That was really good one. They're all good. But that one was really excellent. And then Michael also thinks that we should have a whole podcast about your compositional tool. Do you use Python for any of this stuff or is it kind of a separate world?

01:01:58 Sadly, it's pretty much a separate world. I've been using Ableton a lot and we didn't really get into automating all of this with Python yet?

01:02:07 Not yet. Not yet. Awesome.

01:02:10 Does this kind of stuff?

01:02:13 Yeah. Cool.

01:02:13 It can be done.

01:02:14 Yeah. People should check out Nox. Are you familiar with Fox and that whole program composition, building up music with Python? Have you seen this?

01:02:23 I know.

01:02:24 Oh, my gosh.

01:02:26 Every time I search for it, I think it's Fox Python. I think that's what it is. Check out the videos. There's some neat live coding music with Fox and Python. Every time I just ran and I pick one of these videos, it's not really necessarily the best one, but there's some really neat ones of sort of like adding instruments in with Python. It's cool. People can check that out. Yeah. All right. Well, we are out of time really quickly. Final two questions. You're going to write some Python code. What editor are you using these days?

01:02:54 I use Emacs. I've been using Emacs for a long time and use it with VI bindings. Now, I like it right on.

01:03:01 And almost hesitant to ask you for a notable PyPi package because we covered so many. But maybe just what one stands out to you. Like you want to give a shout out to either one we covered.

01:03:10 Or I just name typeguard because it really deserves typeguard. Three is going to come out hopefully soon, bringing new features.

01:03:18 Cool. All right. Well, final call to action. People want to get started. This hypermodern project idea that you've created, what do they do?

01:03:26 So just go to the cookie cutter, hypermodern Python repo, check out the contributor guide and the code of conduct. And we love contributors.

01:03:37 How relevant is going back and reading the article? Is it drifted too far or is it enough to get some of the Zen or if you've listened to this or you're kind of good to go?

01:03:44 I think the article series is still fun to read. I think these days what I would recommend is that you don't just take the example code, maybe just generate a default project from the cookie cutter and then take a look at that as well, because some things have changed in the two years. But the article series kind of gives you the motivation for everything. And it's probably also more fun to read than the user guide for the project template, which is also there and very detailed, but yeah, fantastic.

01:04:16 All right. Well, thank you so much for being on the show and Congrats on the whole project.

01:04:19 Thank you very much.

01:04:21 You bet. Bye. Bye.

01:04:22 Bye.

01:04:23 This has been another episode of Talk Python to me. Thank you to our sponsors. Be sure to check out what they're offering. It really helps support the show.

01:04:31 Starting a business is hard. Microsoft for Startups Founders Hub provides all founders at any stage with free resources and connections to solve startup challenges. Apply for free. Today at Talkpython. Fm/foundershub, listen to an episode of Compiler, an original podcast from Red Hat compiler unravels industry topics, trends, and things you've always wanted to know about tech through interviews with the people who know it best. Subscribe today by following Talkpython. Fm/compiler. Want to level up your Python? We have one of the largest catalogs of Python video courses over at Talkpython. Our content ranges from true beginners to deeply advanced topics like memory and async. And best of all, there's not a subscription in site. Check it out for yourself at Training.talkpython. Fm Be sure to subscribe to the show, open your favorite podcast app and search for Python. We should be right at the top. You can also find the itunes feed at /itunes, the GooglePlay feed at /Play, and the Direct rss feed at rss on talkpython. Fm.

01:05:35 We're live streaming most of our recordings these days. If you want to be part of the show and have your comments featured on the air, be sure to subscribe to our YouTube channel at Talkpython FM/Youtube. This is your host, Michael Kennedy. Thanks so much for listening. I really appreciate it. Now get out there and write some Python code.