Monitor performance issues & errors in your code
00:00 Do you write tests for your code? You probably should. And most the time, pi test is the industry standard these days. But pi tests can be much more than what you just get from installing it as a tool. There are many amazing plugins that improve pi test in many aspects. That's why I've invited Brian nock into the show to tell us about his favorites. Listen in and your Python testing will be faster, stronger and more beautiful. This is talk Python to me, Episode 267, recorded June 4 2020.
00:42 Welcome to talk Python to me, a weekly podcast on Python, the language, the libraries, the ecosystem, and the personalities. This is your host, Michael Kennedy. Follow me on Twitter where I'm at m Kennedy. Keep up with the show and listen to past episodes at talk python.fm and follow the show on Twitter via at talk Python. This episode is brought to you by linode. And century please check out what they're offering during their segments. It really helps support the show. Brian, welcome back to talk Python to me.
01:08 Thanks. It's fun to be back.
01:09 Yeah, it's good to have you back. You've been on a lot of episodes. You've been on five 2017, Python, your review 2018. Python, you review the Python testing column as a thing. 30 amazing Python projects, which is one of the most popular episodes actually. And the book author panel discussion. So happy to have one more on the list to have you here.
01:32 Yeah, it's nice. I love your show. Listen to it all the time. Thanks.
01:35 Yeah. And it's good to catch up with you. It's been a while.
01:38 Yeah. I mean, we talk to each other weekly. bytes. Right. So of course, of course.
01:44 Yeah. So those of you listening who don't know about Python bytes. That's our other podcasts that Brian and I co host. We each have separate podcasts, right? Talk by thought and I do alone. You do testing code, which is really good, where people find testing code,
01:59 testing. code.com.
02:00 Perfect. But we also do Python bytes, which is like a newsletter for Python with analysis in audio farm. It's a lot of fun. Yeah, it is people should check that out as well. So I want to have you tell your story, because you probably sold it four times out of five there anyway. So let's, let's just sort of check in and see what you've been up to in the last year. So
02:22 it's been actually pretty exciting. We're, I think we've I've told the story before, but I'm, I work at Rohde and Schwarz. So I work at a test measurement company. I'm an embedded software developer, and a team lead and lead engineer, but I've been focusing for the last 10 to 15, half of my career has been on communication testing, most of its been cellular stuff, and we're switching over last year. So I've been focusing mostly on Wi Fi testing. So that's been a lot of fun.
02:51 A nice, like Wi Fi six, that kind of stuff.
02:54 Yes. One of the fun things lately is some of the work on 802 11 x, which is just starting to roll out around the world in different places. So
03:03 yeah, that's cool. I'm actually very excited about that. Yeah, it's neat.
03:06 Also, one of the reasons why I kind of focus on testing a lot is I, I am a proponent of developing software with testing hand in hand so that you develop tests and code at the same time.
03:17 Yeah, it's always a very error prone and, you know, unlikely to succeed to say, Now that you've written the code, let's go back and add all the tests to it.
03:26 Yeah, definitely. But people don't learn how to how to write tests when they're in. I mean, even college classes hardly ever talk about how to write good tests.
03:35 Yeah, there's definitely that sort of splintering there, like on one hand, the stuff you're building in college, it's probably not going to be long lived, right? It's an assignment or a short project, you just want to get it done, because you also got that English paper, and you didn't really even want to take English. But now you're in there, and you got to do it. So let's just get it done. And it's a very different thing to think a team of people have to live with this for a period of time. Yeah, definitely.
04:01 And with more and more reliant on, it's a little bit easier of a thing for me to try to tell people the importance because they're used to work. A lot of people are used to working with open source projects now and you would never pull an open source project off the shelf and use it. If it had zero tests around it Dasa
04:18 yet, a lot of times, you've got to have tests to accompany your contributions back if you're doing a PR, right.
04:24 Yeah, definitely. It's getting more and more attention. I think that's a good thing. What's the work from home story
04:29 for you these days? Are you do you got to go to the office? Like I mean, the world scrambled for everyone, but like, the only part of my world that hasn't changed is my work because I've already been, you know, like fully distributed and whatnot, I guess other than conferences, but how about you?
04:43 Yeah. So it was a hard like a hard shift to one. You know, I think it was early, early March was it really, really March there was a day where we just got told the grab your stuff and go home and we're working from home from now on for a while, at least so and we just moved in. To a beautiful office in July. So that's is really nice. Like it's a place you would want to go to because it's
05:05 got decor and whatnot. Yeah.
05:07 And these giant training rooms and stuff. It's a it's a good building. So working from home, the entire I had just hired a couple people for the team. And so training people remotely was I thought it was going to be a trick, but it's actually been, it's gone really smoothly. It's good. And yeah, so we're remote team. We are so in our county. So I'm in the county, Washington County that we're in, around Portland, we're supposedly entering phase one, we're doing this three stage phase, going back to work thing, but don't know, we haven't allowed people to go back yet. But hopefully, there will be. There's some people that can more effectively work from the office. So yeah, to try to get those people back as soon as we can. But do it safely? Do you
05:49 think you'll have more work from home remote options? Generally, at the company because of this?
05:55 Yeah, I think so. I think we've one of the fears was, then I think a lot of companies have seen this, the fear is that people working from home will, they'll be more distracted, they, you know, they won't be as productive. And I don't even know if this is a fair test. Because everybody's also it's not just working from home, but their kids are home also. And yeah, exactly. There's also the stress of just what's going on in the world is a stressful situation. So even though it's not a fair comparison, we're comparing anyway, and things are people things are going okay, we're still being productive, just about as productive or more. So then we were before. And in a lot of cases, the lack of a commute is really easy. The also the ease of discussion. I mean, you can I don't know, it just is I get more time focused. But also, I think I'm talking with my team more often. Now, because we make it a point to talk every day as a team. So right, right focus type stuff, not just I Oh, I passed them in the hall sort of thing. Yeah, that's cool. Now one more thing, I guess, while we're catching up on this stuff, is you've got these really beautiful training rooms with nice, giant screens. And one of the things that we were doing together is the Python, Portland West meetup which kind of got paused from all this as well, right? Yeah. And that's a big bummer. Because I think it was really serving a community in the west side of Portland that didn't have meetup. And we'd really like to, I have no idea when we're gonna get back to being able to do that. But I'd like to be able to do something.
07:22 Yeah, I don't even know when I go to a restaurant. So you know, things are all helping here.
07:27 is crazy. Yeah, definitely.
07:29 Yeah. All right, cool. Well, good to catch up. Now, let's talk some pi test, I'd like to do just a little bit of high level stuff to just like set the stage. And then we're going to dive into the plugins. So pi test is a external separate way to write and test Python code, just not say, part of Python, like unit tests. So maybe let's just do like a quick survey of the modern testing tools that people might potentially use. And then like, how we'd write tests and pi tests, for those who maybe have only done unit test type testing, or like testing. Maybe I should do that.
08:03 Yeah, sure. Well, actually, you had a few on the list that I deleted, but because they're no longer modern, right, there? Well, I would say not there still is unit test and unit test is being supported. And I do think and it's part of the it's part of the Python distribution. That's the one winning point, I think, right? It's like, I don't have to have a requirements txt or pip install anything,
08:26 potentially. And I can still write tests.
08:28 Yeah, one of the main reasons why it's there is to test Python, and test the standard library. And I think that that's a good reason for it. There's a bunch of people that have built on tools around around it. And I think that's fine. But it's just not my thing. One of the things while we're talking about other testing app things is to it's a good thing that note that nose is still available, but I don't recommend anybody use nose, or nose to or any of its variants, because I don't know, knows is not being supported at all at all anymore. And knows to has very little traction. And I don't know if it's I don't currently know if it's active. So if you're if you're currently working on those sorry about the disparaging remark, but maybe correct me if I'm wrong.
09:13 Yeah. Well, I mean, the thing is, you got all these choices, right? Like, if it's pi test, or some of these other things like nose or something, it's like, Well,
09:22 okay, when I first got into it, I really, I was looking for the easiest, and I thought that maybe the easiest and the most powerful would be two different things. But it turns out that pi test is both the easiest and the most powerful, which is nice. Nice.
09:36 There's some other ones, though, that you might consider.
09:38 Yeah, well, in conjunction with PI test and other testing things, there's a lot of other tools that people use tox is used. It is not just a testing tool. But one of the neat things about it is you can do a test matrix or running different Python environments. It creates a It tests a lot of your application. It's pretty almost an episode and it's right
09:58 you can test on like various versions like Python 353637, a stuff of that, right?
10:04 Yeah. And typically, if you're running like running pi test you pi test doesn't automatically build your application for you. But talks part of its process is, is creating the wheel of your life, you're doing a package, it'll create the package for you, and then install it. And it can does the whole whole roundtrip thing. So that's okay, a lot of people are moving to do continuous integration is definitely a thing that's important everywhere. And a couple years ago, it was probably Jenkins or Travis that people were using. But now I think more, a lot of people are moving to GitHub actions, because they're just right there with your code. So why not? Yeah, that's super easy, and it totally solves the problem. That is, I think one of the really big challenges of testing in teams is you get different levels of buy in from different participants in the team. And some people are very, they just are reluctant to just run the test before they'll do a check in. Yeah, they'll break the test, they'll do a check in and they don't even know they broke it, but they don't care that much. I mean, people care to different degrees, but it's like, well, they're kind of a nuisance, but I guess if you're gonna make me write test that well, right. And so the continuous integration is like, everyone gets checked. Before we get
11:19 your stuff. Right,
11:20 right. So when looking it up within, within a GitHub action, you could do this before, but it's really easy with these is to just say if you'd like for instance, a merge request doesn't even go doesn't even go very far if the test pet fails. So yeah, listing coverage pie, because I'm, if you care about coverage, that's how you do that in Python. I don't do a lot of web testing. But I wanted to point out that Selenium and, and a wrapper around selenium, Gods splinter are very commonly used. And then these are all packaged very well within a plugin called pi test splinters so that you can easily control web applications.
11:57 Nice. I haven't heard of that one. I've heard of selenium but not splinter.
11:59 Yeah, the property based testing is gaining a lot more traction in conjunction with functionality testing. And the solution within Python is a hypothesis for that.
12:10 So everything you've listed here is pi test plus, not instead of Right,
12:15 well, sure. It's pi test plus these other things. However, they I mean, none of them are pi test only. So yeah, you can use unit test with all of these things as well.
12:23 Yeah. All right. Yeah. testing framework plus, right. They're not necessarily alternatives.
12:27 Yeah. Great.
12:28 All right, cool. So those are definitely neat things to look into their separate projects. This portion of talk Python, to me is brought to you by linode. Whether you're working on a personal project, or managing your enterprises infrastructure, linode has the pricing support and scale that you need to take your project to the next level, with 11 data centers worldwide, including their newest data center in Sydney, Australia, enterprise grade hardware, s3 compatible storage, and the next generation network linode delivers the performance that you expect at a price that you don't get started on the note today with a $20 credit and you get access to native SSD storage, a 40 gigabit network industry leading processors, their revamped Cloud Manager cloud not linode.com root access to your server along with their newest API and a Python COI just visit talk python.fm slash linode. When creating a new linode account, you'll automatically get $20 credit for your next project. Oh, and one last thing they're hiring go to lynda.com slash careers to find out more, let them know that we sent you. Now, like I said, just for people who maybe haven't written a test in PI test, like what's the really quick flow? Like, why is it so simple
13:42 pi test has a really great discovery mechanism, but function that just starts with, like, if you have a test file that starts with test underscore, there's some naming convention. So the easiest one thing is to have a test file that has test underscore in it, and then start your functions, test functions with test underscore, and then it'll all just get run. So pi test will find all of the files that start with test underscore and run all of the functions within them that start with test underscore, and then that's it really, and so how does a test pass or fail? It fails if an exception is raised. And the easiest way to raise an exception is to if you're intentionally checking something is to use a cert. So if you like assert equals b or something, and if it's if the assertion fails, your test fails. Yeah, crazy.
14:28 Yeah. So you don't have to have decorators or derive from certain classes or anything like that. It's just the name involves test underscore, and you assert stuff.
14:37 That's pretty easy. You don't even have to import pi test in your test files. They can be just functions. So
14:43 yeah, that's right. So unless you're using some of the unique features that like extend pi tests, like you talked about the way a test fails, if there's an exception, well, you're testing for error handling, sometimes you want to trigger an exception and verify it has happened. So in that case, you have to include pi test stuff to do like the way phrases sort of stuff on it. But in general, for like the straight ahead tests, you don't even need to import pi tests, which is kind of weird. That freaked me out a little bit at first, I'm like, why am I not interacting with PI tests? This is so weird.
15:12 No code level? No, it's it's beautiful. It's one of the things I like is that it's just easy to start. Absolutely.
15:18 Now, one other thing I want to cover before we dig into the list of plugins that we're gonna cover is, there's some non obvious other ways to extend pi test, right? Like it has this concept of fixtures that allow you to create and initialize stuff and then automatically pi test as a way to like, pass it along to your tests, and whatnot, you'll see some of these plugins actually use fixtures, like some stuff about modifying time and whatnot that we'll talk about. What's the deal with those?
15:48 Well, so the unit test model, and the unit test is is similar to a lot of the x unit, yeah, type, J unit and PHP unit and stuff, have a similar model, where you just things are just named also in the test get run. But there's often times where you need stuff to happen before your test gets run, like setting up a resource or generating data or something like that. And there's other setup and teardown functions that you can that are associated with these other test frameworks. Pi test actually also supports those sorts of setup and teardown functions, if you want to use them. They are supported. But it's really not recommended because fixtures are so awesome. So a fixture is a way to combine, like to write a function that is handling a resource or handling some data and be able to do the setup and teardown within one function. And, if necessary, and then also just be able to scope those and you can you can have them have the resource or data shared between one test or many tests or even across the entire project. And you can control you know, like if you're initializing a database, or connecting to a web resource, you might not want to do that for every test, you can do it for one once per session. And then and then do things like roll backs and stuff per test. And that granularity of being able to shift your the granularity of your setup and teardown is super powerful and really helped makes pi test shine.
17:16 Yeah. And the way that you write it is super cool as well, right? Just have a test function that just takes a parameter, the parameter is named the same as the fixture and pi test just says, okay, really call this function and pass it over. It could do really cool things like you could pass off your database data access layer, and it could start a transaction before hand it to you and then rollback the transaction after the test is over. Right. And you don't have to worry about it. It could just take care of that for you.
17:41 Yeah, yep. And the fixtures are really what brought me to pi test some of the, there's some massively cool things you can do with that. Like, one of the things that people don't think about is, if you've got a system under test that has error logs somewhere, one of the things you can do within a fixture is check your logs after the test runs to make sure there's no failures, they got saved somewhere. So yeah, very cool.
18:04 Love it. Okay, so that's one way. The other one is this plugins in general, right? Like, there's ways to extend pi test with plugins as a concept,
18:13 the PI test has a whole set of hook functions as well there's, there's a lot of things you can do, you can, you can write fixtures and package those fixtures for other people to use. So some plugins are just packaged fixtures. Other things you can do with Hook function. So with, I actually was intimidated by first reading about hook functions, but they're just, they're just a way for pi test to allow different parts of the way py test runs, to be hooked into. So you can you know, alter a few things if you need to, like, like some of the test ordering ones. Once all the tests are collected, you can hook into that section and reorder them if you need to, or or filter them to see which ones execute
18:53 right.
18:54 Yeah, there's a whole bunch of ways you can hook into it. And it's fairly complex, but a lot of people have come up with some really clever things. And plugins allow you to lie to do that. And I mean, not just third party plugins that you can pull off of pie, but you can also just, if you want to share some code that you have that works with your tests, amongst different projects, you can package your own plugins to do that. Yeah.
19:19 Oh, that's cool. You could even have them for like, your company, right? like we always do this. In our CI and whatnot, you can just grab a plug in and do stuff.
19:28 Yeah, like I've got plugins that I've written that just are particular to one particular test piece of test equipment. So to control this piece of equipment, you can use this plugin and it helps you some of the common things.
19:39 Oh, that's cool. makes it a lot easier for new people. Yeah, lots easier. Nice. All right. I think it's time when we go down the list or set the stage. We know what fixtures are. We know what plugins are, at least in general. Let's put some sugar on pi test to make it sweet.
19:56 Yeah, so we're gonna run through a whole bunch of really cool pi test plugins. Right. Yeah,
20:00 absolutely. All right, what's this first one here.
20:02 So sugar is a fun one, actually, I ran into this right away with looking at pi test. So the default output of pi test is you, when you run it, you get a dot shows up for every for every test that gets run. And it does list the files but little dot shows up. And maybe that's not enough for you. So piteous sugars shows you like green checkmarks instead, so and then, you know, instead of like, apps for failure, that I think there's like red X's or something that show up. So it alters the output of pi tests so that it looks a little nicer.
20:38 Yeah, it also looks like it might be suppressing the module name, the folder names, and only like highlighting the actual file names. So it does like gray out a little part of the UI. So like the actual test, Bobby run stands out. And there's a vertical progress bar going along that that runs as well. That's pretty sweet.
20:57 Yeah, that's pretty cool. And that was especially really helpful before pi test added that percentage done. So now for each file, pi test list, the percent done Now by default, but it didn't always have that in pi tischer. And that is pretty cool.
21:11 Yeah, that's cool. And there's a nice little animated GIF as all things should have on the there that will all link to in the show notes. So people can check it out and see it going. So yeah, pi test sugar. And this is cool, because all you got to do is install this and it doesn't require effort, right? Like, oh, and now I could do this thing in code. It's just like, I install this. And it looks better.
21:31 Yeah. How do you install plugins? Just do pip install them? Like, do I need to register it with PI test? Or is it sufficient to just pip install it just pip install it? It's about the design of the plugin as well. So some plugins like sugar, you just install it and it starts starts working. You don't have to do anything. All right, but some of them intentionally don't start right away. So some plugins, they are installed, but you have to turn on the functionality with a flag or something like that.
21:55 Yeah. Okay. Cool. All right. What's the next one? Number two.
21:59 So this one's definitely something I test Cove dash Cove, it's the way to run coverage with PI test together. So that I mean, there's really two ways to run coverage on pi test tests. Coverage doesn't require pi test, right? So you can, you can use any test framework. So you can run pi test from coverage, or you can run coverage from pi test. It's kind of a choice. But I usually use PI test Cove because it's just really well supported. It does some really cool things. One of the things that does really nice is we'll talk about access later. But access allows parallel runs and pi test Cove cleanly collects all of the the different parallel runs in pulls the report coverage report together for you. Oh, that's cool, which is really nice.
22:46 Yeah, it has x dist support, which we'll talk about what that means. But that's a pretty high end piece of kit there to go grab a parallel executing stuff and bring it all together.
22:57 Yeah. And there's also it's really good about catching the coverage from right from the beginning. So you've got like some startup code that happens like your your fixtures and stuff that you've set up that run before your test, making sure that those get have coverage turned on, like as soon as possible so that you don't have like startup code that doesn't get run in coverage. That's one of the one of the issues with hooking up coverage, often, but bitez Cove handles that really cleanly.
23:23 Yeah, awesome. It also has sub process support. So you can fork things into a sub process, and it'll get covered automatically. Which you might think, okay, who's gonna do that? And like, do they really care about tracking the coverage of it? But you know, multi processing? Right? You're doing parallelism?
23:40 Yeah, definitely.
23:41 Yeah, that's good as well. Very nice.
23:44 The other nice thing about it is just so if your makes it easy to have a text output, so normally, when you run coverage, outside of like, if you run pi test from coverage, once you're finished, you have to then tell coverage to give you the reports, you have to run coverage report to get the coverage report, but by desktop just prints that at the end. So you get that automatically don't have to run it again. Cool.
24:08 Very nice. Very nice. Would you say it's low stress?
24:11 It is low stress? Yes. Unless the stress is trained to get 100% coverage?
24:17 Well, that's true. I actually, what are your thoughts on? I'll share my thoughts with you as well. But you know, a lot of people feel like they're not testing Well, if they don't have 100% coverage, or they don't have any coverage. So they don't even know, like how much of the code is being run by Tess, what are your thoughts?
24:32 I am a strong proponent for open source projects to have 100% coverage, because it eases new people, new developers in the project. When they add code, it's easy for them to find the code that they added that isn't being covered yet. It doesn't necessarily mean that your tests are good. It just means that there Yeah, because it's covered. For open source stuff. It's good in and then if you take that argument, then it's good for internal projects as well. Because it's easy. To run stuff, however, I'm not a zealot about it, there's I mean, if you're like jumping into a legacy project and adding software tests to a project, and you're testing this the new stuff, and the things that you change, you're never going to hit on our percent coverage of like the rest of the legacy code. And, you know,
25:18 Yeah, I agree with that. I feel like a lot of people get hung up on like, the last mile for applications, like for open source projects, where it's like a library or something. Sure, it should totally all be there. But there's like, in certain apps, there's just little edge cases, like, if I'm running on Windows, I got to do this other thing. But I'm only testing this on macOS or on Linux, right? So like, how much did I stress about trying to like, get that little edge piece of code to run? Or like there's this, this one utility that only I run and a cron job to do this thing? I don't know. There's just all these little edge cases, that can be like a whole lot of work to get to. But I feel like, you know, most the time, you should probably not stress too much about it.
26:01 Yeah. So all those edge cases, the reasons why I think you should test those. Because, okay, so we have ways to combine coverage reports. So you can use something like GitHub actions, or Travis or something to run your run your code on multiple platforms, and combine the coverage from the test runs from multiple platforms. So why not run them on multiple platforms? If that if you have platform specific code? If you have platform specific code? I think there's a reason is because some of your customers use it. And if it's different than the what the developers use. I think you should test that code.
26:38 Yeah, that's probably true. That's a good point. That's the only place it gets run before it gets deployed. All right. Well, my transition to the stress one, we got kind of sidetracked. But this next one is all about stress.
26:50 Yeah. So I think we should actually just cover a couple of them. So the next
26:55 pair, yeah,
26:56 yeah. And piteous repeat. So I just stress is a nice thing. So let's say you've got some code that you just wanted, like overnight, I know that I'm going to leave at like midnight, I want to, I know everybody's going to be out of the building by midnight. And then I want to just beat on the code and run it as many times as I can, for eight hours. So pi test dress allows you to be able to run set a time to say, hey, run this for eight hours, or run over half an hour, run it for certain time period, then that's really cool. If that's where you're thinking, and then repeat. I just repeat allows you to run your test suite or test functions or whatever, a certain number of times. So I want to set it up for 1000 times and bring these up together, because I found pi tests repeat first when I needed to. I had some tests that a test that was failing once in a while. And I just wanted to run it all night long. And so I just set it for a large number like 2000. I don't know. I like it, just guess. Yeah. But really, what I wanted it to do was run all night. And so I just stress would have been better for that case. So Oh, yeah, that's a cool one.
28:01 You know, where I would see these as being useful. Sometimes I'm trying to do profiling, like CPU profiling to figure out or maybe even memory profiling, and the startup of getting the, you know, pi test or whatever it is up and running and all the stuff in place and then running a little bit of code, and then tearing it all down. It's like gets dwarfed. You know, the the actual bit you're trying to understand gets dwarfed by all that startup. But here, you could just let it run for 10 minutes, and then it'll be really focused on what it was slower,
28:29 then that's where actually with profiling, I think that would be a good place to use the the repeat. So you could say, do a nice round number, like 100 times or 1000 times? Because it's easy to divide by that.
28:40 Yeah, yeah, that's a good point. Exactly. Nice. What's this next one, it's the fail
28:45 Soviet a whole bunch of like print statements, and whatever in your code. It doesn't get shown normally with PI test. So pi test will capture the output. By default, I mean, you can turn it off. But by default, it captures the output, and any errors and failures and stuff, those get captured also the like trace backs and stuff. And then afterwards, so after, after the test is done, you get your report, and then you get you get a lot of this output that you can help debug your test failures with. With long running tests. It's sometimes it'd be really nice if you really could see those errors. While it has like while it's happening. So you can start looking at the code start debugging right away. And that's where instant fail will report your failures and errors while the test run is happening. Even with output being captured.
29:34 Nice. Is there a way with plugins where a five installed is to fail, but I only want to see it some of the time to like in code, say, you know, don't run this to fail right now. But sometimes when I want it, let it run without pip on installing it.
29:48 I'm not sure. Actually, that would be sweet.
29:51 Yeah, I don't know. It could be up to them individually, but it would be nice if maybe a command line or something. I don't know some way to say Actually, I want this One turned on now but not always.
30:01 Well, I'm just looking at it. So it looks like even if it has it installed, it doesn't this is one of those that it doesn't run automatically. You have to pass on the flag. I see. Okay, cool. Cool. Turn it on.
30:10 Nice. So speaking of passing flags and whatnot, pi test metadata is the next one. Number six.
30:15 Yeah, this is a kind of an obscure one. And this is often used with it with PI test HTML, which actually, we're not gonna, we're not covering, maybe we should. But pi test insert fail or not, is to fail. Money, this metadata allows you to either through it gives you a couple things, it has a command line flag, where you can pass in extra information to go into your test report. And then also you can within like, so like, for instance, if you were running against a particular set of servers, you could, you could have the server name getting listed in the report. The other thing is, maybe your maybe the data that you really want to log the extra metadata around a test run is generated, or like in real time as developed or gotten from within a fixture or something. So a fixture or a test can set this extra metadata and it gets output with your test data. So
31:09 how cool like maybe what GitHub branch you're running on? Or what environment like staging or production? testing, yeah, whatever. Yeah.
31:18 Yeah, go may or may be adding you know, what operating system you mean, your operating systems already being logged. But if there's other information, like but yeah, what branch? What version? What? Some of the extra specifics? Yeah,
31:31 that's cool. Yeah, that sounds useful. Like, if you're saving the report, you're gonna look at it later, like maybe continuous integration or something like that.
31:40 Yeah. So like, for instance, with when I'm testing against electronic test equipment, we add the instrument that we're testing against. So that's okay. Part of the part of the log.
31:53 This portion of talk Python, to me is brought to you by century, how would you like to remove a little stress from your life? Do you worry that users may be having difficulties or are encountering errors with your app right now? Would you even know it until they send that support email? How much better would it be to have the error details immediately sent to you, including the call stack and values of local variables, as well as the active user stored in the report? With century This is not only possible, it's simple and free. In fact, we use century on all the talk Python web properties, we've actually fixed a bug triggered by our user and had the upgrade ready to roll out as we got the support email, that was a great email to write back, we saw your error and have already rolled out the fix. Imagine their surprise, surprise and delight your users today, create your free account at talk Python dot f m slash century and track up to 5000 errors a month across multiple projects for free. If you use the code talk Python, oh, and word. It's good for two free months of centuries team plan, which will give you up to 20 times as many monthly events in some other features. So create that free account today. Yeah, I don't want to just be random,
33:04 not just random. Yeah. But sometimes you do. So one of the things that pi test will actually I think I've intentionally forgotten the test order, I think it's alphabetical or just just not alphabetical. It's like the order that the tests are in the file. That's how the default order that things get running. But you really don't want your tests to be order dependent. So you want to be able to, to run any test in any order. And sometimes, a good way to test that is with the PI test randomly plugin. So it does a couple of things. One of the things you can do is to can read it can reorder your your tests, so that they're running in a random order. But also you might be using random as a seed for some randomized information. Like, say, like Faker, for instance, and other fake data generators use random seeds. So pi test randomly can help you randomize that stuff so that it's more random.
33:59 Yeah, that's cool. It says, If Faker, which is a really cool library, if fakers installed it, we'll see it'll automatically updated. So that Faker is different this time than other time. Same thing for NumPy. Yeah, this is really cool. And anytime you have your test running reliably in the same order, you could unintentionally pick up some form of dependency on order, like if you call this test it like initializes, the database connection string, and then you can call this other thing gonna work. But if you call it in the other order, it would crash me maybe that's a bad example, because you really want to avoid that dependency. But you know what I mean, like it could set something where the next thing depends on it being set that way, and if you randomly order it, then you're going to discover this.
34:42 Yeah, especially with with your scoping your fixtures to try to speed up test time, you might inadvertently have dependencies, like system dependencies. For instance, you get the system in a particular state in one test, and then you're running another test and exam problems often ism you Got a test that the test that runs fine when you're running it from your editor. And then you go into on the CI system where it's running everything, it fails, and you can consistently fail it with the entire suite, but not by itself. Those are just a pain in the rear to find out where the problem really is. And I mean randomly can help you. I'm not sure how well it can help you debug that problem. But it can help find dependencies. Yeah,
35:25 at least it won't leave it covered up hidden. Yeah, yeah. Well, you already mentioned x just earlier when we talked about that cover pi test Cove. So what's this exsist?
35:38 Well, Xs is super powerful, it's a little bit of a grab bag of a few features. One of the things that it does, and people usually grab it for this is to run tests in parallel. So you can tell it how many CPUs to run on, and it'll just run a whole bunch of tests in parallel. And that's pretty cool. It can speed up tests, it does have a little bit of startup time, because it's, it's communicating with the different, you know, it's doing multi process communications and a little bit of startup time. So if you have independent tests that don't share a resource, this can be really cool. The other thing that you can also just offload it onto different processes or different computers you can have, have it be running on different machines as well. So that's, that's cool. That's really cool. Yeah, that's
36:21 very fascinating.
36:22 And then I don't know why this is bundled into it. But it is a really cool feature of bitez desk x, this is the loop on fail feature. So you can pass it loop on fail, and it'll run your tests repeatedly in a sub process. And then you get to see what the failures are. But if it detects you've changed your project at all, it'll rerun the test, but it just reruns the failing tests. So it just keeps rerunning all of your failures while you're trying to fix it. And then, once all of them are, when everything's running and fixed, it'll rerun everything. So when they'll pass, it'll rerun the whole suite.
37:00 Yeah, super cool. So it's a little bit like and pi charm, or you can say run the test when anything changes. But this only does it for the failure. So you're like you got a, you write a failing test, or you have a failing test you're trying to fix and you can just work on it. And it only focuses on that thing. Yeah, I mean,
37:16 it kind of is all the passing lends to because if your suite is passing, it just keeps where you're running it every time it sees a change. I
37:22 see. Okay, yeah, cool. All right, well, one way your code can fail, is it doesn't execute properly makes a mistake. The other one is it could just not be formatted the way you like, or break the some of the idioms that you're supposed to write. Yeah,
37:38 there's a whole bunch of extra plugins that are like this, but their style guide stuff. So pi test flakes is one where it can run flake, flake eight and other things against your code. As part of the PI test run, it can test both your source code and your test code if you wanted to actually think this is a pretty cool idea. But there is some controversy as to whether whether this should be really part of your PI test run or should it should be part of like, say do a pre commit or something. But I think having in both places is fine. If your workflow is mostly people run pi test to make sure all their codes working. And then they're ready to check it in. If you want to make sure you don't it's just your workflow, whether you like like to catch your flake errors during commit time or during a test time. Yeah. And
38:26 I just noticed the one that I grabbed a throw into the less there was I test lakes, I think and they say you should use PI test ash flake eight. So I'm going to link to that one. But Same, same idea. Yeah. Yeah, there's also a PI test. have eight I think it is. And there's like various different ones for this.
38:45 Yeah. And I gotta admit, I'm like a little, I guess over the top, I use both pi to suffocate and hook up like a to pre commit hook. So yeah, just because I mean, pre commit so fast. Why not?
38:57 Yeah, absolutely. Cool. Cool. Speaking of being fast, you know, your tests sometimes are slow, because it's a lot of work to run your tests, or they could just be stuck waiting on something.
39:08 Yeah, I really like to pi test timeout is a plugin, that is a fairly big hammer to a problem. But you've got tests that maybe it loops forever, you forgot to put a timer on something or it's stuck on a resource or for any reason it's hung. And I suggest people use PI test timeout for any long running sweets all the time. There is a for really short, fast tests, there is a little bit of overhead. But for any sort of long running stuff, the the overhead is is negligible, and you don't see it. But what it does is it runs runs your tests in a in another thread, and then is able to kill it. So it will just put a timer around it and kill the thread if it doesn't come back in time. The why I call it a big hammer is because if it fails, you don't get your test output, but it also doesn't hang forever. And these are especially good for ci systems because you really don't want to come back in the morning and notice that your test suite has been running for 12 hours. And it's only really supposed to run for 20 minutes.
40:11 Yeah, absolutely, yeah. If it's unattended, or like it's really long running, or you're talking to something that who knows how long it's gonna take to respond? Yeah, maybe even have that with instruments that you're testing and whatnot. Yeah,
40:24 we definitely use it. And the The nice thing is, you can have a global timeout to say for no test should run longer than this long. So whatever, 10 minutes, whatever is in your system is a reasonable amount of time. And then you can override that within we can put decorators on individual tests. So to say, well, this test is really fast, and really should only be this long, and whatever, you can fine tune it if you need to. Nice.
40:47 Alright, another one. I think I threw this one in the list is spec.
40:51 Yeah, I don't know anything about this.
40:53 Yeah. So spec, it's a little bit like sugar, it allows you to sort of control the output. So you can group tests by classes or by files, the failed past and skipped ones are marked and colored. You can remove like test underscore and underscore stuff from the test. So it'll like you might have a name, or it's like, test. User can log in with underscores. I think it'll take those out. So it looks more like English, like, user can log in or something like that. So it's just a nice little one for customizing. There's some options you can pass to it and whatnot. Cool. Yeah. I've heard you talk about pi test picked before.
41:31 Yeah, that's a fun plugin. So do you remember what this does?
41:36 Yes. Yeah. So this is the one. And we weren't totally sure, like how smart it was, we had a bit of a conversation about but this one will run tests that are not checked into GitHub, or have changes that are not checked into GitHub. Oh, yeah. So it'll run tests related to unstaged files. And if it sees some that are, you know, there, it'll basically if you're in a GitHub repo, you can run tests that are modified, but not checked in
42:06 not committed. Yeah. And that's usually often what you want to be doing. Right,
42:09 exactly what I would really like it to do, and I don't think it does. And people can add a comment at the bottom of the Show page, if that's changed, or still misunderstood, but I would love it to somehow use coverage to see what are the files that need to be retested by what tests touch, like I've got a, a non test file, I change what tests need to be rerun. I think it only just reruns the tests that are changed. I don't think it uses coverage. But I could be wrong about that. But it nonetheless, it's still pretty cool.
42:43 Yeah. And I mean, all of the pieces are together for somebody to do something like this, because coverage can has a context feature now. So you can tell it. And with PI test cover, you can automatically sets it up so that you can tell which tests touched which parts of your code. And the idea seems like great. One of the issues with that is there's a whole bunch of your code that gets hit by every test. Yeah. Because it's like startup code or something like that.
43:10 Right. Right. Well, you just don't change the startup code. You can't edit that part. Yeah. Now, but it's still it is a really cool idea to say like, just run the stuff that might have saw that's might have had some influence on that's really cool.
43:24 Yeah, some patches, pick doesn't do that. But it, it has this, it does a really cool, it's a nice addition to be able to run the test that you're working on. Yeah, I could totally see just running this and then using like a pre commit hook to run everything or just,
43:39 you know, right before you check in and just run everything or even just check it in. And it runs in ci and doesn't get merged if it fails, right. Like if there's some other gated mechanism further down, then this seems pretty cool. Yeah, definitely. Yeah. So this next one freeze gun is all about time. And this one, I feel like it's probably packaged as a fixture. least it has a fixture concept.
43:59 Definitely, as a fixture, it doesn't run by default, but you have to kind of turn it on when you know it.
44:04 This one is fun. People know when they hear what it does, you don't want this to run by default.
44:09 I like to list this one but just that's such a great name for he's going to just, it's for changing time. So you can easily there's fixtures around it so that you can easily either freeze time so that your anytime you grab the time of day or the daytime stuff, it's always going to be the same. Or you can you know, fast forward or change the date or do things like that. So one of the hard things is around testing code that deals with time because you have to know what the answer is going to be. And that can be different every time. So freeze gonna allows you to test time related tests really easily. It's just such a great name.
44:44 It is any of these functions, they can take a freezer object and then basically that what is that probably behind the scenes patch, like the time functions, date time now and whatnot. At that you can also do like freezer, move to some other time and then like instantly, you are transformed either into the future or the past.
45:02 Yeah, I think some people might be using mock for for that sort of thing in the past, but this is cleaner. Use this look, if you could say let's mock out daytime daytime now. Or you could use freeze gun like which do you want to do? Come on? Exactly, yeah.
45:17 All right, we got one more that's on your list and then an extra and I threw in at the end pi test check.
45:21 Yeah, I wrote a plugin called pi test check and solve the problem my head. So the problem is, the normal way to stop x, or to fail a test is to use a cert and or raise an exception, both of those stop the execution, I wanted to be able to do lots of things like if I want to test lots of attributes of some object, I want to be able to have lots of checks in there. And I want to see all of the failures, not just the first one. So pi test check solves this by allowing multiple failures per test with these little check functions. But you can also It also provides a context manager. So you can say within this context, you know, you can use normal asserts, but within a context that will continue after that context. So a similar solution is available called sub tests and with the PI test sub tests plugin, but I still like pi test check better because it works with stop on fail index fail, whereas sub tests do not
46:18 okay. Yeah, this is really cool. So you can pass in a check fixture to your test. And you can say check, greater AB less than or less than or equal to this is this is not in that and then it'll tell you like it was not greater, and it's not in there. As the air is something like that rain
46:36 is a fairly simple thing. It's a fixture that just collects failure, it like wraps, wraps things around within a certain context manager, or a try except block. And if a failure happens, it just instead of stopping the test, it records it into a list. And then after the test is complete, it'll like kind of hack with the PI test guts to make it fail.
46:57 Okay, that's really cool. I definitely like this. I'm thinking like, if I get just the text printout of the build failed, or the CI failed, even you know that I just see it. And when I'm running and like pi charm or something, I'd like to just look at and go, Oh, it failed. Because if this is true, and that's true, and this, like I see, right, you don't have to go and actually debug through it or be it might be that there's enough information here to actually know what's going on and just fix it. Yep.
47:24 And also, just more information is good sometimes dealing with I mean, I definitely use it a lot with us test equipment with signal processing and stuff. A signal might be wrong, but why is the signal wrong? It might be the wrong frequency. It might be the wrong bandwidth, it might be mean lots of different things. And if I'm checking four or five attributes of the signal, I really want all that information for a failure.
47:46 Yeah, I agree. There might be different aspects that all come together to make it successful. So report all the aspects. Yeah, that's cool. All right, I wouldn't throw one more in that's not really part of pi test, per se, but absolutely can be used with PI test because hey, it raises exceptions. And that's called fluent check. So this one allows you to use a well a fluent API, you know, you say, operation, one dot operation two dot operation three, to have a more English language thing. And to me, it feels a little bit like your check as well. But it's not a PI test plugin or fixture. So it has a couple of API's, I like the is API. So you can say like, is in dot, not none dot float dot between zero and one. And it'll actually check all those things and report like nice messages. Like, if it's not a float, if it's a you know, whatever it is, here's its value is not a float, and you expected it to be right. So a little more readable. Some people like it, some people just want to assert stuff, and either ones fine. But if you like the style, this is a pretty cool library. It's not mine, but I did to some prs. I did basically create the is API for it.
48:54 Cool. Does that have not a hard rock?
48:56 It should also have is a teapot or something like that from the for web testing, you know, the four for 18? Yeah, we should throw something else in there. Like why
49:05 not? We do it.
49:06 Anyway, those are kind of fun. For just like inside your tests expressing what you're testing a little bit better. I think check is really cool as well. And it's a bit similar. No, yeah, but this one doesn't stop. It just throws the first one there. All right. Well, that's a great a great list, or maybe just really quick, we could check in how's your book doing? You wrote a book about some of the stuff?
49:26 Yeah, so I wrote Python testing with PI test. That was, gosh, it was a it's been a couple years now that it's been out. And it's still going going strong. The, of course, the big chunk of sales right after the book was released, and quite a few during beta two and that was neat. But it's still selling really well and I'm really happy with it. It is still valid even though it's it's a couple years old now. And I wrote it in conjunction with talking with a lot of the PI test core maintainers to make sure I wasn't putting any anything in there. That would be deprecated soon.
50:03 It's still valid. It's good. Awesome. Yeah,
50:05 people can pick that up. They want to dive into this much more. And then also you did an episode that also talks about pi test plugins over on testing code. Yeah, it
50:14 was Episode 104. With the Anthony sottile, the episode that you and I just did, we kind of picked our own some of our favorites. Some of the things that we think people should know about what we did with episode one before we took download counts, man took the top some of the top 28 downloads I see
50:30 popular by usage not by curation.
50:33 Yeah, yep. Exactly.
50:35 Awesome. I Oh, this is really fun writing. But before we get out of here, two final questions. As per usual, I forgot to write some Python code. What editor do you use? I term? All right. Right on. It's got some sweet built in test. rhenish.
50:46 Yeah, it's really clean with testing.
50:47 Yep. And then notable pi package. I mean, I guess many of these are, anything in particular comes to mind you want to throw out there
50:55 is something different. I'm kind of a huge fan of flit from. So that's the easiest way to package right now, I think. Right, as opposed to
51:06 some of the other alternatives like pips, and D instead of tools and all those things, right. Yeah, yeah. So
51:15 it was just poetry and
51:17 yeah, poetry as well. That's right. All right. Well, final call to action. People are excited about pi test and pi test plugins. What should they do?
51:24 They should buy my book, actually. So one of the things I made sure to do is to to tell people how to create their own plugins within the book. So I think it's a really good introduction to that. Also, I guess, final call to action. Go subscribe to testing code and podcast also and test your code, go out and test something.
51:44 Do you guys talk about pi tests over there? Sometimes,
51:46 sometimes, yeah. A lot of times we do.
51:49 Okay, that's it. That's great. Also. Cool. Well, thanks for being on the show. Nice to have you back. As always, thank you. You bet. Bye. Bye. This has been another episode of talk Python to me. Our guest on this episode was Brian knockin, and it's been brought to you by linode, and century. Start your next Python project on the nodes state of the art cloud service. Just visit talk python.fm slash linode, Li n o d, you'll automatically get a $20 credit when you create a new account. Take some stress out of your life get notified immediately about errors in your web applications with century. Just visit talk python.fm slash century and get started for free. Want to level up your Python. If you're just getting started, try my Python jumpstart by building 10 apps course. Or if you're looking for something more advanced, check out our new async course the digs into all the different types of async programming you can do in Python. And of course, if you're interested in more than one of these, be sure to check out our everything bundle. It's like a subscription that never expires. Be sure to subscribe to the show, open your favorite pod catcher and search for Python we should be right at the top. You can also find the iTunes feed at slash iTunes, the Google Play foetus slash play in the direct RSS feed at slash RSS on talk python.fm. This is your host, Michael Kennedy. Thanks so much for listening. I really appreciate it. I'll get out there and write some Python code.