WEBVTT 00:00:00.001 --> 00:00:01.960 Do you think it's a good idea to test your software? 00:00:01.960 --> 00:00:05.460 Do you write unit tests or other automated verification for your code? 00:00:05.460 --> 00:00:07.600 I think most of us do these days. 00:00:07.600 --> 00:00:12.540 But a key question is, how do you know whether your tests efficiently verify your code? 00:00:12.540 --> 00:00:14.720 And the standard answer is code coverage. 00:00:14.720 --> 00:00:17.780 But there's a big difference between executing code, 00:00:17.780 --> 00:00:20.920 which is what code coverage measures, and truly verifying it. 00:00:20.920 --> 00:00:23.140 On this episode, we'll talk with Austin Bingham. 00:00:23.140 --> 00:00:26.180 He created a mutation testing framework for Python 00:00:26.180 --> 00:00:29.360 that goes beyond code coverage to actually perform this verification. 00:00:30.020 --> 00:00:31.420 It's a fresh and powerful idea. 00:00:31.420 --> 00:00:32.340 I hope you enjoy it. 00:00:32.340 --> 00:00:37.900 This is Talk Python To Me, episode 63, recorded June 15, 2016. 00:00:37.900 --> 00:00:41.400 Developers, developers, developers, developers. 00:00:41.400 --> 00:00:44.560 I'm a developer in many senses of the word 00:00:44.560 --> 00:00:48.920 because I make these applications, but I also use these verbs to make this music. 00:00:48.920 --> 00:00:53.480 I construct it line by line, just like when I'm coding another software design. 00:00:53.480 --> 00:00:56.660 In both cases, it's about design patterns. 00:00:56.660 --> 00:00:58.160 Anyone can get the job done. 00:00:58.220 --> 00:00:59.680 It's the execution that matters. 00:00:59.680 --> 00:01:01.140 I have many interests. 00:01:01.140 --> 00:01:02.080 Sometimes you can flip it. 00:01:02.080 --> 00:01:05.620 Welcome to Talk Python To Me, a weekly podcast on Python, 00:01:05.620 --> 00:01:08.680 the language, the libraries, the ecosystem, and the personalities. 00:01:08.680 --> 00:01:10.820 This is your host, Michael Kennedy. 00:01:10.820 --> 00:01:12.800 Follow me on Twitter, where I'm @mkennedy. 00:01:12.800 --> 00:01:16.700 Keep up with the show and listen to past episodes at talkpython.fm, 00:01:16.700 --> 00:01:19.280 and follow the show on Twitter via at Talk Python. 00:01:19.780 --> 00:01:22.400 This episode is brought to you by Hired and SnapCI. 00:01:22.400 --> 00:01:29.100 Thank them for supporting the show on Twitter via at Hired underscore HQ and at Snap underscore CI. 00:01:29.100 --> 00:01:30.440 Hey, everyone. 00:01:30.440 --> 00:01:34.800 We have an interesting deep dive into the world of Python testing and Python internals today. 00:01:35.160 --> 00:01:39.420 Before we chat with Austin about mutation testing and his Python library called Cosmic Ray, 00:01:39.420 --> 00:01:42.640 I have a few goodies to give away to a lucky couple of listeners. 00:01:42.640 --> 00:01:48.880 First, Austin and his co-author Rob are giving away a copy of their book, The Python Apprentice, 00:01:48.880 --> 00:01:51.600 as well as two free passes to their online Python course. 00:01:51.600 --> 00:01:56.920 As always, just visit talkpython.fm and make sure you're a friend of the show to be eligible to win. 00:01:56.920 --> 00:01:58.980 I'll pick three lucky winners next week. 00:01:58.980 --> 00:02:00.360 Now, let's meet Austin. 00:02:00.360 --> 00:02:02.320 Austin, welcome to the show. 00:02:02.700 --> 00:02:04.220 Thanks, Mike. I'm really glad to be here. 00:02:04.220 --> 00:02:10.360 Yeah, I'm super excited to share this mutation testing idea that you've sort of manifested in Python. 00:02:10.360 --> 00:02:11.300 That's really cool. 00:02:11.300 --> 00:02:12.540 We'll talk a lot about that. 00:02:12.540 --> 00:02:14.680 Before we get into it, though, what's your story? 00:02:14.680 --> 00:02:15.940 How did you get into Python and programming? 00:02:15.940 --> 00:02:20.920 Well, how I got into programming was when I was quite young, I guess, around 10 years old. 00:02:20.920 --> 00:02:27.840 We had a computer around the house, and it was an old IBM AT or something along those lines. 00:02:27.840 --> 00:02:29.000 I forget exactly the model. 00:02:29.240 --> 00:02:32.820 And it could be programmed in BASIC, and that's really caught my attention. 00:02:32.820 --> 00:02:39.520 My parents got me some magazines and so forth that taught me how to do more complicated things than I could figure out on my own. 00:02:39.520 --> 00:02:40.760 And it sort of took off from there. 00:02:40.760 --> 00:02:45.340 Python, I have tried to figure out where I first started using Python. 00:02:45.340 --> 00:02:47.260 And to be honest, I'm not entirely sure. 00:02:47.260 --> 00:02:49.640 I think it was around in graduate school, though. 00:02:49.640 --> 00:02:52.500 So this would have been in the late 90s. 00:02:52.500 --> 00:02:52.960 Nice. 00:02:53.020 --> 00:02:53.820 What did you study in grad school? 00:02:53.820 --> 00:02:55.300 That was software engineering. 00:02:55.300 --> 00:02:57.060 This was at University of Texas at Austin. 00:02:57.060 --> 00:03:03.420 And we were doing all sorts of stuff related to artifact traceability in large-scale software systems. 00:03:03.420 --> 00:03:10.260 And somewhere in there, Python showed up for a build system or something along those lines, and it really caught my attention. 00:03:10.260 --> 00:03:13.560 And it sort of started there and grew and grew with my career. 00:03:13.560 --> 00:03:17.240 It has shown up everywhere since then to larger and larger degrees. 00:03:17.240 --> 00:03:22.460 So it's something that I've really enjoyed using for the past, I guess, 20 years at this point or so. 00:03:22.460 --> 00:03:24.200 Yeah, that's definitely a while. 00:03:24.200 --> 00:03:25.520 That's almost from the beginning, right? 00:03:25.520 --> 00:03:27.120 Not quite, but it's pretty close. 00:03:27.120 --> 00:03:28.380 Yeah, not quite the very beginnings, yes. 00:03:28.380 --> 00:03:29.700 But it's been a long time, yeah. 00:03:29.700 --> 00:03:30.520 Yeah, yeah. 00:03:30.520 --> 00:03:30.820 Awesome. 00:03:30.820 --> 00:03:40.900 So we were in Oslo together, Norway, last week with a bunch of other speakers and developers at the NDC, the Norwegian Developer Conference there. 00:03:41.340 --> 00:03:48.060 And I would say that you and I were the guys carrying the Python flag, if you will, right? 00:03:48.060 --> 00:03:48.460 Yes. 00:03:48.460 --> 00:03:53.560 We were kind of the Python guys in the sea, among other types of folks, right? 00:03:53.560 --> 00:03:54.400 Yes. 00:03:54.400 --> 00:03:56.400 That's very true. 00:03:56.400 --> 00:04:00.140 I mean, traditionally, Python doesn't have a large footprint at that conference. 00:04:00.140 --> 00:04:05.860 And so you and I definitely were the diplomats, I think, the ambassadors for Python. 00:04:06.440 --> 00:04:10.280 But I was surprised at how much interest there was in it among the other delegates. 00:04:10.280 --> 00:04:17.600 A lot of people have some glancing experience with it, and they were, I think, interested to see or to learn more about it. 00:04:17.600 --> 00:04:24.880 So I think it's a growing topic of interest, even at not traditionally Python-heavy conferences and venues. 00:04:24.880 --> 00:04:25.100 Yeah. 00:04:25.380 --> 00:04:31.460 Yeah, I've done Python talks at several conferences that were, I would say, decidedly not Python conferences. 00:04:31.460 --> 00:04:33.320 And they've been received really, really well. 00:04:33.320 --> 00:04:40.480 And I think it's just one more manifestation or one more piece of evidence that Python is really a growing ecosystem. 00:04:40.480 --> 00:04:41.340 Yes. 00:04:41.340 --> 00:04:42.560 Yeah, very much so. 00:04:42.560 --> 00:04:44.840 And it's just gaining in popularity every year. 00:04:44.840 --> 00:04:45.820 It's incredible. 00:04:46.400 --> 00:04:57.300 Yeah, it is, I say this a lot on the show, but it's amazing to me that, like, the language grew at a pretty respectable but not insane growth rate for a really long time. 00:04:57.300 --> 00:05:01.600 It kind of germinated and then, you know, just caught fire in the 2000s. 00:05:01.600 --> 00:05:02.240 It's cool. 00:05:02.240 --> 00:05:03.240 Yeah. 00:05:04.100 --> 00:05:05.620 Now is the time for Python, yeah. 00:05:05.620 --> 00:05:06.760 It is definitely the time. 00:05:06.760 --> 00:05:15.660 So, well, the reason I brought up NDC is you had a really cool presentation there on this concept, which is a general programming concept. 00:05:15.660 --> 00:05:18.300 It's available in Java and maybe some other languages. 00:05:18.300 --> 00:05:19.440 I'm not sure. 00:05:19.440 --> 00:05:21.440 But called mutation testing. 00:05:21.440 --> 00:05:24.580 So I've done a lot of unit testing and other kinds of testing. 00:05:24.580 --> 00:05:26.560 I've heard of genetic algorithms. 00:05:26.560 --> 00:05:28.380 So maybe it's genetic algorithms. 00:05:28.380 --> 00:05:31.880 I actually know, seeing the talk, I know that it's not necessarily. 00:05:32.420 --> 00:05:34.280 Why don't you tell us what mutation testing is? 00:05:34.280 --> 00:05:37.020 I thought it was really interesting, and I wanted to share it with the audience. 00:05:37.020 --> 00:05:38.060 Sure, sure. 00:05:38.060 --> 00:05:43.260 Mutation testing is, you can think of it in some sense as a test for your tests. 00:05:43.260 --> 00:05:49.580 The main goal of mutation testing is to gauge the effectiveness of your existing tests. 00:05:49.580 --> 00:05:58.960 So if you take the sort of theoretically perfect standpoint that you have a test suite that tests 100% of your functionality, at least in principle, 00:05:58.960 --> 00:06:03.580 then mutation testing can tell you if your test suite actually does test your functionality. 00:06:03.580 --> 00:06:14.420 You can find holes in your test system, and it can also help you find code in your code base that isn't tested, and maybe you can just be removed because it doesn't actually contribute to any real functionality. 00:06:14.420 --> 00:06:19.580 So mutation testing, as you said, has nothing to do with genetic algorithms. 00:06:19.580 --> 00:06:23.900 It doesn't try to search out failing test cases or something. 00:06:24.100 --> 00:06:25.900 It's a very dumb algorithm. 00:06:25.900 --> 00:06:28.200 It systematically makes small modifications. 00:06:28.200 --> 00:06:29.400 It's kind of exhaustive, right? 00:06:29.940 --> 00:06:30.360 Exactly. 00:06:30.360 --> 00:06:39.180 It's an exhaustive, brute force search through a pretty large space to try to just trick your tests into passing a mutant. 00:06:39.180 --> 00:06:46.160 I mean, the basic idea is that you make these very small changes to your code base and then run your test suite. 00:06:46.160 --> 00:06:51.560 And if your test suite passes, then we say that the mutant test survived, and this is what you don't want. 00:06:51.560 --> 00:06:53.380 This means that your test suite is incapable. 00:06:53.380 --> 00:06:57.920 It doesn't have the fidelity for detecting the change you've made, which we consider an error. 00:06:57.920 --> 00:06:58.560 Okay. 00:06:58.560 --> 00:07:03.160 So before we get into that, though, you said something that I thought was interesting. 00:07:03.160 --> 00:07:07.920 You said you have 100% code coverage, and yet your tests are not doing their thing. 00:07:07.920 --> 00:07:14.440 So I think there's like layers of, or levels rather, of sort of verifying your test. 00:07:14.440 --> 00:07:15.840 Like you're writing tests. 00:07:15.840 --> 00:07:16.640 That's level one. 00:07:16.640 --> 00:07:17.340 You have tests. 00:07:17.340 --> 00:07:17.900 They exist. 00:07:17.900 --> 00:07:19.360 It's an existing thing. 00:07:19.360 --> 00:07:19.660 Yeah. 00:07:19.660 --> 00:07:26.180 Step two or level two of enlightenment would be you have a significant amount of code coverage. 00:07:26.180 --> 00:07:32.360 Because without code coverage, you can have like 1,000 tests, but they could all be about some small, useless part of your app. 00:07:32.360 --> 00:07:35.200 And the important core section might actually not be tested. 00:07:35.200 --> 00:07:39.560 But this supposes you're kind of, you're at level two enlightenment, right? 00:07:39.620 --> 00:07:40.660 You have tests. 00:07:40.660 --> 00:07:44.320 You have good, maybe not 100%, but you have pretty solid code coverage. 00:07:44.320 --> 00:07:50.620 And now you want to say, is this actually, there's a difference between executing code and verifying code, right? 00:07:51.080 --> 00:07:51.260 Right. 00:07:51.260 --> 00:08:07.600 And this is a really important distinction that mutation testing gets to the heart of, which is, as you say, you could have 100% coverage in the sense that your test suite causes 100% of your instructions to be executed, however you define that instruction set. 00:08:07.600 --> 00:08:15.780 But it doesn't tell you whether or not, traditional coverage doesn't tell you whether or not your tests are verifying the functionality. 00:08:15.780 --> 00:08:24.320 So you could have a glaring defect in your program that your test suite is exercising, but not actually verifying. 00:08:24.320 --> 00:08:34.860 And mutation testing goes to the next level and tries to tell you if your test suite is actually verifying functionality, if it's capable of detecting actual errors. 00:08:34.860 --> 00:08:37.340 And if it's not, then we say that you need a more powerful test suite. 00:08:37.340 --> 00:08:40.340 That's kind of the whole point of mutation testing. 00:08:40.340 --> 00:08:44.140 It's this sort of adjunct to an existing test suite. 00:08:44.400 --> 00:08:52.340 And just to add on that, I mean, this question of 100% coverage, no matter how you slice it, is a really hard thing for most projects to have. 00:08:52.340 --> 00:08:57.620 In fact, most projects in the world don't have anything close to 100% coverage. 00:08:57.620 --> 00:09:02.280 But mutation testing can still be useful, even with systems that don't have 100% coverage. 00:09:02.280 --> 00:09:10.660 It just is going to throw up a lot of, initially at least throw up a lot of flags telling you that you have problems where you just don't have tests yet. 00:09:10.860 --> 00:09:16.860 It's not a technique that can just be used on systems that have 100% coverage already. 00:09:16.860 --> 00:09:18.040 I guess that's my point. 00:09:18.040 --> 00:09:23.380 That's good, because that means it would have been excluded from quite a wide bit of code out there, right? 00:09:23.380 --> 00:09:23.680 Yeah. 00:09:23.680 --> 00:09:24.960 Nobody would be able to use it. 00:09:24.960 --> 00:09:26.000 Yeah, that's the truth. 00:09:26.000 --> 00:09:26.420 Yeah. 00:09:26.420 --> 00:09:26.960 Right. 00:09:26.960 --> 00:09:27.780 Okay. 00:09:27.900 --> 00:09:30.300 So there's some really cool ideas here. 00:09:30.300 --> 00:09:31.420 You're talking about a mutant. 00:09:31.420 --> 00:09:36.760 This is the idea of changing your program, introducing some mutation. 00:09:36.760 --> 00:09:42.500 It's just some, almost randomly find a spot and make a change and see what the effect is. 00:09:42.500 --> 00:09:49.420 Because theoretically, you should be able to detect this change which broke something, presumably by changing it, right? 00:09:49.420 --> 00:09:52.020 Yeah, that's exactly the case. 00:09:52.020 --> 00:09:56.960 The mutations we're talking about, the modifications we're talking about are typically very, very small. 00:09:56.960 --> 00:10:03.200 And so the kind of canonical example that it's always trotted out is replacing a relational operator. 00:10:03.200 --> 00:10:08.340 So if I have some line of code that's, you know, X is less than one, I could change that. 00:10:08.340 --> 00:10:15.340 The mutation would be to change that to X is greater than one, for example, and then make that one small change and then run the test suite again. 00:10:15.420 --> 00:10:23.720 So these changes are very, very small, but the point is that the changes should all, in principle, be detectable by a sufficiently powerful test suite. 00:10:23.720 --> 00:10:24.400 Okay. 00:10:24.400 --> 00:10:32.760 That sounds like that would be true most of the time, but I think there might be some cases where it might not be detectable. 00:10:32.760 --> 00:10:39.680 Before we get into that, though, I just want to clarify for the listeners, you're not the one doing the mutations, right? 00:10:39.680 --> 00:10:42.020 Like, as a developer, that's not you. 00:10:42.020 --> 00:10:42.800 Correct. 00:10:42.920 --> 00:10:54.020 The whole point of a mutation testing tool is that it will do the hard work, the boring work of plowing through your code, finding the places that can potentially be modified, modifying them, and then running your test suite. 00:10:54.020 --> 00:11:06.420 So in principle, you should be able to point the tool at your code, tell it what your test suite is, and then walk away for probably a very long time and then come back to get your results. 00:11:07.780 --> 00:11:15.040 And, yeah, you don't – it takes away all the drudgery associated with that and actually gives you some really interesting results typically in the end. 00:11:15.040 --> 00:11:16.380 Yeah, that is really cool. 00:11:16.380 --> 00:11:21.740 So maybe we could think about, like, what does the mutation testing tell us? 00:11:21.800 --> 00:11:27.460 Because sometimes you might make a change and your code will then fail the test, right? 00:11:27.460 --> 00:11:37.880 So if you were testing that, I select a user and it's equal – the count of users I got back was one, and you change that to not equal, for example, your framework changes it to not equal. 00:11:37.880 --> 00:11:42.380 Obviously, that test would fail, but it could also change things that I don't detect up. 00:11:42.380 --> 00:11:43.260 I don't detect. 00:11:43.900 --> 00:11:44.840 Yes, that's true. 00:11:44.840 --> 00:11:58.020 You can have a result where your test suite passes a mutant, and then you go examine the code, and you realize that there's really no way to write a realistic test that would detect that change. 00:11:58.020 --> 00:12:03.240 And this is a class of mutants that, if you read the literature, is called an equivalent mutant. 00:12:03.240 --> 00:12:07.620 And an equivalent mutant is exactly that, a mutant that is functionally equivalent. 00:12:07.940 --> 00:12:08.620 It's still a mutant. 00:12:08.620 --> 00:12:13.160 It's been changed, but it's still functionally equivalent to the original program. 00:12:13.160 --> 00:12:19.780 And for some reason or the other – and this is a very language-specific thing – but for some reason or the other, you simply cannot detect it. 00:12:19.780 --> 00:12:28.300 And this is one of the really tricky, difficult aspects of mutation testing is ferreting out and somehow avoiding these equivalent mutants. 00:12:28.300 --> 00:12:28.720 Yes. 00:12:28.720 --> 00:12:30.060 Yeah, that's interesting. 00:12:30.060 --> 00:12:37.340 So obviously, if we mutate it and then the test fail, that's an upvote for our test, right? 00:12:37.480 --> 00:12:38.880 We've made a change to the code. 00:12:38.880 --> 00:12:40.840 We reran the test. 00:12:40.840 --> 00:12:42.700 The test said, your code has changed. 00:12:42.700 --> 00:12:43.720 It's no longer good. 00:12:43.720 --> 00:12:47.480 But then sometimes it might not come back. 00:12:47.480 --> 00:12:53.200 Is it possible that if, like, you change some kind of, like, while loop condition, it could just go forever? 00:12:53.200 --> 00:12:54.860 That's entirely possible. 00:12:54.860 --> 00:13:00.440 And this is yet another class of complexities that we have to deal with in mutation testing. 00:13:00.440 --> 00:13:06.400 These mutants that – the great example, the canonical example is what you said. 00:13:06.400 --> 00:13:07.600 They go into an infinite loop. 00:13:07.600 --> 00:13:12.600 So if I change, for example, one mutation would be to change a break to a continue. 00:13:12.600 --> 00:13:23.420 And if you do that, then you typically create a situation where an infinite loop is very, very likely because you've taken a place where your code is in the exit condition and then where it wants to break. 00:13:23.420 --> 00:13:24.380 And you said, no, don't break. 00:13:24.380 --> 00:13:25.260 Continue the loop. 00:13:25.380 --> 00:13:28.340 And it's going to stay in the exit condition and just kind of continue forever. 00:13:28.340 --> 00:13:34.080 So that kind of mutant falls into the category that we call incompetent. 00:13:34.080 --> 00:13:39.240 And I guess I should back up and say there are sort of three main categories for mutants. 00:13:39.240 --> 00:13:41.120 After you've run your test suite, you have some results. 00:13:41.440 --> 00:13:47.060 You talked about just a second ago where the test suite fails. 00:13:47.060 --> 00:13:50.720 And we say that in that case, your test suite has killed the mutant. 00:13:50.720 --> 00:13:53.860 Your test suite has failed, indicating that it knows that you've made a change. 00:13:53.860 --> 00:13:56.880 The other broad category is that your mutant survives. 00:13:56.880 --> 00:13:58.260 That is, your test suite passes. 00:13:58.460 --> 00:14:01.900 And this is where we start to look for weaknesses in our test suite. 00:14:01.900 --> 00:14:06.520 The third sort of smaller category is this category of incompetent mutants. 00:14:06.520 --> 00:14:13.460 Most incompetent mutants fail immediately by throwing an exception or failing to compile or doing something along those lines, 00:14:13.460 --> 00:14:18.060 something catastrophic that prevents them from actually even being run under the test suite. 00:14:18.060 --> 00:14:20.160 And these, we still count these as killed. 00:14:20.160 --> 00:14:22.200 These go in the checkbox category. 00:14:22.200 --> 00:14:22.920 This is good. 00:14:23.220 --> 00:14:30.860 But there are some incompetent mutants that do things like, you say, just run forever or maybe run for a very, very, very, very long time, 00:14:30.860 --> 00:14:34.200 so long that we don't really want to try to see if they stop. 00:14:34.200 --> 00:14:38.740 So this area is a difficult one. 00:14:38.740 --> 00:14:46.560 It's one that you have to address on a practical level when you develop tools to do mutation testing, this problem of incompetence. 00:14:46.560 --> 00:14:51.720 And, you know, you start looking into the theory of detecting incompetent mutants, 00:14:51.720 --> 00:14:55.720 and you run smack into Alan Turing's famous proof about the halting problem, 00:14:55.720 --> 00:15:02.100 saying that you cannot look at a program and determine a priori if it's going to stop running at some point in the future. 00:15:02.100 --> 00:15:06.180 And that's the problem you face with incompetent mutants and mutation testing. 00:15:06.180 --> 00:15:06.620 Yeah. 00:15:06.620 --> 00:15:10.380 There's not even much reasoning about it because it's just a brute force method anyway. 00:15:10.380 --> 00:15:11.260 Correct. 00:15:11.260 --> 00:15:11.520 Yeah. 00:15:11.660 --> 00:15:12.640 Yeah, that's a big challenge. 00:15:12.640 --> 00:15:18.760 Can you give me some idea of, like, how frequent that category shows up? 00:15:18.760 --> 00:15:22.020 Is that, like, 0.1%, 5%, 10%? 00:15:22.020 --> 00:15:29.760 That's a tough question to answer on a global scale because, I mean, I obviously haven't run mutation testing on every program. 00:15:29.760 --> 00:15:33.220 But in my experience, it's a relatively small amount. 00:15:33.220 --> 00:15:38.640 I mean, less than 1% of mutations, far less than 1% of mutations become incompetent. 00:15:39.140 --> 00:15:44.940 They're not a huge problem in practice because the strategies we use to deal with them are really simple, 00:15:44.940 --> 00:15:47.400 which basically what we do is we timeout. 00:15:47.400 --> 00:15:51.160 We establish, using one method or another, a timeout for your test suite. 00:15:51.160 --> 00:15:54.420 And if it takes longer than the timeout, then we just count that as incompetent. 00:15:54.420 --> 00:15:59.560 We say that mutant is in an infinite loop or in a huge loop, and we're going to say that it didn't get to run. 00:15:59.560 --> 00:16:03.880 If you consider performance part of your feature set, maybe it's failed anyway, right? 00:16:03.880 --> 00:16:04.540 Right. 00:16:04.540 --> 00:16:07.200 Yeah, it's clearly problematic at that point. 00:16:07.200 --> 00:16:08.400 Okay, interesting. 00:16:08.400 --> 00:16:11.140 And you actually had two ways of timing out. 00:16:11.140 --> 00:16:14.360 Like, you could just say, well, we're never going to run tests more than five minutes. 00:16:14.360 --> 00:16:16.980 But you had a cool thing to do with baselines as well, right? 00:16:17.080 --> 00:16:27.400 In the tools that we have right now for doing mutation testing in Python, the approach we take is to, well, as you said, one way is let the user provide a timeout. 00:16:27.400 --> 00:16:29.880 They can just provide an absolute timeout, and we'll honor that. 00:16:29.880 --> 00:16:40.300 Or we can run the test suite over unmutated code and time that and use that as a baseline and then let the user provide some multiplier, say, two or three. 00:16:40.300 --> 00:16:48.420 And then if a mutant's test suite takes longer than n times the baseline timing, then we consider that an incompetent mutant and we kill it off. 00:16:48.820 --> 00:16:55.840 So this is our really simple but generally very effective approach to dealing with the halting problem in practice. 00:16:55.840 --> 00:16:57.840 Yeah, it's way easier than proving it, right? 00:16:57.840 --> 00:16:59.120 Right, yeah. 00:16:59.120 --> 00:17:00.980 So that would be really difficult to do. 00:17:00.980 --> 00:17:02.140 Yeah, nice. 00:17:02.140 --> 00:17:05.820 Okay, so we have obviously the case where the mutant is killed. 00:17:05.820 --> 00:17:10.280 We have this incompetent mutant, which we kind of can't really deal with. 00:17:10.680 --> 00:17:13.620 But then we have the more challenging case. 00:17:13.620 --> 00:17:21.200 It's maybe the interesting case, you would say, where you've changed the code, you run the test, and the tests all still pass. 00:17:21.200 --> 00:17:22.240 Right? 00:17:22.240 --> 00:17:24.700 So there's a couple of conclusions you could draw from this, yeah? 00:17:24.700 --> 00:17:26.480 Well, yeah. 00:17:26.480 --> 00:17:34.420 If you've made a mutation and the tests still pass, then you have a couple of things to look into. 00:17:35.300 --> 00:17:40.880 One possibility, the standard thing that happens in that case is that you just don't have enough tests. 00:17:40.880 --> 00:17:45.680 You need more or better tests because you have some change that was undetected by your test suite. 00:17:45.680 --> 00:17:53.620 The other very common, the less common but still quite common case is that you have code in your program that doesn't need to be there anymore. 00:17:53.620 --> 00:17:56.340 It's extra code that doesn't contribute to any functionality. 00:17:56.340 --> 00:18:02.780 So your test, in that case, your test suite is perfectly good because it's testing the things it needs to be testing, 00:18:02.780 --> 00:18:04.700 the important functionality of your program. 00:18:05.280 --> 00:18:10.520 But you've got bits of code that can be mutated but aren't being tested, so you should yank those bits of code out. 00:18:10.520 --> 00:18:16.560 If you view code as a liability rather than something important to keep around, you can just get rid of it at that point. 00:18:16.560 --> 00:18:23.620 The third possibility, and this is really a subcategory of the first, is that we come back to this notion of equivalent mutants. 00:18:23.620 --> 00:18:26.460 Mutants that they have been changed. 00:18:26.460 --> 00:18:31.300 Our test suite hasn't detected them, but there's no practical way to write a test for those. 00:18:31.300 --> 00:18:35.040 And there's all sorts of interesting examples of these. 00:18:35.040 --> 00:18:41.660 They're a bit difficult to describe, perhaps, purely without showing some code. 00:18:42.660 --> 00:18:49.900 Speaking of showing, all the videos of the sessions, including yours and mine from NDC, will be online shortly. 00:18:49.900 --> 00:18:55.200 And so as soon as they are online, I'll put a link to your presentation so people can go back and see it. 00:18:55.200 --> 00:18:59.920 But yeah, it is tough to talk about code examples on audio, right? 00:19:00.400 --> 00:19:01.580 Yeah, it's quite difficult. 00:19:01.580 --> 00:19:06.580 But for Python, I think I could probably describe the Dunder main example. 00:19:06.580 --> 00:19:08.000 Yeah, go for it. 00:19:08.000 --> 00:19:23.300 So one equivalent mutant that, in retrospect, is quite obvious, but I hadn't really anticipated, is the standard idiom in Python of using Dunder name equals Dunder main to set up your main block when you're writing a program. 00:19:23.760 --> 00:19:33.900 But of course, if you've got that in your program and you have any kind of code in that block that can be mutated, the mutation testing suite will mutate that code. 00:19:33.900 --> 00:19:42.160 But of course, that block is never executed in a test because it's not accessible inside a test because Dunder name does not equal Dunder main in that case ever. 00:19:42.660 --> 00:19:50.960 So you have this really interesting case where you have this whole body of code that's really important to your program in a way, but that cannot be tested and never will be tested. 00:19:50.960 --> 00:19:55.400 So that's the flavor of at least some equivalent mutants. 00:19:55.400 --> 00:20:06.880 But one of the joys of equivalent mutants is that when you find them, you have these aha moments almost every time because they're surprising, they're interesting, and they kind of make you scratch your head a little bit. 00:20:07.220 --> 00:20:12.420 And so it's one of the, I guess you might say, strange joys of mutation testing. 00:20:12.420 --> 00:20:15.280 Yeah, that does sound pretty interesting. 00:20:15.280 --> 00:20:18.840 It definitely gives you some insight you probably wouldn't normally get. 00:20:30.140 --> 00:20:32.220 This episode is brought to you by Hired. 00:20:32.220 --> 00:20:37.820 Hired is a two-sided, curated marketplace that connects the world's knowledge workers to the best opportunities. 00:20:37.820 --> 00:20:45.320 Each offer you receive has salary and equity presented right up front, and you can view the offers to accept or reject them before you even talk to the company. 00:20:45.320 --> 00:20:51.060 Typically, candidates receive five or more offers within the first week, and there are no obligations ever. 00:20:51.060 --> 00:20:52.620 Sounds awesome, doesn't it? 00:20:52.620 --> 00:20:54.280 Well, did I mention the signing bonus? 00:20:54.280 --> 00:20:57.700 Everyone who accepts a job from Hired gets a $1,000 signing bonus. 00:20:58.140 --> 00:21:00.460 And as Talk Python listeners, it gets way sweeter. 00:21:00.460 --> 00:21:06.180 Use the link Hired.com slash Talk Python To Me, and Hired will double the signing bonus to $2,000. 00:21:06.180 --> 00:21:08.320 Opportunities knocking. 00:21:08.320 --> 00:21:11.760 Visit Hired.com slash Talk Python To Me and answer the call. 00:21:16.480 --> 00:21:21.580 One example that I was thinking of when you were talking about that category is like logging, right? 00:21:21.580 --> 00:21:27.400 So maybe you've got some test, and it says, if this, then log this thing, else log that. 00:21:27.400 --> 00:21:32.000 And, you know, would it really make sense to, like, write a test to detect what you're logging? 00:21:32.000 --> 00:21:32.760 Right. 00:21:32.760 --> 00:21:33.760 That's a really good point. 00:21:33.900 --> 00:21:38.480 I mean, most, a large category of equivalent mutants are exactly of that flavor. 00:21:38.480 --> 00:21:46.680 The changes caused by the mutation are things that you, in principle, could test for, but you never would because there's no reason to do it. 00:21:46.680 --> 00:21:49.460 There may be no business reason to do it. 00:21:49.460 --> 00:21:54.300 There may be just no practical reason to do it, depending on what values are driving your project. 00:21:55.220 --> 00:21:57.440 And so you end up not ever writing tests for those. 00:21:57.440 --> 00:22:12.240 And one of the challenges of writing mutation testing tools is allowing your user to specify, in some way, shape, or form, that mutations should not be performed on certain bodies of code for various reasons. 00:22:12.240 --> 00:22:19.180 And this is something that all the tools that do mutation testing have to account for in some way, shape, or form. 00:22:19.180 --> 00:22:19.540 Okay. 00:22:19.540 --> 00:22:24.480 So I'm going to hold my question on, like, how you deal with that until we get to your framework. 00:22:24.480 --> 00:22:25.480 Okay, good. 00:22:25.480 --> 00:22:30.300 Because that is a really interesting problem, and I want to dig into it. 00:22:30.300 --> 00:22:39.820 But before we do, like, could you give us some, you know, you talked about the sort of the basic changes, like, if you've got a less than, change that to a greater than. 00:22:39.820 --> 00:22:42.360 There's a whole variety of different types. 00:22:42.360 --> 00:22:44.580 There's, like, language agnostic changes. 00:22:44.580 --> 00:22:47.660 There's changes you can make that affect object-oriented programming. 00:22:47.660 --> 00:22:50.080 Can you give us, like, a sense of those? 00:22:50.080 --> 00:22:51.540 Sure, yeah. 00:22:51.540 --> 00:22:51.580 Yeah. 00:22:51.580 --> 00:22:56.700 So this is actually one of the areas where there's active research into mutation testing. 00:22:56.700 --> 00:23:05.200 There's not a huge group of people doing this research, but the research that is going on is, to a large degree, into which kinds of mutations should we be actually performing. 00:23:05.460 --> 00:23:14.900 So you mentioned that there's some mutations that are language agnostic in the sense that they apply to almost all programming languages you can imagine. 00:23:14.900 --> 00:23:18.980 And so a typical example would be something like replacing a constant. 00:23:19.100 --> 00:23:27.440 If you found the constant 4 in your code, the mutation testing suite might change that to 5 or 19 or negative 6 or something like that. 00:23:27.440 --> 00:23:29.520 So this is, you know, it's an obvious change. 00:23:29.520 --> 00:23:31.960 I mean, it's a sort of blatant change. 00:23:31.960 --> 00:23:33.260 It should be obviously testable. 00:23:33.260 --> 00:23:36.460 And so it's ripe for that kind of thing. 00:23:36.540 --> 00:23:43.360 But other examples include things like replacing arithmetic operators, removing unary or adding unary operators. 00:23:43.360 --> 00:23:46.880 We talked earlier about relational operators replacing those. 00:23:46.880 --> 00:23:52.400 And all of these sort of fall into – they're broadly applicable. 00:23:52.560 --> 00:24:01.260 You could see them being applied in a functional language or an OO language or any other procedural language, whatever kind of language you happen to be working on. 00:24:01.260 --> 00:24:06.880 But some research has looked into mutations that are specific to, for example, object-oriented languages. 00:24:06.880 --> 00:24:15.620 Most – well, not Python, but a lot of object-oriented languages have access modifiers, for example, private, public, protected, and so forth. 00:24:15.620 --> 00:24:27.840 And so one really clever and interesting mutation is to replace public with private or vice versa, to basically go in and mess with the access modifiers and see if that is detected by the test suite. 00:24:27.840 --> 00:24:29.760 Often that results in something that can't compile. 00:24:29.760 --> 00:24:35.540 In C++, for example, if you changed public to private, that would probably break compilation of many programs. 00:24:35.540 --> 00:24:38.740 But changing private to public, it's hard to say. 00:24:38.740 --> 00:24:41.520 That's actually very, very difficult to test for. 00:24:41.520 --> 00:24:48.260 Other examples of OO-specific mutations would include, for example, changing base class order. 00:24:48.260 --> 00:24:54.420 This is another one that can have really dramatic effects on what your program does. 00:24:54.420 --> 00:24:57.620 Or in other cases, it can have absolutely no effect whatsoever. 00:24:57.620 --> 00:25:07.320 So you could see how changing base class order in Python, for example, could have a completely undetectable – almost completely undetectable change to a program. 00:25:07.380 --> 00:25:12.960 The only way to detect that would be if you had a test that was checking base class order, was checking the MRO for the class. 00:25:12.960 --> 00:25:15.360 And, of course, nobody's going to write that test. 00:25:15.360 --> 00:25:17.860 And I'm not advocating that anybody write that kind of test. 00:25:17.860 --> 00:25:19.760 That's not the kind of code you want to write, that's for sure. 00:25:19.760 --> 00:25:20.300 Right. 00:25:20.300 --> 00:25:21.460 It's a complete waste of time. 00:25:21.460 --> 00:25:29.220 So that's, I think, a fairly enlightening example of the kinds of problems you face doing mutation testing. 00:25:29.220 --> 00:25:36.660 As elegant and interesting and straightforward as the approach sounds, there are these really difficult, thorny edge cases you have to deal with. 00:25:36.660 --> 00:25:41.560 Other branches of research into, for example, mutations for functional languages. 00:25:41.900 --> 00:25:47.660 In the classic example there, a lot of Haskell-like and F-sharp-like languages have pattern matching, for example, on their functions. 00:25:47.660 --> 00:25:52.840 And changing the order of pattern matching is a common mutation you might perform on a language like that. 00:25:52.840 --> 00:25:59.160 And that, again, is another area where sometimes changing the order makes a huge difference, and sometimes changing the order makes absolutely no difference. 00:25:59.160 --> 00:26:08.020 And in those cases, I actually don't know how you would test for them because they're undetectable unless you have introspection and you have reflection capabilities. 00:26:08.020 --> 00:26:11.920 You actually go in and do the kinds of tests I talked about a second ago that you would never write. 00:26:11.920 --> 00:26:20.720 So it's a fascinating field to kind of dip your toe into, and the papers are pretty accessible if you want to read about these kinds of things as well. 00:26:20.720 --> 00:26:31.860 It sounds to me like one of the major challenges that you're going to run into for almost any reasonable-sized program is that it's going to be really slow, right? 00:26:31.860 --> 00:26:39.580 Because you're looking at basically every permutation of all the operators in, you know, inheritance, methods. 00:26:39.580 --> 00:26:43.060 There's a crazy number of things in play here, right? 00:26:43.060 --> 00:26:44.580 That's absolutely true. 00:26:44.580 --> 00:26:55.360 I think the single biggest practical problem with mutation testing, the single biggest practical roadblock for using mutation testing is that it takes a long, long, long time to do. 00:26:55.360 --> 00:27:12.260 If you consider the possibility of having, you know, dozens or, you know, 100 operators, you know, kinds of mutations that you might make in your code, and you have a large code base, you know, hundreds of thousands of lines of code is not uncommon in valuable systems. 00:27:12.760 --> 00:27:15.180 Or even not very valuable systems, for that matter. 00:27:15.180 --> 00:27:19.540 And then you consider the fact also that a test suite might take a considerable amount of time to run. 00:27:19.540 --> 00:27:27.960 So you have this triply nested loop of the operators, the places those operators can be applied, and the amount of time it takes to run your test suite. 00:27:27.960 --> 00:27:31.940 And you're talking about, you know, if you do the math, you can find on some systems that adds up to years. 00:27:31.940 --> 00:27:37.860 I mean, literally, it's not something you can do on a practical basis for all your code in any way, shape, or form. 00:27:37.860 --> 00:27:42.060 But there are some strategies that we can apply to try to deal with that. 00:27:42.060 --> 00:27:45.280 The most basic strategy is simply to parallelize. 00:27:45.280 --> 00:27:51.400 For all the problems we have with long run times and mutation testing, the saving grace is perhaps that it's embarrassingly parallel. 00:27:51.400 --> 00:28:00.520 You can run each mutation slash test suite run in a completely separate process, all at the same time if you want to, and it won't affect the results. 00:28:00.520 --> 00:28:11.820 So you could, in principle, go to Azure or Amazon and rent for five minutes, you know, 10,000 or 100,000 machines or whatever they'll let you get, run all your tests, and then be done with it. 00:28:11.820 --> 00:28:15.920 But that's not something that is probably economically feasible for most people. 00:28:16.300 --> 00:28:23.540 So other approaches that there are to dealing with this, well, there's not that many other approaches that I'm aware of. 00:28:23.540 --> 00:28:25.800 But one is another form of baselining. 00:28:25.800 --> 00:28:31.200 We talked earlier about baselining for your timeouts, you know, when to kill the test suite and call it incompetent. 00:28:31.260 --> 00:28:38.540 Another kind of baseline that you can do is to run a full test suite with all your operators over all your code and get those results. 00:28:38.980 --> 00:28:48.520 And then as you start making changes to your code base, only run the tests that, you know, work and exercise modified code, code that you've changed. 00:28:48.520 --> 00:28:54.140 And that way you can drastically reduce the scope of the number of tests you need to run. 00:28:54.140 --> 00:28:59.980 And that drastically reduces the number of operators that get applied, the amount of code space that can be potentially modified, and so forth. 00:28:59.980 --> 00:29:04.120 Also, you can tell your mutation testing system to only mutate code that was modified. 00:29:04.120 --> 00:29:14.720 So basically, we're analyzing deltas, analyzing our git diffs, so to speak, and saying only run the tests that we know could possibly have an impact or be impacted by the changes that were made. 00:29:14.720 --> 00:29:22.140 And this is a heuristic approach to speeding things up because, of course, it's not watertight. 00:29:22.140 --> 00:29:28.860 You could, of course, make changes to your code that influence the code paths that your tests are now exercising. 00:29:29.300 --> 00:29:36.280 And if you purely use this kind of baselining for determining which tests to run and what to test, then you'll be missing things. 00:29:36.280 --> 00:29:39.620 So you have to occasionally do, or at least in principle, you'll be missing things. 00:29:39.620 --> 00:29:43.920 So you'll have to do occasional rebaselinings to make sure that you've kept up with all of your changes. 00:29:43.920 --> 00:29:47.900 And it also assumes that you have some way of correlating your tests with lines of code. 00:29:48.020 --> 00:29:57.360 So this is where mutation testing and traditional coverage analysis tools can come into play, where they can work together. 00:29:57.360 --> 00:30:00.540 Because now you can say, okay, I take the coverage analysis information. 00:30:00.540 --> 00:30:02.720 I know which tests exercise which lines of code. 00:30:02.720 --> 00:30:08.420 I can couple that and kind of compare it to the deltas and determine which tests need to be run by the mutation testing suite. 00:30:09.100 --> 00:30:12.680 It's basically like an inverted code coverage, right? 00:30:12.680 --> 00:30:22.000 So if I look at this test, what part of my code in my real app changed or somehow was affected by running this test, right? 00:30:22.000 --> 00:30:26.760 And so you could just focus on, say, like 10 lines of code or probably way more than that. 00:30:26.760 --> 00:30:28.500 But focus it in on that area, right? 00:30:29.000 --> 00:30:29.360 Exactly. 00:30:29.360 --> 00:30:32.720 That's the point is drastically reduce the space. 00:30:32.720 --> 00:30:45.360 And I think in principle you could get this down to where you could run things fast enough that you could do it on every commit or bundles of commits rather than once a week or something along those lines, which may or may not be desirable. 00:30:45.360 --> 00:30:48.980 But it's an interesting goal from a tool developer's point of view. 00:30:48.980 --> 00:30:50.500 Yeah, it's definitely an interesting goal. 00:30:50.500 --> 00:31:00.500 Well, one of the things I was thinking of as you said this was, is this a thing that needs to run on, say, every check-in or every time you want to run your test? 00:31:00.500 --> 00:31:06.100 Because if you have a good set of tests, hopefully your tests are actually catching your bugs. 00:31:06.100 --> 00:31:16.800 And this feels to me like a validation of your tests rather than it seems like it could theoretically run less often and still be really valuable. 00:31:16.800 --> 00:31:18.440 I think in practice you're right. 00:31:18.580 --> 00:31:20.260 It doesn't need to run on every check-in. 00:31:20.260 --> 00:31:36.080 But if you're working on a team that wants perfect code coverage, for example, and that requires, say you have a policy on a legacy code system that any change you make needs to be backed up by tests, which is a common thing to do with existing legacy systems that are trying to improve their lot in this world. 00:31:36.080 --> 00:31:38.020 You might have that policy. 00:31:38.020 --> 00:31:38.980 This is on every commit. 00:31:38.980 --> 00:31:41.600 Whatever changes you've made need to be backed up by tests. 00:31:41.600 --> 00:31:43.280 And this is a good way to verify that. 00:31:43.820 --> 00:31:51.680 Not to verify just that you've made tests, but to verify that the tests you've created actually test the functionality correctly. 00:31:51.680 --> 00:31:57.880 And so if you can make mutation testing fast enough, you could actually enforce that kind of constraint in a pretty strong way. 00:31:57.880 --> 00:31:59.800 And that's an interesting thing. 00:31:59.800 --> 00:32:09.960 Yeah, that is quite interesting because my experience is there's a massive difference among team members on their level of embracing testing and how much they run the test. 00:32:09.960 --> 00:32:11.460 Like some people are really into it. 00:32:11.460 --> 00:32:14.940 Some people only run it if there's something making them run it, basically. 00:32:14.940 --> 00:32:15.520 Right. 00:32:16.340 --> 00:32:18.100 Yeah, that's very true. 00:32:18.100 --> 00:32:23.660 And so now you have a new stick to beat people around the head with if you have mutation testing in place. 00:32:23.660 --> 00:32:26.040 Nice. 00:32:26.040 --> 00:32:28.820 So let's bring this down to Python. 00:32:28.820 --> 00:32:29.680 Let's make it concrete. 00:32:29.680 --> 00:32:33.780 Let's talk about this thing called Cosmic Ray that you created. 00:32:33.780 --> 00:32:34.480 Okay. 00:32:34.480 --> 00:32:34.800 Yeah. 00:32:34.800 --> 00:32:39.420 Cosmic Ray is, as you just hinted, it's a mutation testing tool for Python. 00:32:39.420 --> 00:32:42.620 I should say it's not the first mutation testing tool for Python. 00:32:42.620 --> 00:32:47.780 There were a few available when I started writing it, but none of them were work. 00:32:47.780 --> 00:32:50.300 They didn't quite work the way I wanted or they were unmaintained. 00:32:50.300 --> 00:32:53.220 And really, this was an interesting project in its own right. 00:32:53.220 --> 00:32:56.880 So this started out almost as just a fun thing to do. 00:32:56.880 --> 00:32:59.920 And it turned out to be a really fascinating project all around. 00:32:59.920 --> 00:33:10.180 But Cosmic Ray is a system for searching through your Python code, finding places to mutate, making those mutations, and then running your test suite. 00:33:11.140 --> 00:33:14.500 And it's a fairly young project. 00:33:14.500 --> 00:33:17.300 And it has quite a bit of work left to be done on it. 00:33:17.300 --> 00:33:19.740 But it has produced some results already. 00:33:19.740 --> 00:33:21.940 So it's looking quite promising. 00:33:21.940 --> 00:33:25.420 It's about a year and a half old, I think, at this point. 00:33:25.420 --> 00:33:29.160 And really, it's only been used by me and a few sort of close, trusted friends. 00:33:29.160 --> 00:33:31.000 But it's open source. 00:33:31.000 --> 00:33:31.540 It's on GitHub. 00:33:31.680 --> 00:33:38.860 And anybody who wants to try it or make contributions or give any feedback is more than welcome and, in fact, encouraged to go take a look at it. 00:33:38.860 --> 00:33:39.080 Yeah. 00:33:39.080 --> 00:33:39.800 Yeah, awesome. 00:33:39.800 --> 00:33:42.520 And I'll be sure to link to the GitHub repo and things like that. 00:33:42.520 --> 00:33:43.920 And it's on PyPI, of course, right? 00:33:43.920 --> 00:33:46.600 Honestly, I'm not sure. 00:33:46.600 --> 00:33:47.380 I think it is. 00:33:47.420 --> 00:33:50.780 But I don't know the last time I pushed up a version to PyPI. 00:33:50.780 --> 00:33:52.320 Let me see here. 00:33:52.320 --> 00:33:53.920 Yes, it is. 00:33:53.920 --> 00:33:54.720 It is on. 00:33:54.720 --> 00:33:57.200 It's cosmic underscore ray on PyPI. 00:33:57.200 --> 00:33:57.520 Woo. 00:33:57.520 --> 00:33:57.860 Okay. 00:33:57.860 --> 00:33:59.060 Saved. 00:34:00.280 --> 00:34:05.880 I guess the interesting parts for a lot of people are going to be how cosmic ray works internally. 00:34:05.880 --> 00:34:06.640 Yeah, absolutely. 00:34:06.640 --> 00:34:09.700 There is some really amazing stuff in there. 00:34:09.700 --> 00:34:14.040 Before we get into that, could you just really quickly, like, tell me, what do I need to do? 00:34:14.040 --> 00:34:22.260 Like, if I've got some Python app with some tests, you know, I'm using pytest or something like that, like, what are my steps to apply this? 00:34:22.260 --> 00:34:24.880 The steps are, well, pretty straightforward. 00:34:24.880 --> 00:34:28.860 Identify the parts of your code that you want to mutation test. 00:34:29.520 --> 00:34:38.480 Very often you'll have some part of your code that has a good test suite and is heavily, thoroughly tested and is central to the functioning and other parts that aren't. 00:34:38.480 --> 00:34:44.180 And you can use cosmic ray to divvy out and slice and dice the parts you do and do not want to test. 00:34:44.180 --> 00:34:49.520 So if you just want to take it for a spin, identify some module that you're interested in. 00:34:49.520 --> 00:34:53.300 Because you want it to happen in a non-trivial, you know, in a short amount of time, right? 00:34:53.300 --> 00:34:54.820 Well, that's one of the other reasons. 00:34:54.820 --> 00:34:58.560 Yeah, you'll get more bang for your buck if you're just trying to test drive this than if you try to run it over. 00:34:58.760 --> 00:34:59.600 You know, a 10,000. 00:34:59.600 --> 00:35:01.100 If you want to run this over Django, forget it. 00:35:01.100 --> 00:35:01.760 It's not going to work. 00:35:01.760 --> 00:35:03.580 But I mean, not in any practical sense. 00:35:03.580 --> 00:35:09.040 But if you want to run it over a single module in Django or, you know, some other package, then you'll have more luck. 00:35:09.040 --> 00:35:10.900 That's been my experience, at least, with it so far. 00:35:11.060 --> 00:35:12.960 But yeah, you'll need a test suite. 00:35:12.960 --> 00:35:19.560 Right now, we only support unit tests, the standard library unit tests, and pytest as the test suites we support. 00:35:19.560 --> 00:35:22.180 But there's a plug-in system for other testing systems. 00:35:22.940 --> 00:35:26.140 If you feel you need one supported, they're pretty easy to add. 00:35:26.140 --> 00:35:30.040 Point Cosmic Ray at your module and at your test suite. 00:35:30.040 --> 00:35:34.060 And you'll pass it a few other parameters, you know, things having to do with timeouts and so forth. 00:35:34.580 --> 00:35:39.740 And it will build up a work order, you know, basically the list of things it's going to do and put those in a little database. 00:35:39.740 --> 00:35:42.680 And then you'll need to set up Celery. 00:35:42.680 --> 00:35:50.480 Celery is a task distribution queue that runs on top of RabbitM queue by default. 00:35:50.480 --> 00:35:57.760 And this, we use Celery to distribute work out to workers that actually do the mutation, run a test suite, and then send results back. 00:35:57.760 --> 00:36:01.540 And so you'll have workers sitting on your Celery queue. 00:36:01.540 --> 00:36:06.140 And then you'll tell Cosmic Ray to run the work order that it's built. 00:36:06.140 --> 00:36:11.380 And it will start doling out work to these workers and collating the results back into the little database it's got. 00:36:11.380 --> 00:36:15.500 And that's, I mean, that's the short version of what you need to do. 00:36:15.500 --> 00:36:20.080 Once you have results back, then you start analyzing them and trying to figure out what Cosmic Ray is telling you. 00:36:20.080 --> 00:36:24.920 Right, you look at those three categories, you decide what to ignore, what not to ignore. 00:36:24.920 --> 00:36:45.120 Gone are the days of tweaking your server, merging your code, and just hoping it works in your production environment. 00:36:45.120 --> 00:36:49.080 With SnapCI's cloud-based, hosted, continuous delivery tool, 00:36:49.400 --> 00:36:55.040 You simply do a git push, and they auto-detect and run all the necessary tests through their multi-stage pipelines. 00:36:55.040 --> 00:36:58.560 Something fails, you can even debug it directly in the browser. 00:36:58.560 --> 00:37:03.740 With a one-click deployment that you can do from your desk or from 30,000 feet in the air, 00:37:03.740 --> 00:37:06.520 Snap offers flexibility and ease of mind. 00:37:06.520 --> 00:37:08.340 Imagine all the time you'll save. 00:37:08.340 --> 00:37:14.400 Thanks, SnapCI, for sponsoring this episode by trying them for free at snap.ci.com. 00:37:14.400 --> 00:37:30.140 And is there a way to flag it and say, this thing you've detected here, I want to ignore that? 00:37:31.140 --> 00:37:31.820 Not yet. 00:37:31.820 --> 00:37:38.200 And this is actually one of the big open areas for development, is how do we let users specify exceptions effectively? 00:37:38.200 --> 00:37:42.020 How do we let them say, don't make this mutation on this line of code? 00:37:42.020 --> 00:37:44.820 Or even more coarsely, don't make mutations on this line of code. 00:37:45.560 --> 00:37:54.080 Because we need that kind of thing because of the problems of equivalent mutants and so forth that we have no real solution to. 00:37:54.080 --> 00:37:58.980 Right now, there's a thought about the direction to take this in, how to deal with it. 00:37:59.200 --> 00:38:06.980 If you look at tools like PyLint, they have great systems for putting essentially orders into comments in your code telling PyLint, 00:38:06.980 --> 00:38:09.680 please don't apply rule such and such to this line of code. 00:38:09.680 --> 00:38:13.300 We could probably apply the same kind of technique to Cosmic Ray. 00:38:13.300 --> 00:38:19.920 But I'm not sure yet if that's better than having some extrinsic description of exceptions. 00:38:19.920 --> 00:38:21.740 It's basically an open question. 00:38:21.740 --> 00:38:27.680 And if anybody has ideas or wants to take a swing at it, this really is one of the big things that we need to sort out soon. 00:38:28.340 --> 00:38:29.340 Let's look inside. 00:38:29.340 --> 00:38:35.480 Basically, you point Cosmic Ray at your module and you say, go shred this thing. 00:38:35.480 --> 00:38:39.520 And for every shred that you create, go run the unit test, right? 00:38:39.520 --> 00:38:41.340 That's exactly right. 00:38:41.340 --> 00:38:43.060 Walk us through the internals there. 00:38:43.060 --> 00:38:44.700 There's some interesting stuff you're doing. 00:38:44.700 --> 00:38:50.440 Well, at the core of all of this is the standard library module AST. 00:38:50.440 --> 00:38:54.300 AST is short for, what's it, acronym for Abstract Syntax Tree. 00:38:54.640 --> 00:39:01.260 An Abstract Syntax Tree is just a programmatic structure defining a program, the syntax in your source code. 00:39:01.260 --> 00:39:05.300 When Python parses your source code, it produces an abstract syntax tree. 00:39:05.300 --> 00:39:10.920 And then you can access this looking at the different nodes in the tree, you know, looking at the different parts of your program. 00:39:10.920 --> 00:39:13.500 And not just look at them, but you can also change them. 00:39:13.780 --> 00:39:18.680 So what AST allows us to do in Cosmic Ray is load up your source code. 00:39:18.680 --> 00:39:21.140 We literally read your source code from your .py file. 00:39:21.140 --> 00:39:26.880 And we pass it into a parse function, which parses the source code into the abstract syntax tree. 00:39:26.880 --> 00:39:31.520 And then AST has other components, which allow us to walk down that tree. 00:39:31.720 --> 00:39:41.420 And if we want to make changes, the details of exactly how it does that, it might be difficult to talk about operators and things like that in too great of detail. 00:39:41.420 --> 00:39:42.440 Yeah. 00:39:42.440 --> 00:39:50.420 So, well, basically you get this abstract syntax tree and then you start applying your transformations to it, right? 00:39:50.420 --> 00:39:52.020 Your mutations, if you will. 00:39:52.020 --> 00:39:52.240 Yeah. 00:39:52.740 --> 00:39:54.360 Well, that's the fundamental idea, yes. 00:39:54.360 --> 00:39:59.720 So you have the AST and you find a place that you want to make a modification and then you make a modification to it. 00:39:59.720 --> 00:40:02.940 And there's support in the AST module for doing that kind of work. 00:40:02.940 --> 00:40:08.620 Once you've modified the AST, you then need to get it, you need to make it available to your test suite. 00:40:08.620 --> 00:40:09.760 You need to make it importable. 00:40:09.760 --> 00:40:13.220 And that's a whole other kind of second level trick. 00:40:13.220 --> 00:40:16.400 Yeah, because there's one thing to say, hey, Python, run this module. 00:40:16.500 --> 00:40:22.800 It's another to load up an individual AST and then turn that into executable things, right? 00:40:22.800 --> 00:40:23.600 Exactly. 00:40:23.600 --> 00:40:25.940 Everything it depends upon and so on. 00:40:25.940 --> 00:40:32.300 That was sort of the second big phase of work in building Cosmic Ray was figuring out how to do that. 00:40:32.300 --> 00:40:38.180 So once you have an AST, a modified AST, you can pass that to the built-in compile function. 00:40:38.180 --> 00:40:40.500 And that spits out what's called a code object. 00:40:40.620 --> 00:40:46.820 And that's this kind of thing that modules can use, so to speak, that we can execute to populate a module. 00:40:46.820 --> 00:40:56.340 So figuring out how to make that available, make your modified AST available through standard import was a big goal of Cosmic Ray. 00:40:56.340 --> 00:40:59.940 We didn't want people to have to modify their test suites to do mutation tests. 00:40:59.940 --> 00:41:05.900 We wanted the test suites to just naturally say import, you know, import of the module and get the right one. 00:41:05.900 --> 00:41:09.840 So we had to do a lot of investigation into how Python does this. 00:41:09.980 --> 00:41:17.200 And at the core, there's three main moving parts to how Python does imports, how it lets you control imports. 00:41:17.200 --> 00:41:20.180 The first thing is what's called a finder. 00:41:20.180 --> 00:41:32.820 And a finder is an object that's responsible, a class typically, but a function or a class that's responsible for telling Python that it knows how to load a module given that module's name. 00:41:32.820 --> 00:41:36.200 So Python will ask the finder, I've been asked to import foo. 00:41:36.200 --> 00:41:38.000 Do you know how to do anything with foo? 00:41:38.000 --> 00:41:39.400 And a finder can say yes or no. 00:41:39.400 --> 00:41:45.900 If a finder does know how to load something, it returns what's called a loader. 00:41:45.900 --> 00:41:52.220 And the loader is then responsible for populating essentially the shell of a module. 00:41:52.220 --> 00:42:04.800 So Python will make the empty shell of the module, pass it to the loader, and say, okay, now you populate this with the names, the functions, the name bindings, the constants, all that kind of stuff that come from the module that you're supposed to be loading for me. 00:42:05.520 --> 00:42:09.840 What we do is in Cosmic Ray, we have our own custom finder. 00:42:09.840 --> 00:42:14.640 And that finder is given the modified AST, and it's told the name of the module. 00:42:15.300 --> 00:42:19.200 And if it's then asked by Python, do you know how to load that module? 00:42:19.200 --> 00:42:19.840 It'll say yes. 00:42:19.840 --> 00:42:21.600 And then it hands back a loader. 00:42:21.600 --> 00:42:23.940 We have a custom loader, which also has this AST. 00:42:24.520 --> 00:42:36.120 And the custom loader is able to compile the AST, and then use that compiled AST to populate the shell module. 00:42:36.120 --> 00:42:40.600 And then that shell module is passed back to Python, and it gets naturally imported so that everybody can use it. 00:42:42.360 --> 00:42:46.960 I guess the last sort of moving part in this whole system is something called sys.metapath. 00:42:46.960 --> 00:42:49.960 If you import sys, you'll see it has an attribute called metapath. 00:42:49.960 --> 00:42:52.720 Metapath is just a list of finders. 00:42:52.720 --> 00:42:58.840 And when Python wants to import something, and some experts might tell me that I'm a little bit wrong with the details, but this is effectively correct. 00:42:59.640 --> 00:43:04.720 Python marches down metapath, asking each finder in order, do you know how to load this name? 00:43:04.720 --> 00:43:08.460 And the first finder that responds is the one that wins. 00:43:08.460 --> 00:43:16.100 So what we do is we take our custom finder, we populate it with its AST and its name, and we stick it at the front of metapath inside our worker processes. 00:43:16.100 --> 00:43:27.740 And these worker processes then are able to hijack the import system, in a sense, and put these mutated ASTs directly into place so that nobody has to know they're there, but they get imported naturally by whoever wants to use them. 00:43:27.820 --> 00:43:33.620 So that's the long and the short, I guess, of how we stick mutated ASTs into Python programs. 00:43:33.620 --> 00:43:37.860 Yeah, you really had to dig deep down inside the guts of Python. 00:43:37.860 --> 00:43:40.460 You had to take the red pill, not the blue pill, right? 00:43:40.460 --> 00:43:45.020 Yeah, there's a lot of pet archaeology and stuff here to get to the bottom of this. 00:43:45.020 --> 00:43:46.720 But at the end, it's very elegant and powerful. 00:43:46.720 --> 00:43:57.460 One of the joys of this project was learning all this stuff that I may never apply again, but I feel like I've reached the next level of my Python expertise, in a sense. 00:43:57.460 --> 00:43:58.960 Yeah, that's really cool. 00:43:58.960 --> 00:44:02.880 But it's awesome because you don't change your code to make this happen, right? 00:44:02.880 --> 00:44:07.320 It adapts to what it has to do to basically take over. 00:44:07.320 --> 00:44:08.220 Exactly. 00:44:08.220 --> 00:44:18.840 We work, Cosmic Ray works at a deep enough level that neither your test code nor your code under test needs to be modified to use Cosmic Ray. 00:44:19.180 --> 00:44:23.160 It should work transparently in all ways. 00:44:23.160 --> 00:44:24.940 Yeah, that was a big goal of the project. 00:44:24.940 --> 00:44:26.500 You talked about Celery. 00:44:26.500 --> 00:44:27.700 Celery is really awesome. 00:44:27.700 --> 00:44:33.540 There's a couple of other really cool projects that you're kind of built upon. 00:44:33.540 --> 00:44:35.860 One of them was this thing called TinyDB. 00:44:35.860 --> 00:44:38.140 Yeah, TinyDB. 00:44:38.140 --> 00:44:41.980 Well, it is what its name says. 00:44:41.980 --> 00:44:42.920 It's a tiny database. 00:44:42.920 --> 00:44:52.760 It's a little embedded file-oriented JSON database that you can import into your Python program and use with basically no configuration. 00:44:52.760 --> 00:44:57.540 So it was exactly what I was looking for when I was looking for a database for Cosmic Ray. 00:44:57.540 --> 00:45:03.060 We use the database for basically keeping track of the work order I described earlier. 00:45:03.580 --> 00:45:08.400 The first thing you do in a mutation testing run is figure out what it is you're going to do and write all that down. 00:45:08.400 --> 00:45:09.560 We write that into the database. 00:45:09.560 --> 00:45:14.280 And then as the results arrive back via Celery, we stick the results back into this database. 00:45:14.280 --> 00:45:18.960 So TinyDB is something that's worked out really well for us so far. 00:45:18.960 --> 00:45:23.240 And it was, as I said, super easy to use, and it's stuck around so far. 00:45:23.240 --> 00:45:26.860 I have a feeling that it's going to end up being a bottleneck in larger projects. 00:45:26.860 --> 00:45:29.740 But that's a gut feeling. 00:45:29.740 --> 00:45:31.980 I don't have any evidence to indicate that. 00:45:31.980 --> 00:45:36.140 But if it has to be replaced, then we'll start looking at something like SQLite, 00:45:36.140 --> 00:45:41.120 or maybe we'll give the user the power to specify MongoDB or whatever they want. 00:45:41.120 --> 00:45:48.440 But TinyDB is really worth looking at, I think, if you don't have really sophisticated needs in a database 00:45:48.440 --> 00:45:49.940 and you want something that's just file-oriented. 00:45:49.940 --> 00:45:54.820 It's a really beautiful little program that worked out of the box with really no reading on my part whatsoever. 00:45:54.820 --> 00:45:55.820 That's lovely. 00:45:55.820 --> 00:46:00.380 I really like to use SQLite and SQLAlchemy together. 00:46:00.580 --> 00:46:03.660 And those work really well in sort of an equivalent way. 00:46:03.660 --> 00:46:06.500 But I'm a huge fan of the document databases. 00:46:06.500 --> 00:46:13.740 One of the big selling points, what made me stick with TinyDB, is that it literally is a JSON file. 00:46:13.740 --> 00:46:16.280 I can open it up in Emacs and just look at it. 00:46:16.740 --> 00:46:20.880 And I don't have to have any extra tools to examine its contents. 00:46:22.660 --> 00:46:26.540 I think that that JSON nature is what's going to be its downfall. 00:46:26.540 --> 00:46:29.120 That's what makes me think it's not going to last that long for this project. 00:46:29.120 --> 00:46:35.580 But that's been a real selling point, is I can run my tests as I'm testing CosmicWrite, 00:46:35.580 --> 00:46:40.360 which, as you might imagine, is a real challenge, and then see what's in the database really, really easily. 00:46:40.360 --> 00:46:44.600 And so my cycle time has been pretty high by using TinyDB. 00:46:45.200 --> 00:46:45.460 Oh, yeah. 00:46:45.460 --> 00:46:45.880 That's cool. 00:46:45.880 --> 00:46:47.880 And it's 100% Python, according to GitHub. 00:46:47.880 --> 00:46:49.600 That sounds right. 00:46:49.600 --> 00:46:49.820 Yeah. 00:46:49.820 --> 00:46:52.060 I don't remember any compilation happening when I used it. 00:46:52.060 --> 00:46:52.320 Yeah. 00:46:52.320 --> 00:46:53.140 Nice. 00:46:53.140 --> 00:46:54.260 Yeah, it has 1,000 stars. 00:46:54.260 --> 00:46:55.640 So it's doing pretty well. 00:46:55.640 --> 00:46:56.780 I definitely want to check it out. 00:46:56.780 --> 00:46:58.820 The other one was DocOpt. 00:46:59.280 --> 00:46:59.680 Yeah. 00:46:59.680 --> 00:47:07.500 DocOpt is one of my current favorite packages for not just Python, but for lots and lots of languages. 00:47:07.500 --> 00:47:13.760 DocOpt is a tool for building command line parsers. 00:47:13.760 --> 00:47:20.680 But unlike things like ArtParse or the other sort of standard tools for doing this, it takes a kind of a backwards approach. 00:47:20.680 --> 00:47:28.740 You provide it with a string, which is the POSIX standard help output that you would get from any program. 00:47:29.260 --> 00:47:33.900 You know, saying usage colon program name, blah, blah, option names and all that kind of stuff. 00:47:33.900 --> 00:47:38.220 And the text information somebody gets when they type, you know, program dash H. 00:47:38.220 --> 00:47:44.760 You give that string to Cosmic Ray, and from that it generates a parser that can then parse command line arguments. 00:47:44.760 --> 00:47:49.960 So you never have to think really hard about, you know, building up these parser objects yourself. 00:47:49.960 --> 00:47:54.620 Everything is done magically, and all you have to do is think about how your pretty help message is going to look. 00:47:54.620 --> 00:47:55.680 Which you've got to write anyway. 00:47:56.440 --> 00:48:00.860 Which, yeah, you either have to write or have to get generated by some other tool. 00:48:01.020 --> 00:48:12.820 This has the neat effect that embedded in your code somewhere is your full help message that is great documentation, not just for your users, but also other programmers looking at your code. 00:48:12.820 --> 00:48:21.500 It really, it solves a really annoying problem that everybody and every programmer in the world has, which is writing parsers for command line arguments. 00:48:21.600 --> 00:48:23.000 And it does it in a really slick way. 00:48:23.000 --> 00:48:31.180 And one of the interesting things is that, I didn't know this until I looked at DocOpt, is that there actually is a POSIX standard for these help messages. 00:48:31.180 --> 00:48:36.360 So it can rely on an actual existing standard for defining these things, which is really cool. 00:48:36.360 --> 00:48:37.020 That is cool. 00:48:37.020 --> 00:48:41.540 Actually, that's the first way that I had heard of that there was a standard was by learning about DocOpt. 00:48:41.540 --> 00:48:43.500 Like, wait, there's a standard for the help message? 00:48:43.980 --> 00:48:44.580 Interesting. 00:48:45.520 --> 00:48:50.840 I highly recommend that anybody who has to write command line tools and who hasn't tried DocOpt, take a look at it. 00:48:50.840 --> 00:48:52.040 It's really addictive. 00:48:52.040 --> 00:48:55.300 And you can produce really, really powerful command line parsers. 00:48:55.300 --> 00:48:59.400 You know, things like you had with Git, you know, sub-command-based tools and stuff. 00:48:59.860 --> 00:49:06.720 I guess the other interesting thing about DocOpt is that while it was originally written in Python, the canonical implementation is Python. 00:49:06.720 --> 00:49:08.820 It exists now for something like 30 languages. 00:49:08.820 --> 00:49:17.160 So if you're a sometime C# developer, sometime Java developer, sometime whatever developer, you can continue using DocOpt in those languages as well. 00:49:17.160 --> 00:49:21.200 It's a neat project from that point of view, something that you don't see a lot of. 00:49:21.200 --> 00:49:24.940 Definitely means the idea of it resonated super well, right? 00:49:24.940 --> 00:49:25.360 Yeah. 00:49:25.360 --> 00:49:26.460 Yeah, it did. 00:49:27.020 --> 00:49:34.200 Okay, so we're getting kind of near the end of the show, and I wanted to ask you, you know, you have a company called 60 North, right? 00:49:34.200 --> 00:49:35.460 That's correct, yes. 00:49:35.460 --> 00:49:35.780 Yeah. 00:49:35.780 --> 00:49:38.360 You and Robert Spolscher, is that right? 00:49:38.360 --> 00:49:39.400 That's right, yes. 00:49:39.400 --> 00:49:40.320 You guys work together. 00:49:40.320 --> 00:49:43.580 Yeah, you guys are up in Norway, which is why I ran into you in Oslo. 00:49:43.580 --> 00:49:44.200 That's right. 00:49:44.200 --> 00:49:44.500 That's awesome. 00:49:44.500 --> 00:49:46.640 Although we also seem to run into each other in London. 00:49:46.640 --> 00:49:47.440 Yeah. 00:49:47.440 --> 00:49:48.660 Yeah, what do you guys do there? 00:49:48.660 --> 00:49:53.680 Well, 60 North, we're not terribly pigeonholed, but we do do a lot of Python work. 00:49:53.680 --> 00:49:59.540 We do consulting, training, some development of our own as well. 00:49:59.540 --> 00:50:03.400 We've made some courses for Pluralsight. 00:50:03.400 --> 00:50:08.940 So if you go to Pluralsight and you look for the Python training courses, we have the Python Fundamentals as our first course. 00:50:08.940 --> 00:50:13.840 And Python Beyond the Basics, which is sort of the next step, intermediate level, is there. 00:50:13.900 --> 00:50:18.820 And we're working on a third one, which is Advanced Python, I think, is the official name. 00:50:18.820 --> 00:50:20.620 And that will be out by the end of the year, hopefully. 00:50:20.620 --> 00:50:21.680 Okay, yeah. 00:50:21.680 --> 00:50:24.220 You and I, we're both very passionate about online courses. 00:50:24.220 --> 00:50:27.980 Tell me, what's in your intermediate and your advanced courses? 00:50:27.980 --> 00:50:32.340 Oh, I'm going to stretch my brain to remember exactly the contents of those courses. 00:50:32.340 --> 00:50:42.860 But I know the intermediate course, we start getting into things like decorators, class properties, some of the details of classes beyond just functions and methods. 00:50:42.860 --> 00:50:45.600 Here's how you define a class and add fields to it. 00:50:45.600 --> 00:50:46.380 Right, yeah. 00:50:46.380 --> 00:50:47.460 Getting beyond that. 00:50:47.460 --> 00:50:48.980 A couple of things that are beyond the basics. 00:50:48.980 --> 00:50:54.300 You'd be surprised at how many things there are that go into the basic course that are really, really basic. 00:50:54.440 --> 00:50:57.980 I mean, the course is quite long and doesn't really scratch the surface of Python. 00:50:57.980 --> 00:51:02.700 So anything like, like I mentioned, decorators or... 00:51:02.700 --> 00:51:04.780 Probably Lambda, Lambda expression type thing. 00:51:04.780 --> 00:51:10.460 I think Lambda's in there, context managers, implementing a lot of the Dunder magic methods. 00:51:10.460 --> 00:51:12.020 That kind of stuff is in the intermediate. 00:51:12.020 --> 00:51:17.420 And then the advanced classes where you start to get into things like what we talked about earlier, finders and loaders. 00:51:17.420 --> 00:51:20.060 Or you start getting into meta classes. 00:51:20.380 --> 00:51:29.980 Things that we classify to a degree as things you might do once a year instead of things that you do every day as a professional Python programmer. 00:51:29.980 --> 00:51:33.740 I mean, finders and loaders, I programmed Python for 20 years and never used it. 00:51:33.740 --> 00:51:35.520 But it's an interesting and important part of the language. 00:51:35.520 --> 00:51:37.140 So it needs to be in there somewhere. 00:51:37.140 --> 00:51:44.240 Yeah, and once you understand it, maybe you don't use it often, but knowing the mechanics helps you understand a lot of things often at that level. 00:51:44.240 --> 00:51:46.760 Yeah, and you know you have that in your pocket. 00:51:46.760 --> 00:51:53.000 And so that might be the most elegant solution for some particular problem you face rather than some horrible hack you would have to come up with otherwise. 00:51:53.000 --> 00:52:03.820 So the advanced stuff is for people who are using Python a lot and need to find the best solutions and really understand the inner workings of the Python runtime. 00:52:03.820 --> 00:52:04.440 Yeah, cool. 00:52:04.440 --> 00:52:09.820 So if you guys have a Pluralsight subscription, go over there and type in Python in the search box and you'll find Austin. 00:52:09.820 --> 00:52:10.160 Yeah. 00:52:10.160 --> 00:52:12.160 Yeah. 00:52:12.600 --> 00:52:13.000 Nice. 00:52:13.000 --> 00:52:14.220 And you also wrote some books, too. 00:52:14.220 --> 00:52:15.340 We do have some books. 00:52:15.340 --> 00:52:20.020 Yeah, we have – the books are based largely on the same material as the Pluralsight courses. 00:52:20.020 --> 00:52:24.080 And the first one is I think 90% done now. 00:52:24.080 --> 00:52:24.840 It's on LeanPub. 00:52:24.840 --> 00:52:26.500 It's called The Python Apprentice. 00:52:27.400 --> 00:52:36.500 The second and third books, The Python Journeyman and The Python Master, are in the works and will be published probably not this year but soon. 00:52:36.500 --> 00:52:43.180 And since they're on LeanPub, you can get the early version and we'll keep sending you updates as we make updates to the books. 00:52:43.180 --> 00:52:50.900 But if you prefer books, these are available as, I think, PDFs and Mobi's and EPUBs on the LeanPub site. 00:52:50.900 --> 00:52:51.840 Nice. 00:52:51.900 --> 00:52:52.960 And that's self-publishing, right? 00:52:52.960 --> 00:52:54.780 That is self-publishing, yes. 00:52:54.780 --> 00:52:55.580 Yeah, very cool. 00:52:55.580 --> 00:52:56.820 I'm a big fan of self-publishing. 00:52:56.820 --> 00:52:59.400 So I like to see when people are succeeding at that. 00:52:59.400 --> 00:52:59.780 That's great. 00:52:59.780 --> 00:53:03.820 Yeah, so I'll be sure to link to all those things in the show notes as well. 00:53:03.820 --> 00:53:04.400 Okay. 00:53:04.400 --> 00:53:05.060 That'd be great. 00:53:05.060 --> 00:53:05.960 Yeah, absolutely. 00:53:05.960 --> 00:53:06.660 Very interesting. 00:53:06.660 --> 00:53:07.480 Definitely cool. 00:53:07.480 --> 00:53:09.800 Two more questions before I let you go. 00:53:09.800 --> 00:53:12.540 What's your favorite PyPI package? 00:53:12.540 --> 00:53:16.780 I saw the other day there's over 80,000 distinct packages out there. 00:53:16.780 --> 00:53:17.740 That's an insane number. 00:53:17.740 --> 00:53:18.940 That's probably why people like PyPI. 00:53:18.940 --> 00:53:25.300 But there's got to be something that you've had exposure to that you want to share, like, oh, you should check this out. 00:53:25.300 --> 00:53:30.080 Well, this is going to feel like a bit of a cheat, but DocOpt. 00:53:30.080 --> 00:53:36.880 DocOpt is one that I, once I learned about it, I started using it on almost every project I use. 00:53:37.580 --> 00:53:41.820 But I know that it's not that well-known, at least not as well-known as I think it should be. 00:53:41.820 --> 00:53:45.280 So I'll just put a second vote in for DocOpt. 00:53:45.280 --> 00:53:51.460 For my money, that's the tool I keep going back to in PyPI every time. 00:53:51.460 --> 00:53:55.380 And it should be more widely known and more widely used because it's awesome. 00:53:55.380 --> 00:53:56.580 Yeah, that's awesome. 00:53:56.580 --> 00:54:03.340 I'll go ahead and throw one for Cosmic Ray in there for you because that's pretty awesome and very interesting to check out. 00:54:03.340 --> 00:54:04.020 Thanks. 00:54:04.020 --> 00:54:04.520 Thanks. 00:54:04.920 --> 00:54:10.220 And then you mentioned Emacs earlier, but if you're going to write some Python code, what do you typically open up? 00:54:10.220 --> 00:54:11.900 Well, the short answer is Emacs. 00:54:11.900 --> 00:54:16.000 I've been using Emacs for almost as long as I've been using Python, I think. 00:54:16.000 --> 00:54:18.600 And it's in my fingers to a degree. 00:54:18.600 --> 00:54:25.420 If I know that I'm working on just a dedicated Python project, then PyCharm is a wonderful IDE. 00:54:25.420 --> 00:54:30.440 And it's got a lot of powers that Emacs doesn't have when it comes to working with Python. 00:54:30.440 --> 00:54:33.420 Well, that Emacs doesn't have yet, I should say. 00:54:34.180 --> 00:54:36.960 But no, it's really great for just pure Python editing. 00:54:36.960 --> 00:54:41.800 I guess the reason I stick with Emacs is, well, stubbornness to a degree. 00:54:41.800 --> 00:54:43.800 I just, I'm old and don't want to change. 00:54:43.800 --> 00:54:48.600 But I'm also very often working on multiple languages at the same time in any given project. 00:54:48.600 --> 00:54:54.660 You know, everything from JavaScript to Python to, you know, L to whatever happens to be part of that project. 00:54:54.660 --> 00:54:57.220 And I find that Emacs makes it easier for me to do that. 00:54:57.220 --> 00:55:03.300 Or at least it has, it's not, it's the best of, it's the best of breed for that kind of work from what I can tell. 00:55:03.840 --> 00:55:07.560 And honestly, Python as an Emacs IDE is pretty good. 00:55:07.560 --> 00:55:11.760 You can do all sorts of fancy stuff in there if you want to spend the time to configure it. 00:55:11.760 --> 00:55:22.100 And if you use a package like, or a canned Emacs configuration like SpaceMax, you'll find that you get pretty sophisticated support for things like completion right out of the box. 00:55:22.100 --> 00:55:23.980 You know, you get Jedi support and things like that. 00:55:24.540 --> 00:55:32.520 So it's, I know that I try not to recommend Emacs to new people, to people getting new to Python because that adds a whole level of complexity. 00:55:32.520 --> 00:55:35.920 But Emacs as a way of life is, it's an interesting place to be. 00:55:35.920 --> 00:55:38.640 So you could do a lot worse as a programmer. 00:55:38.640 --> 00:55:43.100 So any final call to actions for all the listeners while you got the mic? 00:55:43.100 --> 00:55:45.600 Any more calls to action? 00:55:45.600 --> 00:55:47.720 Are you looking for contributors to your project? 00:55:47.720 --> 00:55:48.380 Certainly. 00:55:48.380 --> 00:55:48.940 Sounded like you were. 00:55:48.940 --> 00:55:54.080 Certainly Cosmic Ray could use some people who are willing to put in some work. 00:55:54.080 --> 00:56:01.620 We have, of course, the GitHub issues page where I keep track not just of defects, but also of, you know, the higher level issues that need to be done. 00:56:01.620 --> 00:56:12.760 I mentioned earlier that we have this pressing need for being able to embed exceptions and processing instructions in our code so that Cosmic Ray can not do certain kinds of mutations. 00:56:12.760 --> 00:56:16.860 And that's a big project that somebody might be able to take on. 00:56:17.560 --> 00:56:24.240 We have, I guess, the two other big topics I could think of are support for different kinds of modules. 00:56:24.240 --> 00:56:28.980 Right now, Cosmic Ray can only work against modules that are written in pure Python codes of .py files. 00:56:28.980 --> 00:56:32.440 But, of course, there are plenty of other exotic kinds of modules out there. 00:56:32.440 --> 00:56:37.260 So Cosmic Ray needs to either gracefully skip over those other kinds or learn how to process those. 00:56:37.260 --> 00:56:38.720 And there's no support for that right now. 00:56:38.900 --> 00:56:40.280 And that's a big limiting factor. 00:56:40.280 --> 00:56:45.900 And the other is, this is more of a researchy thing, but the integration with coverage testing that I talked about earlier. 00:56:45.900 --> 00:56:56.840 Being able to take output from, say, coverage.py and use that to determine how we can narrow down the scope of Cosmic Ray mutation testing runs and make it a more practical tool. 00:56:56.840 --> 00:57:02.840 But, really, it's, you know, go to the issues page on GitHub and look and you'll see the nature of the things that are going on. 00:57:02.840 --> 00:57:05.640 Yeah, that would be my call to action, I guess, for Cosmic Ray. 00:57:05.640 --> 00:57:06.880 All right, fantastic. 00:57:06.880 --> 00:57:09.460 I'll put the link to GitHub repo in the show notes. 00:57:09.460 --> 00:57:14.140 So, Austin, it's been really fun to talk about this idea of mutation testing. 00:57:14.140 --> 00:57:22.040 I think it's a really interesting evolution, if you will, of all the testing tools, right? 00:57:23.040 --> 00:57:32.120 I can see a place when this algorithm gets tuned and, like, the various optimizations you talked about get in there that this could be a big part of day-to-day work. 00:57:32.120 --> 00:57:32.480 It's cool. 00:57:32.480 --> 00:57:33.160 Cool. 00:57:33.160 --> 00:57:34.000 I'm glad you think that. 00:57:34.000 --> 00:57:36.660 And thanks for having me on the show to talk about it. 00:57:36.660 --> 00:57:39.520 It's something I really enjoy talking about in public. 00:57:39.520 --> 00:57:40.880 So, yeah. 00:57:40.880 --> 00:57:41.640 Yeah, you bet. 00:57:41.640 --> 00:57:42.320 You bet. 00:57:42.320 --> 00:57:43.160 Thanks for being on the show. 00:57:43.160 --> 00:57:44.300 And it was great to see you last week. 00:57:44.300 --> 00:57:44.840 Take care. 00:57:44.840 --> 00:57:45.900 It was great seeing you last week. 00:57:45.900 --> 00:57:46.180 All right. 00:57:46.180 --> 00:57:46.580 Bye, Mike. 00:57:46.580 --> 00:57:50.580 This has been another episode of Talk Python To Me. 00:57:50.580 --> 00:57:52.540 Today's guest was Austin Bingham. 00:57:52.820 --> 00:57:55.200 And this episode has been sponsored by Hired and SnapCI. 00:57:55.200 --> 00:57:57.100 Thank you guys for supporting the show. 00:57:57.100 --> 00:57:59.880 Hired wants to help you find your next big thing. 00:57:59.880 --> 00:58:08.360 Visit Hired.com slash Talk Python To Me to get five or more offers with salary and equity presented right up front and a special listener signing bonus of $2,000. 00:58:08.360 --> 00:58:12.260 SnapCI is modern, continuous integration and delivery. 00:58:12.260 --> 00:58:14.880 Build, test, and deploy your code directly from GitHub. 00:58:14.880 --> 00:58:18.120 All in your browser with debugging, Docker, and parallels included. 00:58:18.120 --> 00:58:21.200 Try them for free at Snap.CI slash Talk Python. 00:58:21.600 --> 00:58:23.760 Are you or a colleague trying to learn Python? 00:58:23.760 --> 00:58:28.520 Have you tried books and videos that left you bored by just covering topics point by point? 00:58:28.520 --> 00:58:36.500 Well, check out my online course, Python Jumpstart by Building 10 Apps at talkpython.fm/course to experience a more engaging way to learn Python. 00:58:37.160 --> 00:58:42.600 You can find the links from this episode at talkpython.fm/episodes slash show slash 63. 00:58:42.600 --> 00:58:44.980 Be sure to subscribe to the show. 00:58:44.980 --> 00:58:47.180 Open your favorite podcatcher and search for Python. 00:58:47.180 --> 00:58:48.420 We should be right at the top. 00:58:48.420 --> 00:58:57.720 You can also find the iTunes feed at /itunes, Google Play feed at /play, and direct RSS feed at /rss on talkpython.fm. 00:58:58.400 --> 00:59:02.780 Our theme music is Developers, Developers, Developers by Corey Smith, who goes by Smix. 00:59:02.780 --> 00:59:06.820 You can hear the entire song at talkpython.fm/music. 00:59:06.820 --> 00:59:08.880 This is your host, Michael Kennedy. 00:59:08.880 --> 00:59:10.160 Thanks so much for listening. 00:59:10.160 --> 00:59:11.340 I really appreciate it. 00:59:11.340 --> 00:59:13.500 Smix, let's get out of here. 00:59:17.000 --> 00:59:34.660 I'll see you next time. 00:59:34.660 --> 00:59:35.160 Bye.