#45: The Python Testing Column, Now a Thing Transcript

00:00 What is the role, the core purpose of writing tests for you application? Should you write more unit tests and fewer integration tests, or is it actually the other way around? You may have heard of the test pyramid with unit tests building the foundation. In this episode we talk about a variation on that theme called the test column. We talk about this and more with Brian Okken on this episode of Talk Python To Me.

00:00 [music]

00:00 Welcome to Talk Python to Me. A weekly podcast on Python, the language, the libraries, the ecosystem, and the personalities. This is your host, Michael Kennedy, follow me on twitter where I'm @mkennedy.

00:00 Keep up with the show and listen to past episodes at talkpython.fm, and follow the show on twitter via @talkpython.

00:00 This episode is brought to you by Hired and Snap CI. Thank them for supporting the show on twitter via @Hired_HQ and @snap_ci.

01:17 Michael: Hey everyone. I hope you are looking forward to an interesting conversation on testing software using Python. This week our guest Brian Okken is giving away 5 copies of his "Python Testing Frameworks" book. If you want to be in the running, be sure to visit talkpython.fm and enter your email address to become a friend of the show. Now, let's get right to the interview with Brian. Brian, welcome to the show.

01:37 Brian: Hey, thanks Michael.

01:39 Michael: Yeah, it's really great to have another podcaster and a friend and a fellow Portlander here all together on Talk Python show to talk about testing in Python.

01:49 Brian: I'm excited to be here.

01:51 Michael: Yeah, yeah. It's really cool. So, for the listeners out there who don't know, Brian is the host of Python Test Podcast which is a fairly new, but- how many episodes have you had, are on 9th or 10th?

02:06 Brian: Yeah, I can't remember I think I did number 9 recently.

02:09 Michael: Yeah, number 9, with Harry Percival, excellent, I listened that one, that's great I'll put a link to the podcast out there. So you spent a lot of time talking about testing and exploring sort of the whole spectrum or pyramid of testing in Python on your show.

02:25 Brian: Yeah, I really want to cover, actually the podcast covers everything I want to cover everything in software development and software testing.

02:35 Michael: So we are definitely going to spend a good amount of time talking about that. But let's start at the beginning- how did you get into programming, Python, that sort of thing?

02:42 Brian: My first introduction to programming was in high school. I took a Basic class that high school offered Basic and Pascal and I figured Basic has to be easier, because it's got Basic in the name. But, at the same time, I didn't really get into it too much then but at the same time I had a 3:01 and I spent a little time like typing in game listings from magazines like Lunar Lander was one of my favorites, and then I didn't do much more until college. I entered college intending to be an art and math major and about 2 or 3 years in switched to computer science and then finished up with a bachelor and masters at computer science.

03:30 Michael: That's cool. So where did Python come into the story?

03:32 Brian: Well, mostly the focus right after that was, I learned all sorts of languages, Perl, and C++ and others in college. I got a job at Hewlett Packard doing at first test systems and then embedded programming of instruments. And, that was all C++ and still is. And 4, or 5 years later, I was in a group that had a Python test framework for system testing the instrument, and that was probably in maybe 2002, and I've been using Python for testing instruments ever since then.

04:17 Michael: One thing that I think is interesting about that is a lot of times people feel like the programming language that the thing you are testing, the application, or whatever is built in, that's the language in which you have to write a test. So if you are writing the C++ app you have to fire up C++ unittesting. That's not really what you are doing, right, like you are actually testing code that maybe isn't Python, right?

04:43 Brian: Yeah, it's C++ and other languages inside, but the interfaces available, anything but the interface that is available from python you can use Python test frameworks to test them. And you know, I had frameworks that the company gave and didn't even think about it but yeah, it was probably 2010 when I started looking into other frameworks like unit test and pytest.

05:13 Michael: Yeah, ok so let's dig into a little bit of the type of stuff that you test that work and then we'll come back and talk about your podcast a little more. So, what exactly- you are talking about instruments, and physical things that you are testing and these are written in C++ or in like embedded system with C++ or C?

05:31 Brian: Yeah, it's C++, but these are, there is lots of levels of embedded programming there is embedded programming like your car, your phone or your watch or whatever, but these are big instruments, these are- I don't know, they look like stereo components but they've got many processors inside the box I work on right now runs Windows, so I am really writing a Windows application but I don't really do nay Windows stuff myself, but it's all written in C++ all of my code, yeah the user interface a lot of people use this for the boxes I work for are mainly used in production lines, and the production line control uses I guess a string based- it's called skippy but it's special language to control instruments with. Yeah, so that's how our users control the box so that's how we test them.

06:31 Michael: Is that like for like managing factories so like of you are building cars, like the machines, like put the door on whatever, like those types of devices, or what exactly are you thinking?

06:41 Brian: Well, ok, so the box I work on right now is called "the communications test box" one box tester, so it tests all the RF the transmit and receive abilities of mobile phone. SO, pretty much yeah, every phone that sells in the world has to go through the testing to make sure that since it's a transmitter it has to follow FCC rules and-

07:09 Michael: I see, so if I like left down in San Diego and worked at 7:13 maybe I would buy your company's product to test the phone internals that we are building, something like that?

07:19 Brian: Yeah, and anybody that's got any- we do not just cellular but also like wi-fi broadcasting so we do wi-fi testing and bluetooth and anything that transmits or receives the boxal test pretty much.

07:36 Michael: Ok, cool. And so to test this embedded C++ app basically, do you do some sort of C level interop with Python, or do you expose some kind of API that Python can understand or do you do the skippy API consider, what's the story there?

07:55 Brian: The magic that glues it all together is a library that is available on PyPy called PyVisa, so yeah, there is 8:07 and national instruments and other have these visa DLLs that you can use to interact to send skippy over lan and I use PyVisa to get at that, so it's really easy, you connect those app and you just have an object that you can send, write and read commands to. Works great.

08:31 Michael: Would you say that that kind of testing is a sore more holistic or is that more like unit testing- it 8:36 to me that my first guess is like you are kind of heading the outside so maybe an integration test type story, yeah?

08:44 Brian: Yeah, it's definitely like end to end test, I am testing our entire system but the kind of the, there is so we are looking at test frameworks, I am way more interested in how the fixture model is because we use setup and tear down to actually do things like move signals around, and in hit switch boxes. Definitely more of an end to end, but we do use ideas like if I am testing the measurement capabilities for instance it's difficult tot est that against a like a moving target, like an actual mobile so we'll have an arbitrary way form generator, generate a known signal and we test the receive side against the known signal. And then we can do the other end also we send our transmition to like a different instrument to test it against something else.

09:41 Michael: When you think of software architectures, you often hear people talk about things like dependency injection, and basically managing and decoupling the dependencies of your software so that you can test individual pieces, right?

09:56 Brian: Yeah. Definitely.

09:58 Michael: Yeah, but it sounds like your dependencies are more solid than maybe others, right, like you've got physical devises and like scientific information coming in and way forms and so- what kind of stuff do you do?

10:15 Brian: That part actually isn't that different, really, I mean because, like if you had like a big user database or something on a web application, the real database in the world is going to be a lot different than your like test data base, so that's similar to how our real signals are different form our test signals, the difficulty is the fuzyness, so we often don't really get because even with a pure signal, from a generator, we've got noise in the system, so you cant just get a number out of the measurement and say, "If it's 3,2 then it's correct, otherwise it's wrong," and almost all of our measurements are we've got tolerances around them.

11:00 Michael:That's really interesting. A lot of people write software that is basically deterministic in ways that are easy to access so I get a boolean back or I query for a user and the user is either there or not and they are an admin or they are regular user, right, these super clear distinctions. But when you get into sort of sciency spaces it gets really hard to actually just know that you are right, yeah, I mean, if you get back in number and it's off by a 1000 is that still the same, does that count as ok or not, that can be really challenging.

11:41 Brian: Yeah, but there is ways to do it, we've got- when we are testing the entire system it's difficult to tell because you know, if it's off by 10% it's hard to visually know but we can- the actual numbers that go through the system go through- there is- the different pieces are tested with different 12:06 so it all work out.

12:10 Michael: Yeah, that is an interesting point.

12:11 Brian: And I am actually not the guy that tests the box to make sure that it- if we say we are putting out like -10db that it's actually really -10db. That's like some other guy in manufacturing facility. I am mostly concerned with making sure all the software didn't mack things up, in the end I trust that the hardware is working correctly, and I just want to make sure that the software hasn't macked up the pristine hardware.

12:45 Michael: Right, so you assume that the hardware you've got is actually processing everything ok but you've got to gather it, save it, interpret it, all that kind of stuff, and so you kind of start at that level, right?

12:58 Brian: Yeah, and then the measurement data will return it back, I mean, we are sending back, we've got like thousands of different measurement points that we are sending back and making sure that we don't mack one of those up and that we don't put some array int he wrong place, that's a sort of stuff that I am really concerned with.

13:18 Michael: Yeah, that's interesting, one of my first jobs was working in the place called "ITracking Inc" and we would build these scientific visualization tools and sort of data analyses on top of ITrack in data right, people will dialation over time where you 13:35 looking, that kind of stuff. And we would end up doing really complicated math sort of pipelines on it, you know, 48 analyses, decomposition, those kinds of things, and you try to test the results of one of those things, you know, it's super hard so we basically would come up with a reference input and a reference output and go if the deviation from this graph to that graph is not too large we'll call the same right, because maybe they use something like Matlab to generate their reference results and you know, the way it deals with floating point numbers might be slightly different than the language we are using, something like that. It's interesting challenge that I think a lot of people building more standard business apps it just don't really get on your radar, right, because the user is either an admin or not, and you know a s14:24 done.

14:27 Brian: Yeah, I mean, actually people that have that sort of test job, I'm like, "What's so hard about this, just write the test and get on with life."

14:37 Michael: [laugh] Yeah, so ok, so let's look at some of the frameworks and get some of your thoughts on that, so let's say the majority of people listening know what unit testing is, what the goals are but I know a lot of my listeners are sort of scientists getting into programming or new programmers, so maybe give us just the elevator pitch what is unit testing, why do we care?

14:59 Brian: I'm not sure why we care about unit testing, actually.

15:05 Michael: Or software testing.

15:05 Brian: So the phrase unittest is one that's kind of a sticky point for me because the in extreme programming they use the term unit test and that just meant any developer test, anything written by developers, not written by the QA team, and in a lot of communities it's stuck with that, it is that's when a lot of people commonly, term a unittest is something written by developers. And in TDD community that often isn't the definition, the nit test is often the smallest possible like one function you test and one function at the time. And, I look at- so there is the most important is that the entire thing is tested, the entire software is tested, and unit testing is I think there is certain places where you want to go in and test I like to put test anywhere where I have a promise, if I am promising anybody other than myself than things are going to work right, we need tests there, so if I've got a library that other team members use, or other groups use, that needs tests around it, but anywhere there is interfaces, the interfaces between groups or between components, those are great places to put tests, and then of course the entire system function tests. And I think the focus on unit testing unfortunately takes some of the responsibility out of system level testing where I think it's unfortunate.

16:44 Michael: I think there is different scenarios where things like unit test matter more and other times they matter less. So, times I think they matter a lot, are when you have a lot of people all in the same general code base making changes, and you don't necessarily know everything that's changing, right, so the people are checking in changes and you are not necessarily, you are doing code reviews and stuff, but you are not necessarily like always aware like, "Oh they actually change this method that I was depending on and they didn't understand this like implicit contact I built around the return value here," something like that, right? Yeah, so I think it's really valuable there, and I think it's valuable for helping build apps where fewer people are working on- there's less turn but what are the things that I saw early 2000s in response to the extreme programming was people saw that they, they thought that they had to do 95% code coverage, everything is TDD, if you don't start that way it's bad software, you just can't write it, how could you possibly

17:52 Over time I saw a lot of people saying, "Well, we can't do that, we are in too much of a hurry," or whatever, and they have whatever excuse, a lot of times that wasn't totally unreasonable. They would basically say, "We can't match that level of commitment to testing, so we are not going to do any." And I think that's one of the places where it can actually, that message of like test everything can become negative, because if they had written like you would say the thing you were making a promise about, like if you are writing like stock trading software, the part that actually says, "buy or sell" there should probably be test there, the part that the app started probably doesn't need unit test, right, there is lots of parts of apps that are just they support some sort of core and that core is much more important that is tested I think. What do you think?

18:44 Brian: Yeah, totally, it's like I think you brought it up in one of your podcasts recently that it's the thing that's making you money, the reason why somebody is buying your thing, that's why you should test the hack out of it.

18:58 Michael:Yes, absolutely, like if that is your thing you built for the world, that little essence of it, you better test that, right. But there is as you know, you 19:08 it's like 20% of the code you write, there is 80% that isn't that.

19:15 Brian: And then, I like code coverage tools, but a lot of times I think that there instead of looking at the- there is two ways to get to 100% right, you can either write more tests or you can delete code, I think you should delete more code, if you've got a bunch of software that is not reachable from the user interface, if I can make some section of the code hit that piece of code with normal user input maybe it doesn't need to be there.

19:50 Michael: That's a really interesting point, yeah.

19:50 [music]

19:50 This episode is brought to you by Hired. Hired is a two-sided, curated marketplace that connects the world's knowledge workers to the best opportunities.

19:50 Each offer you receive has salary and equity presented right up front and you can view the offers to accept or reject them before you even talk to the company.

19:50 Typical candidates receive 5 or more offers in just the first week and there are no obligations ever.

19:50 Sounds awesome, doesn't it? Well did I mention the signing bonus? Everyone who accepts a job from Hired gets a $2,000 signing bonus. And, as Talk Python listeners, it get's way sweeter! Use the link hired.com/talkpythontome and Hired will double the signing bonus to $4,000!

19:50 Opportunity is knocking, visit hired.com/talkpythontome and answer the call.

19:50 [music]

20:50 Michael: I've spent more than once, I've been giving some big software projects and said, "Have a look at this and then we are going to need a have you help us at a feature or recommend some changes, and there will be like some method," and they are like, "what does this do, what does this have anything to do with this code, it doesn't seem to do anything," and I would like study the code and then, 2 hours later I am like, "it's never called, at all."

21:15 Brian: Yeah, somebody should have just deleted it.

21:18 Michael: They should have just deleted it and the world would have been a better place, but instead, you know, I wasted 2 hours, somebody else probably wasted 2 hours, and it just gets kicked down the road.

21:27 Brian: The other thing, one of the things that, I actually like I am writing more low level tests than I used to, I used to mostly focus on high level tests, I still do, but lower level tests do have their place, but I like the model of in test driven development of the red green refactor, but just don't forget the refactor part; so putting a test in place makes it so that you cannot refactor something then maybe that test isn't quite right, I don't want to restrict my ability to change my mind and redesign something, so...

22:06 Michael: I had brought up the perfect being the enemy of the good, sort of, a thing where people say, "If I can't have really good test coverage, I am not even going to start, forget this, it's too much work to do testing," right?

22:17 Brian: Yeah, well you see that a lot when a team like looks at their code and they get a new manager and somebody comes in, it's the same when you 22:28 a bunch of tests to this, but where do you start when you have like a ten years of legacy code, you can't just cover everything, so, yeah, you've got to just start somewhere.

22:38 Michael: I have a book recommendation actually, now that you bring that up, so Michael Feathers is a guy that in the early days of test driven development and extreme programming, all that sort of stuff, he was really active and he wrote a book called, "Working Effectively With Legacy Code" and it's given the time it was a little more C++ focused but it has some really amazing ideas of like how do I take this app, which is a million lines of code and zero tests, and start carving out little 23:13 epocs of code functionality that are supported by tests and ways you can do that that are safe and definitely recommended, people are dealing with like huge apps that are not yet tested and put together in a way that makes them basically untestable. Check out Michael Feathers book.

23:32 Brian: Ok.

23:32 Michael: Have you seen that one?

23:34 Brian: Definitely, I've got difference of opinion on that one, but obviously-

23:40 Michael: No, no, no, I want to hear it, tell me.

23:41 Brian: I think it just seems silly to try to add a bunch of unit tests to a bunch of existing code, I think it's way more profitable for your company to put functionality testing around it, and because like the end to end functionality testing of your system, if you know, like, you need to know, if you are going to refactor something you need to know that the end functionality is still working solid and if you have test in place to guarantee that, then what value is it to go in and add like thousands of unit tests-

24:18 Michael: Sure, I think that's a totally valid point, and I think it's been probably 12, 13 years since I've read that book, but I the idea was like, suppose you have a library you are calling and like you hit the outside method like some do your thing library and it give you and answer back, rather than trying to setup a whole bunch of tests inside there, the question was like how could I refactor some dependency or some bits, so maybe what you do is you just come over the bunch of inputs, save the outputs and go as long as those don't change, we are going to call it good, so some like really basic scaffolding, to say the thing kind of looks still ok.

25:03 Brian: Yeah, I mean, I think putting tests around the outside and then also looking at the different interfaces inside the system, and maybe sure, maybe adding some different interfaces that wren't there before to try to separate the design, but all that involves like a redesign and you are not going to guarantee that you are not breaking external functionality just by adding unit tests, it's just they don't test for that.

25:28 Michael: Yeah, that's right. I think that's probably interesting thing to explore. Can you talk when unit tests and the whole extreme programming origins of it and all that, but there is kind of what you refer to on your shows it's like a pyramid of testing capabilities or functionality. Can you go into that a bit?

25:47 Brian: Yeah, sure. Well, I don't really like the test pyramid, but if you do much reading about software testing you will run into the test pyramid. All right, so the test pyramid, if you do any reading on software testing you will ran across references to a test pyramid, so the idea is a foundation of unit tests and some integration tests, and very little functional tests, end to end tests.

26:18 Michael: Kind of like a food pyramid where like the dessert is at the top and you should have just a little tiny but, and that's the functional bit at the top?

26:24 Brian: Yeah. And, the reason is because supposedly functional system testing doesn't give you much benefit and they break and the brittle, and they are fragile, I just don't buy it, and what happens I think is developers see that and they go, "Oh, ok, focus on unit tests and leave the end to end test to QA." The big problem with that is a lot of teams do not have a QA anymore. So there is nobody doing end to end tests. Somebody's got to, so if there is no QA then it's your customers. And, I don't think that's the right person to do quality assurance. So I think that focusing on maybe a column of tests, it should be a column not a pyramid, do the tests where they make sense, you've got to have full behavior coverage, why is to make that your system works the way you promised it's going to work, and also like, especially with the continuous integration if you are relying solely on automated tests those have to be promises that you are trying to keep to the customer. And then, i don't really know what integration tests are, there is a lot of definitions for it but just the middle part and sometimes it's integration test means integrating with the outside world, so if you are integrating with the database library it's a test against that but then also some people think of integration tests as between two components or between two functions, and then unit tests are the low level while you are designing your code. I don't know, I think that you just need to use your own, I don’t think there is a rule, I think you've got to use your judgment, and put tests where it makes sense.

28:14 Michael: I really feel that that's good advice, right, the whole- let's make the top part of the pyramid a little fatter and make it more like a column. So, I think that makes a lot of sense because let's say you have a thousand unit tests actually I'm with you I don’t really know what integration is, but say you have a hundred of those and you have ten functional tests right, do you really need the 1000 unittests? or those 10 functional tests are enough maybe?

28:44 Brian: Well, a lot of the arguments for the a lot of the unit tests are to make sure that somebody else doesn't break your units or something, I just think functional testing needs to be done more, so...

28:59 Michael: You know where I think the origins of it are? I think it has to do with the TDD side of things, like it's very difficult to functional TDD.

29:09 Brian: I think it works great. Harry brings it up and it's been brought up by others, this kind of a double loop TDD where you where you start with the functional test and then while you are trying to fix that functional test that's failing you write a bunch of unit tests. I think that works good I just once you have the functional test working , do you need to keep the unit test- I don't know, it's kind of up to you. I think keep them if they are not, I think of a unit test as like scaffolding right, so if it helps you to build your software to put a bunch of scaffolding around it, great, but once it's built, I don't know some of the scaffolding can come down maybe.

29:57 Michael: Yeah, you are left for columns or something, right? Interesting. Yeah, I think that's a good way to look at it, I think one of the really big values that unit test provide and to some degree the functional test and integration test is they get you to think about how your function or object or API is used from the outside, sooner.

30:21 Brian: Yeah, definitely. And that's why I think focusing on interfaces if you are really building an interface that you need to have cleaned, then it needs to be changeable and tested, but say I'm right now in a handful of classes that all work together, and I am the main developer in this section of code, it's just going to be me, right, so I don't think I am going to put a unit test around every function because that means every time I want to refactor the system, I've got to change all the unit tests. As long as this subsystem works from the outside world, I think that's enough.

31:02 Michael: I think one of the big mistakes that people also make in addition to being letting like the perfect e the enemy to good and just saying, "Well, we can't do perfect so we are not starting." I think the other problem is that people write unit tests code and they different mental state or let's just say test code, because it's up and down the column that you are defining, so I think they write those in a different mental state, then the core of their application, like I've seen people just go to unit test and just copy paste, like 10 lines of code and I am like, "What are you doing," like, "Would you do that in normal code, of course you would not, you make that a function or some sort of object or something, right?" But, that makes it way more painful than refactor or evolve your code, when you've got 20 copies at the thing that you have to maintain.

31:57 Brian: Well, yeah, that's another reason why I think they've got to be careful with the whole like have as many unit tests as you can write in a day. Because, it's a code asset but it's also a wait around you, you've got to keep it all the time and you've got to change it, it's code maintenance, there's test code maintenance just as there is normal code maintenance.

32:18 Michael: Yeah, so I think you can structure your test code most people's test code can probably be refactored, to be way more maintainable, and so that's good, and people should do that, but like you say, more is not always better. I think if you think of what are the main use cases that a user is going to do with your application- if you can test those, chances are, all those little edge cases, that you are testing, if they were to go wrong, your main user stories or whatever, don't really work anymore, right? So do you have to write those little tests, that's an interesting question.

32:52 Brian: Well, I think that if a user, all the edge cases from a user's perspective form like an end user, those should be tested, I definitely think you have to test the edge cases, but the edge case of every function, I don't think so, if you've got like one of the examples I have given before is if I've got a username that goes through the system, and in some function that has to intercept that, takes a string then right, so do I need the test to make sure that string can handle a 1000 characters? Maybe not.

33:29 Michael: It's interesting to question these assumptions, about how many tests did you have, should they be in a pyramid that it's decreasing as you go up or should they be in kind of more flat and so on-

33:42 Brian: It's sort of theoretical right, because it just changes with every situation, and the test need to be an asset so that you can, they need to be such that you can trust that if the test pass your code is good. And whatever the tests have to be so that you can trust that you can check something in and it's good, because the test pass, that's where you need to be, whatever it takes to get to there.

34:08 Michael: Yeah, and related to that is if you have tests such that it takes 15 minutes to ask the question do my test pass, people are going to stop running them, and they just become stale, all right.

34:08 [music]

34:08 This episode is brought to you by SnapCI, the only hosted, cloud based continuous integration and delivery solution that offers the multi stage pipelines as a built in feature.

34:08 SnapCI is built upon best practices like automated build, testing before integration and it provides high visibility into who is doing what. Just connect Snap to your GitHub repo and it automatically builds the first pipeline for you, it's simple enough for those who are new to continuous integration, yet powerful enough to run dozens of parallel pipelines. More reliable and frequent releases, that's Snap.

34:08 For a free, no obligation 30 day trial just go to snap.ci/talkpython.

34:08 [music]

35:15 Brian: That's the other 35:24 with the lot of the unit test stuff, with the testing and isolation, so I don't even get that, so one of the things is if the test suite is slow, you are not going to run it, so we've want to try the speed up test suites. But, one of the ways to do that is to isolate every single function and every single test so that the test doesn't rely on it, this function doesn't rely on anything else, but then, if I've got such isolated tests, why am I running the whole suite? Just run the tests over the thing that I am testing.

35:58 Michael: Right, because if you are not testing the integration then you are doing like 95% rework that theoretically, doesn't affect you, right?

36:06 Brian: Right, if I am writing code that is modular and everything, if like the stock example, if I want to check and see if the buy button works, I don't need to test to make sure that that function that I changed didn't break the printing function.

36:24 Michael: Yeah, the about page shouldn't fail when I change the trading engine, right?

36:28 Brian: Yeah, exactly.

36:30 Michael: So one thing that you talked about on I think it was show 8 if I recall, of your podcast was, the theme was like writing software is like nothing else. And I found that really interesting and it really resonated with me, not necessarily on a testing level, but on a software engineering level, like people say people typically or not skilled in the art of writing software, will come and say, "Hey look, you are tidal software engineer, we have other types of engineers, they build like bridges, they can tell us within 10% accuracy of not just the cost of the project, but the duration of the project and how many people need to be on it. Why can't you do that with software?" And I think you are sort of a, "software is like nothing else" was really, I think that was neat.

37:29 Brian: I appreciate that, I like that as well that notion that it's like engineering, or it's like architecture, or it's like whatever, it isn't, it's its own thing, it's hard to define what it's like. And now as I'm in a management role, I've got to do some of those things, I've go to come up with testaments, and I'm often an evil guy that says, "Well, how long is this going to take, are you going to be done next week?" And I know the answer is "nobody knows", but you know, you've got to ask those questions anyway, so...

38:01 Michael: Yeah. I read a book by somebody and I totally forgot all the details, so... I think it's lost, it was talking about this and it said, people often refer to software engineering and I actually personally didn't really like the term software engineer, I call myself a software developer because I think it's I don't want to put it too near architecture or engineering in people's mind, but it was saying, one of the fundamental differences between like a civil engineer that build a bridge and somebody that builds a piece of rich and interesting software is that you have to build the bridge over and over and over; if I am going to build a bridge across this river, I am going to build it here. And then, when you need another bridge, it's about like that bridge, but it goes across this river and it's just this much wider, and so if this was software you would just like take the bridge and change a setting and move it or like you would make a copy of it, but you can't copy bridges. So you rebuild them, over and over and over, and in software, if a thing already exists, you just use it and you copy it and you incorporate it as a library. If you are writing something, often, I know there is a lot of wasteful reuse or rewriting, but generally in software, if you are writing software, it's because it doesn't exist, it hasn't been done before. And that is what really gave me a different perspective. What do you think about that?

39:29 Brian: Yeah, well there is a lot not admitted here that I run into as well. And, it takes a certain kind of person that is willing to dog through libraries and see if it applies to your problem right now, but yeah, the thing that I think software is most like, actually is probably like technical writing or just any kind of creative writing, because like in literature and lots of form of writing, we often talk about the first draft is- I mean, we talk about as the draft, we have a first draft, and that's not even usually the first thing, usually you start with notes, and then you go to a first draft, and then you have many drafts before you have something that's ready for somebody else to read. That's probably closer to software because we in the iterative approach except for, there is a lot of software engineers or software developers that think they are like awesome, and that their code is perfect, all the time, and that's just ridiculous, you need to be willing to throw stuff away and redo it.

40:39 Michael: Yeah, throwing away the code that you worked on is hard to do.

40:44 Brian: It's fun though, you should do it more often. That's why I like to put tests around the things that I am not willing to throw away and those are the interfaces that I have promises to. So, that's really where I like the creative freedom of software to be able to take the entire chunk of code even a week before ship date and say, "this stuff is just rotten", we need to just rewrite it. Instead of trying to figure out what it does, let's just re-implement it.

41:14 Michael: Certainly I spent hours trying to figure out what a piece of software does, and then more time trying to understand well can it be adapted to this situation, and is it sufficiently generalized, and we are not even talking about like is it thread safe and all these other like weird edge cases you might have to care about, right? By the time you are done with that exercise, it was probably like too 41:38 to write that functionality from scratch, right? Possibly, you know, I mean not always, but possibly.

41:46 Brian: And then, other things where like I've spent a week writing something, but I think I like to try to think of the first time writing a piece of software as learning about what the problem is, and so you don't have to relearn the problem, you already know it, so if you even like deleted all the files and started over, it would take you a day for the second time. And the third time maybe an hour. Just don't be afraid to throw things away.

42:13 Michael: That's interesting, because you think I spent a week working on this but it's understanding the problem like you say, and like thinking through the trade offs and a lot of that probably gets better with time as you understand it better and not worse.

42:27 Brian: Now, the place where people really want to throw code away is other people's code, and that's where I think people should like hold off a little bit and like the Michael Feathers book that you brought up, around that time frame there was a lot of talk about what to do with all this horrible legacy code, and I never think of legacy code as a bad word, it was a bad word for a while I don't know if it still is, but I think of legacy code as this stuff that's paying y salary. So I really like legacy code, so I treat it like that, I treat it as the thing that's giving me money right now.

43:04 Michael: Legacy code, that's still being used by definition as something that was super valuable and useful, right, or I guess somebody left some old website running, but generally speaking, it's if it was not useful, it would have been turned off or somehow outdated, right?

43:19 Brian: Yeah, well, I mean I am thinking more along the lines of my job right now, we've got, I mean test instruments got thousands of lines of code and it goes back like I started on the code base that I am working on now, maybe 5 years ago, but it had 10 years of history before I got there, so. This code is old, even if the people are still around that made the decisions, they don't remember why they made certain decisions so if there is some weird switch case statement you've got to try to find out, and that's more of the hard part, let's try to find out was this an intentional decision or just a way of the developer.

44:02 Michael: How many different types of version control has it gone through, like was it CVS in the beginning or something like this, or is it still on the same one.

44:10 Brian: I think it did switch, I'm not sure what it was initially because there are some like the time where they would put like the version string in the heather of the file, those are still around, but they are not filled out anymore. Right now we are actually using clear case which I am not really thrilled about but it's not terrible.

44:33 Michael: One of the things that you have going on at pythontesting.net is you have a book that sort of compares the 3 major frameworks that people might use to their called unit testing frameworks but like you sort of pointed out you can choose the level of aggregation or isolation you want to get into to sort of run the spectrum right?

44:53 Brian: Yeah, thanks for bringing that up.

44:54 Michael: Yeah, what's the story with your book?

44:55 Brian: It was around 2010, I found myself without a test framework and I looked at those 3, I looked at unittest, nose and pytest and I looked at them like what I mean I searched on the web and tries to read things about them, but I couldn't tell whether they were going to be useful for my job or not and I needed something right away. So I wrote my own but and while I was maintaining my own test executive, for work, I started researching these and I wanted to try to play with them and create something so that nobody else had to go through what I did if trying to track down what's true information, and what's old and what's new. So, I went through like a simple math problem and a little string conversion problem and used those 3 frameworks on the same problem so people can see apples to apples, compare the 3 frameworks. And, then extended just you know, the basics of getting a test running, through to how fixtures work and it's really the mechanics and syntax of test writing and I have written about other things but the book is a basically a collection of those mechanics posts so that people can come up to the speed really quickly on any of those frameworks and compare them as well.

46:29 Michael: Yeah, that sounds really helpful if you are getting started and you are not really sure what to start with. So do you pick a winner in the book?

46:35 Brian: You know, I didn't in the book, but I think pytest is a clear winner for me because of the fixture mechanism that is built in, plus it's a super set, pytest really does everything that the others do and more, there is definitely applications and when I got into it I expected to pick one and like not like the others, I'm not there yet, I think all 3 of those are reasonable. The most reasonable is probably unit test and pytest, those are the two-

47:10 Michael: Yeah, unit test is probably the most like straightforward.

47:15 Brian: I don't think it's straightforward, I think it's the most confusing to start with, because it's this x unit style that came from Java that you have to derive from a test case class, and also they discourage you from using straight asserts, you should use these assert helper functions, and I think there is more to memorize, if you just start now pytest is a lot easier. And I don't think anybody new to this should use nose, because it's not really supported by a very- I don't even know if it is supported anymore, there has been a little bit of effort on it but not much but I included writing about it because there is quite a few people that still use it, and if it's not your decision and it's your job to use it, then I want to have e reference for people to be able to come up to speed quickly. And I would like to write a follow up book, so the book I have now, it's like I said, it's a $5 collection of the whole bunch of posts but I read a lot on the Kindle, so I wanted to make sure that those are available on for somebody to be able to read on an e reader or something.

48:35 Michael: Yeah, nice, and I will be sure to put a link to the book and what not in the show notes.

48:40 Brian: But I do want to write a follow up that instead of just a collection of posts is actually in a technical book format and walks through stuff and it doesn't repeat things too much, and I don't know if I am going to include nose, I think it would be sufficient to just include unit test and pytets in the discussion.

49:00 Michael: Yeah, ok, so you know, six years ago maybe the space looked differently, and now you are like all right, pytest is kind of the clear winner in some ways and then well, unit test is a built in so people are choosing it because it's a built in.

49:15 Brian: I mean, it's also for a lot of like algorithmic things like the things I do, that are heavy on fixture use, pytest is a clear winner. If you got a whole bunch of tight algorithms and test speed really is the overhead of pytest isn't very much, but the overhead of unit test is less. So, if test speed really is- if you are trying to catch every milisecond out of your test suite, unit test will be a little faster.

49:45 Michael: Right, ok, good to know.

49:46 Brian: I'll probably get a bunch of hate mail from the pytest people, after this, but-

49:51 Michael: Oh yes, I think we set ourselves out to receive a lot of feedback on various areas but-

49:58 Brian: There is a lot of respect between the people that are developing unit tests and the people that are developing pytest because there is 2 ways to solve the same problem and they learn from each other and I think having both around is a very good thing.

50:14 Michael: Yeah, absolutely. All right, few more things I want to talk to you before I let you go. Tell us a little bit about your podcast, it was kind of a follow on to the book, right, you wrote the book and you got the website that's doing good and you were like, "let's make this more of a weekly semi weekly sort of conversation" right?

50:35 Brian: Right, so the intent was actually just start off weekly and get more like maybe a couple a week, but it's I am really trying to do one a week but it's often once a month. I want to cover more topics than I can write blog post about and so that's the real reason why I'm doing the podcast. It's to try to cover more topics and also things that I think somebody wouldn't necessarily read about, like I did an episode on the waterfall model that I think it's probably an interesting thing to listen to but somebody is probably not going to go and read about the history of the waterfall model. So, I have noticed that a lot of my audience is coming from developers and from testers, and I know that from my own experience as a developer that there is huge gaps in my education when it comes to testing, so a developer used gets the introduction to testing as to talk about test driven development and the writing a test part- we get a statement that says first write a failing test but no information of how to do that or what the test should look like and then on the other end, you have got testers that come maybe from a different kind of the background and they don't have a lot of the skills and education that developers had, but they are expected to write automated test now. I really want on both sides, I want to fill that education gaps in both testers and developers to fill all those gaps in. And I know that people are busy they have already got a ton of stuff to do so I thought an audio format would be a good way to do that so people can get a little bit on their way to work, or whatever.

52:19 Michael: Yeah, and it's definitely a different type of interaction than a blog or a book, right, even different than an audio book, you can just fire up your podcast while you are driving, biking walking, doing the dishes, whatever, I really like that format, obviously. I spend a lot of time on it, right.

52:39 Brian: Yeah, and I also I was finding that I wasn't writing as much as I wanted to and I thought maybe I could and I have less time now, as I go into a manager role, but I thought maybe it would be easier to do a podcast than it would be to write, but I have found out that it actually takes more time then writing.

52:58 Michael: It's a lot of work, but it's also a lot of fun, right, to hear about from everybody, so I just love it.

53:03 Brian: And, one of the things I was worried about that my- since I am not a web developer I thought that a lot of the audience is from web development community, but I thought maybe that would be a hindrance, but when I was talking to Harry, I think there is a lot of similarities to how web systems go together and complex embedded systems, so I think it all works out.

53:25 Michael: Yeah, I would say so and I think that diversity of viewpoints, it's super helpful, like not everybody is writing straight up web apps, so...

53:33 Brian: Plus I heard from a lot of people that I would never have heard from otherwise.

53:38 Michael: Yeah, that's really cool, so I am really glad to see it going well for you, that's awesome. I know we spoke just a little bit before you got started, so it's cool. Good to see you going. And, speaking of that, we are both from Portland and something awesome is happening in Portland in June, right?

53:54 Brian: Yeah, PyCon is here.

53:57 Michael: Yeeey, PyCon comes to our home town. I happen to be living in Germany for a year so I am flying like 12 hours to get there, but none the less, I an sill going to go, it's going to be awesome. Are you going?

54:07 Brian: Yeah, definitely. And I put, actually I have submitted a 4 proposals, 3 talks and one tutorial. I don't know if I will get any but we’ll see.

54:19 Michael: Very nice, that's awesome. So hopefully, you will get accepted, that would be great. So if people are going, and I am sure a lot of listeners are going to go, feel free to reach out to me and I am sure Brian will be interested in meeting people that want to chat as well, so...

54:33 Brian: Yeah. One of the things we talked about was giving away some copies of my book-

54:36 Michael: Yeah, so if you go to talkpython.fm and at the top it says "friends of the show" as long as you are subscribed to that list, all it takes is an email address we will, I will I suppose, only one of us will do this, randomly choose 5 copies. So the week that this show comes out I will randomly choose 5 listeners from the friends of the show and we will be shipping you e-books of Brian's Python Testing book.

55:07 Brian: Yeah, unfortunately it's got a really long title that I don't even remember.

55:12 Michael: We'll ship you the book about testing with the really long title. Now, we will definitely send you an e-book copy, so sign up and you will be int he running. Thanks for doing that Brian, it's awesome. All right, parting thoughts, I always ask my guests if you are going to wrote some Python code, what editor do you open up?

55:31 Brian: I use Sublime, but I use it for everything, not just Python code.

55:36 Michael: Sublime is good. Very nice. And, PyPi packages, what ones are your favorite, or people maybe do not know about and they should?

55:42 Brian: Well, this is probably, nobody is going to care about this, but my favorite is PyVisa because I couldn't do my job without PyVisa.

55:49 Michael: Nice, nice, yeah, so PyVisa for talking to the machines. That's cool, and then maybe I'll throw in a pytest as well, right, because that's not built in.

55:58 Brian: Oh right, Yeah I guess I should have brought that up.

56:01 Michael: Perhaps. It's all good, they all go together.

56:03 Brian: The one of the things, because it's not a package but I would like to do a shout out to the Anaconda team or whoever does that, because that's the distribution that I encourage people to use.

56:16 Michael: They are doing really amazing stuff, yeah, that is Travis Oliphant and the Continuum guys, they were on show #34 and they definitely make Python in general more accessible especially on Windows, so very cool.

56:28 Brian: Yeah, it's actually that show that got me to download the Anaconda package and try it out, so...

56:32 Michael: Oh nice, very cool. So, final call to action before we say goodbye? Subscribe to your podcast, how do they do it?

56:40 Brian: Oh yeah, on the pythontesting.net/podcast there is a bunch of information there and I also got, I send people information, I've got a little email newsletter that goes out whenever I post anything new, you can reach me on Twitter, at @brianokken, or the podcast is @testpodcast.

57:03 Michael: Awesome, and all that is going in the show notes. Brian, this has been really fun and it's interesting to get different opinions on testing, so thanks so much for sharing yours.

57:11 Brian: Thanks a lot for having me on.

57:14 Michael: You bet, talk to you later.

57:14 This has been another episode of Talk Python To Me.

57:14 Today's guest was Brian Okken and this episode has been sponsored by Hired and SnapCI. Thank you guys for supporting the show!

57:14 Hired wants to help you find your next big thing. Visit hired.com/talkpythontome to get 5 or more offers with salary and equity right up front and a special listener signing bonus of $2,000 USD.

57:14 Snap CI is modern continuous integration and delivery. Build, test, and deploy your code directly from github, all in your browser with debugging, docker, and parallelism included. Try them for free at snap.ci/talkpython

57:14 You can find the links from the show at talkpython.fm/episodes/show/45

57:14 Be sure to subscribe to the show. Open your favorite podcatcher and search for Python. We should be right at the top. You can also find the iTunes and direct RSS feeds in the footer on the website.

57:14 Our theme music is Developers Developers Developers by Cory Smith, who goes by Smixx. You can hear the entire song on our website.

57:14 Don't forget to check out the podcast T-shirt at talkpython.fm/shirt and share your love for Python with the world.

57:14 This is your host, Michael Kennedy. Thanks for listening!

57:14 Smixx, take us out of here.