#141: Python tricks Transcript

00:00 Michael Kennedy: How many Python developers do you know that learned Python quickly but then plateaued pretty quickly as well? Maybe this is someone you work with, or maybe it's even you. Python's clean and simple syntax can mean it's easy to learn but hard to master. After all, if you learned it in a week, what else is there to this language? How much more do you need to dig into it? Well, plenty, and Dan Bader is here to share his very popular Python Tricks with us. You'll learn to look deeper for more than just the how but the why and when of many of Python's more subtle features. This is Talk Python to Me, episode 141, recorded December 4, 2017. Welcome to Talk Python to Me, a weekly podcast on Python: the language, the libraries, the ecosystem, and the personalities. This is your host, Michael Kennedy. Follow me on Twitter where I'm @mkennedy. Keep up with the show and listen to past episodes at talkPython.fm, and follow the show on Twitter via @talkpython. This episode is brought to you by Linode and Rollbar. Thank them both for supporting the show by checking out what they have to offer during their segments. Hey, everyone, before we get to the conversation with Dan about his Python tricks, I want to talk about Python Bytes for a second. I know many of you who listen to Talk Python also listen to my other podcast, Python Bytes, but many of you don't, and it's been a while since I've talked about it on the show. So if you're looking for almost the opposite of Talk Python; instead of long-form deep conversations on one topic, you want to just catch up on the news, think newsletter in audio form for the Python space, head over to Pythonbytes.fm and subscribe to the short, news-focused 15-minute weekly podcast I do with Brian Okken. I hope you guys check it out, and I hope you're enjoying both of the podcasts. Now, let's chat with Dan. Dan, welcome back to Talk Python.

02:02 Dan Bader: Hey, thanks, Mike. Thanks for having me on the show again.

02:05 Michael Kennedy: Yeah, I love to talk with you about technology and programming, so I'm really excited to do it once again. You were previously on the show for our panel with Anthony and Ronald about contributing to open source, and that was really a well-received episode. That was 132.

02:19 Dan Bader: Cool, yeah. That was a really fun experience, and I really liked the panel format. I listen to your show, and those are some of my favorite episodes, when you bring on a bunch of people, and it's just, you know, the amount of information you can soak up so quickly. Yeah, it's just great.

02:33 Michael Kennedy: Awesome. Thanks. The next one that I have scheduled, I don't know when it's actually going to air, like four or five episodes out, is a catch-up for Python and machine learning at the Large Hadron Collider and in particle physics. So that should be really fun.

02:46 Dan Bader: That sounds intense.

02:47 Michael Kennedy: It's going to be intense. That means lots of research for me. So that'll be awesome. All right. So, let's talk about not contributing to open source so much. Let's talk about making our Python better with Python tricks. So that's what we're going to talk about this week. You know, I know you talked a little bit about it in 132, Episode 132, but it was the panel format, so you didn't go into it too much. So let's talk just quickly kind of like how you got into programming and what you do day-to-day before we get to your tricks.

03:15 Dan Bader: Sounds good, yeah. So, well, I guess how I got into programming was when I finally managed to convince my parents to buy a used, old dingy Commodore 64 for me. And, you know, that was the end of the home computing era, I guess, and those machines all booted into some kind of interpreter directly so you wouldn't, you know, you don't get like a graphical desktop or anything. You just get BASIC, the programming language.

03:38 Michael Kennedy: Yeah, that was crazy. 'Cause you would like, you couldn't just interact with the computer. You had to like, converse with it to get it to do things. It was interesting.

03:47 Dan Bader: Exactly. It was like a glorified calculator. You just turned it on. It booted instantly, and then you were just sitting there like ah, okay. I'm getting this blue screen. What am I going to do now? It really forced you to pick up some basic programming skills to do anything. You know, if you wanted to play a game, well, tough luck. You had to actually figure out like, how to load the game from disk and then run it, or, you know, actually type it in.

04:13 Michael Kennedy: Yeah, it was like a load command and stuff to actually do it, right?

04:16 Dan Bader: Exactly, yeah. The load eight, comma, one, or something. It's like the drive ID. And I think I'm kind of fortunate that I caught the tail-end of that, because it was already like the PC era and like Windows 3.11 or something, but because my parents were really, really against computers and technology in some way, the only thing I managed to convince them was to get this, you know, used old Commodore 64. And I got it from this guy, and it came with like stacks of disks and notes this guy took. And so it was just a treasure trove for me, and, you know, it hooked up directly to your TV Screen, and it was just, yeah, this amazing like miracle machine, and that's how I got into programming.

04:57 Michael Kennedy: Nice. And then how'd you get over to Python?

04:58 Dan Bader: Oh yeah, that's a long, that's a long jump, a big leap there. Right? I think BASIC, programming in BASIC on the Commodore 64, I think it just kindled some kind of passion in me, and eventually I went on and got a bachelor's and master's degree in computer science, and just wanted to become a professional programmer. At some point in my university days, I think I was on a ski trip with some friends, and one person, he brought a book, a Python programming book, one of these like learn Python in 48 hours, one of those that never work out that way, and this was the first time I really...

05:33 Michael Kennedy: Was that the Sam's learn Python on a ski vacation or your money back guarantee book?

05:41 Dan Bader: I should ask for my money back.

05:42 Michael Kennedy: Remember those Sam's books? So, awesome. So you had the book, and you had some kind of down time after your skiing to chill, flip through it?

05:49 Dan Bader: Yeah, it was a very geeky skiing trip. But this was the first time when I saw the actual language, you know, just learned a little bit more about it, because my friend, he actually hadn't used Python either, but it just seemed like an interesting new language. Or I guess it wasn't new at the time, but for us, it was new. And so I just really fell in love with the way the language looked. It just seemed like a perfect blend of, well, this looks really appealing, almost like pseudo code, and in some way, like really poetic and nice, but also really powerful, where it's not just your basic or BASIC programming language, but it's actually something where you can take it to places and you can build full-blown applications. And, so, yeah, that's how it started.

06:30 Michael Kennedy: That's awesome. Yeah, there's plenty of simple programming languages, but they often have like a ceiling, where it's like, oh, you would build this and this with it, but you wouldn't really build anything bigger than that, right, where it's pretty sweet that Python generally doesn't fall into that category.

06:43 Dan Bader: Yeah, totally. I think it was you, you called it a full-spectrum language at some point, or like on that interview that we did on my blog, and this is, I love this. You know, I bring this up all the time when people ask me about Python, because yeah, it is really a full-spectrum language, where someone can dive in, and they can learn the basics of Python and know enough to be dangerous, and then you have giant, very, very complex systems built in this language, and it can kind of scale that whole gamut, and I think that's just really cool.

07:12 Michael Kennedy: Yeah, it's amazing. What I really like about that spectrum is a lot of languages make you take the stuff you need for the advanced type of large applications. You've got to use that syntax and that structure and those design patterns right from the start, like Java or C#, static main void, assign a program class. You're like, whoa, I just want to like, do a few things. What's all this about? Whereas Python, as you need the features, you layer them in, but they're not required until you need them, which is I think a part of the magic.

07:40 Dan Bader: That's a huge part of the appeal of Python that I see, you know, when people use Python and learn about Python, but it's also kind of the dirty little secret where it's very easy to kind of get stuck at that beginner level and you're like, well, okay, so how would I actually go to the point where I could build, I don't know, something like the Instagram backend or even, you know, think about it?

07:59 Michael Kennedy: Yeah, absolutely. And that is one of the curses of its sort of success or its features, is that it's, a lot of people quickly get into it, and they just get comfortable, and they're just like, well, I just sort of, I knew C or I knew JavaScript, so now I know Python. I just don't do semicolons or curly braces, right? That's okay, and it works, but you know, I think our topic today is really about understanding like, where people get stuck and then going, okay, these are actually how you should go farther and do it correctly or pythonically or whatever, right?

08:31 Dan Bader: Yeah, pretty, pretty much. I mean, this is, I feel like this is the biggest challenge in taking the next step with Python, like how, you know, what does it even mean to write Python in code, and how do you get there? Because it's such an opaque concept, and yeah, whatever can be done there, I think it's going to make people's lives easier and it's going to make them more successful as developers.

08:50 Michael Kennedy: Yeah, absolutely. So let's talk about your book, Python Tricks. It's a buffet of awesome Python features, which is really nice, and you released it a little while ago in a digital-only version on your site, dbader.org, right? Right URL?

09:07 Dan Bader: That's right, yeah.

09:08 Michael Kennedy: Yeah. And you also released it on Amazon, and it kind of went a little bit crazy on Amazon. Well done, man.

09:15 Dan Bader: Thank you.

09:15 Michael Kennedy: How did it do? Like, right now, if I'm looking here, it's number 21 in all of the books in Python, like, that's amazing for such a new book. But it was much higher, right?

09:26 Dan Bader: Yeah, so crazy enough, it actually hit the number-one spot in Python programming and in programming languages in general. At some point, it was book, it ranked number 250 in Amazon sales rank across all books on Amazon. And I actually, you know, I took a screenshot of that, and this is like...

09:47 Michael Kennedy: Printed it out and put it on the wall in a frame.

09:50 Dan Bader: It was like visible proof to my in-laws that I was just, I was not, not just unemployed and addicted to the internet. It was like, here! I'm an author on the internet!

10:02 Michael Kennedy: This is what I've been doing in my office! I promise you I have a job!

10:06 Dan Bader: Yeah, it was a cool experience.

10:07 Michael Kennedy: That's really cool. Congratulations.

10:09 Dan Bader: Thank you.

10:10 Michael Kennedy: Yeah, yeah. And so it's still going strong. Like, it's number 29 in web programming and stuff right now. Very cool. So we'll definitely link to your book from the show. So maybe give us the quick elevator pitch. I mean, we've kind of been building up to it, but what's the idea behind the book, and what do you talk about in there?

10:27 Dan Bader: Well, like you said, the subtitle for the book is A Buffet of Awesome Python Features. And so basically, the book is a collection of actionable tips that will help you write clean, professional, and developer style quote-unquote Python code. And, you know, the feeling that I want to create for my readers is not like they're sitting through an advanced computer science lecture where I'm throwing a bunch of jargon at you, but more like we're on the same development team. You know, we're sitting together. You maybe know the basics of Python, and I want to help you get up to speed as quickly as possible and help you grasp some of the intermediate and more advanced Python topics that often seem weird and opaque, things like decorators, first-class functions, what's the difference between an iterator and a generator, and generator expressions, and list comprehensions, and just some of the really useful things in Python that border on the arcane, but they're really not, if you approach 'em from the right perspective.

11:26 Michael Kennedy: Sure, there's a lot of these things that, you know, I feel like Python is, you can learn the language really quick, but it takes a long time to really master it. And part of that mastery is like discovering these little trade-offs, like I could use a tuple, but oh, by the way, did you know there's a name tuple, and did you know in Python 3.6, there's a new version of a name tuple that is more flexible but similar in performance. And like, oh, wait, there's a new name tuple? What is that, right? You don't really easily fall into that discovery or things like, yeah, you can list list as a queue, but it's like a thousand times slower than if you use this other proper thing for queuing. Oh, but don't use that for multithreading. Use this other one for parallelism, and things like that, right? Those types of things are somewhat arcane, but they're not easily discoverable, I don't feel.

12:13 Dan Bader: Exactly. You know, where it's, a lot of times, like, when someone has some experience with other programming languages and they're switching to Python, it can be really hard to understand how these different concepts map to Python, because I guess Python uses a very like, human-friendly naming style for a lot of things. You know, like you just mentioned the queue and list example, but if you're coming from let's say a Java background, how does a list map to what I know from the Java world? You know, what kind of time complexity guarantees does it make, and kind of, you know, touching on some of these things as well, like how can you take some existing algorithm and actually bring it over to Python and then make sure, you know, it doesn't just look pretty, but it actually also really works and is fast enough?

12:55 Michael Kennedy: Absolutely. And one of the things, even more than like, lists versus say, array lists in Java, is array. Like, there's not really like the traditional C-style array in Python in the built-in language sense. So there's the Array.array class, but there's not the like, square bracket means something different than list, right? Like, those are actually the same thing. That gets confusing. But I think we should probably just start going through some of your individual tricks that you thought are particularly like, noteworthy in at least a radio format.

13:28 Dan Bader: That'll be interesting.

13:30 Michael Kennedy: Yeah. So we'll try to not talk too much about code, but, you know, cover some of the ideas behind it, all right? So actually, you know, the first one, you said let's start with something simple, and I totally agree that it's a good place to start. You talked about comma placement for dictionaries and lists and just like, structuring those for, basically for both avoiding errors and making them source code control friendly. What's the story there?

13:55 Dan Bader: It's hard with sort of, this being a pet peeve of mine. Like, if you use lists or any of the other building collection classes like a dictionary or a set, and you're trying to define a constant in your code. Like, let's say you know a list of names or a set of some other objects. If those get too long and they don't fit into a line, like, usually, according to PEP 8, we're spacing them out and we're kind of spreading them out across line breaks. Now, when you do that, this usually looks pretty good, but then when you add an element to the end of the list, you've got to make sure you have your trailing comma in that list. Because if there's no trailing comma, and you're working with strings, for example, see Python, the interpreter, or the parser is actually going to mash two consecutive strings together into one. So you're going to end up with a completely different value, and it's not an interpreter error or parser error or syntax error. But this, you actually get the wrong output. And so, if that makes sense so far.

14:56 Michael Kennedy: So if you had a list of four, four names, name one, two, three, and four, and you forget the comma, 'cause you added a new line, but you forgot the comma, what's in the list is name one, name two, name three and four as one string, like, as if you'd put a plus there, and that drives me crazy about Python. Like, I understand, okay, there's a few conveniences, but the ability to introduce bugs by having two string variables next to each other and forgetting a comma or something is really frustrating.

15:26 Dan Bader: It is totally frustrating, yeah. And so this is something that I saw come up in code reviews, you know, as I was, we were at the time, like, onboarding two new developers, and this actually came up in the code review. And we're like, what's going on? Like, I hate this language, right? Like, it was one of these things where you run into it, and you're like, well, why did this just happen? And it's like, a really nasty error or bug that you can introduce into your programs. And so basically, we came up with this rule of just ending every single item with a comma, because in Python, you can actually have a trailing comma even in the last, in the final item in one of these collection constants.

16:04 Michael Kennedy: Right, the dictionary, the list, whatever, right? It can always have that comma there. The other benefit is, if you don't put the trailing comma and you're going to create new items in that list and you look at a diff in source control, it looks like the previous item is changed as well, 'cause that line changed with a comma, whereas it only changed so you could extend it. But if you always put the comma there, like, literally, the lines that change started the lines that changed.

16:29 Dan Bader: Yeah, that's right. You can keep your divs, your get divs, nice and clean. And, I mean, you know, I don't know, like, this probably makes us sound like super nit-picky, 'cause you're like, oh, you're like the senior developer reviewing someone else's code, and you're pointing out stuff like that, but honestly, I think a lot of times, it's the little things, right? Like, if you see like, it's a little bit hard to talk about this, I guess, on a podcast where we can't show people code examples, but if you see this in front of you, usually, people go like, oh yeah, that makes sense. I'm just going to start doing that. Because you can just remove that class of errors or that potential for errors there. You can just remove that completely just by making one tiny, tiny small change to your coding habits, or, you know, how you write your code.

17:09 Michael Kennedy: It's super simple to do and understand how it works. It's great. The other place that that's really helpful is for code generation. So imagine you've got, like, a cookie cutter template, and you enter some values, and that's powered by Jinja 2. So, if it's a variable, give it a list, and it'll generate, like, it could generate a bunch of code that's going to generate your starter project. And because you can trail everything with a comma, you don't need some weird if case to like, filter out the last comma. You just throw that puppy in there, and it's all good.

17:36 Dan Bader: Oh, that's nice, yeah. I see that would make sense, yeah.

17:40 Michael Kennedy: This portion of Talk Python to Me is brought to you by Linode. Are you looking for bulletproof hosting that's fast, simple, and incredibly affordable? Look past that bookstore and check out Linode at talkpython.fm/linode, L-I-N-O-D-E. Plans start at just five dollars a month for a dedicated server with a gig of RAM. They have 10 data centers across the globe, so no matter where you are, there's a data center near you. Whether you want to run your Python web app, host a private Git server, or a file server, you'll get native SSDs on all machines, a newly upgraded 200 gigabit network, and 24/7 friendly support, even on holidays, and a seven-day money back guarantee. Want a dedicated server for free for the next four months? Use the coupon code PYTHON17 at talkpython.fm/linode. Let's move onto sort of a more protective, inventive way of coding by covering our assets with asserts.

18:36 Dan Bader: Yeah, the assert statement.

18:37 Michael Kennedy: Yeah, so you talked about the assert statement, which I don't see being used all that much in the code that I run across, but maybe it's just what I've been looking at. But I don't see it that often.

18:46 Dan Bader: Yeah, and I think that's a shame. I don't know, like I always felt the assert statement in Python was really, really useful. I feel like it's under-utilized. It can really help people write, well, making their code easier to debug, first and foremost. You know, it's like a defensive programming technique. And basically what an assert statement does, it's like a shorthand for testing a condition, and then, if the condition is false, it will automatically raise an exception. It will just throw an assertion error. So you can use assert statements in your code just to quickly test conditions if you don't want to write, you know, like a longer if statement and really, you know, maybe define a custom exception or anything like that. You just want to make sure that, for example, a product price, it shouldn't be negative or something like that, where you don't really expect this to happen, but you kind of want to make sure that it can never actually really happen in production, because crazy things sometimes happen in production. And this is just a great little tool or little feature in Python that you can use.

19:46 Michael Kennedy: Yeah, and I really like the little examples that you give. So like when you introduce assertions, you talk about like, let's imagine you have a store, say, for buying online classes or something, and you want to set up a discount coupon type of mechanism. And so you might assert that after applying the discount, the price is no less than zero and no more than the actual maximum price. You don't get like a negative coupon or something, charge 'em more than if they didn't enter it.

20:11 Dan Bader: The worst coupon in the world.

20:13 Michael Kennedy: Please don't put it in. Yeah, so those are all these little examples you have in the book are really great. So like asserting after you apply the discount code that the price is zero to the full price is pretty straightforward. So, you do talk about some really interesting things in here. One is that the idea behind assertions is not the same as proper error handling, and that these are really meant to handle conditions that are like bugs, not just invalid input.

20:43 Dan Bader: That's right, yeah. You want to basically use an assertion if you want to inform yourself or fellow developers about an unrecoverable error in your program. So I like to think of them more as a debugging aid rather than just, you know, a glorified, or like a fancy or weird way to throw an exception. So this is really, you know, your app is probably still going to come down crashing, but at least you will know exactly what caused your application to crash. So it's usually going to make it easier to have your application, to have it fail fast right when it hits that condition that it should not or when it goes into a state that it should not be in rather than having that carry on and at a later time, you know, cause trouble further down the road. And so using an assert statement or using multiple assert statements, that's a great way to just have these little checkpoints in your code that you can just sprinkle around, these little self-checks, and they can really save you a ton of time, because they give you all the context you need. They make it easier for you to have that context to debug your programs, because of course nothing is a silver bullet here.

21:49 Michael Kennedy: Yeah, sure. But I do think it's good to have, like you said, that context, because a lot of times, maybe the website would start crashing because Stripe is rejecting the price that you're trying to charge when the price is negative. And you're like, what is going on here? Why is this happening, right? Whereas if you had an assertion that said, well, when we applied the discount code, the price went negative, like, oh, I see. I know what's going on here. So it's really nice. A final warning that you had in here is you should not use assertions for regular checks. Like if you ask a person to enter a number between zero and 10 and they enter 11, you shouldn't like, assert that that's between zero to 10, right, because they can be turned off, the dunder debug setting that you talk about, and also, they don't tell you anything about it from a perspective of the exception, right? It's always an assertion error, not like a negative number custom exception thing you made or something.

22:49 Dan Bader: Yeah, you can add a message to the assertion. That will also be part of the exception stack trace. But yeah, like you said, it's pretty dangerous to use assertions for actually handling conditions or checking for conditions that you expect to happen as part of the normal program flow. So, for example, if you do your user validation with an assert statement, if you go assert user is admin or something like that, well, that's going to work great. But as soon as these assertions are turned off, and they can be globally disabled at the interpreter level, then all of the sudden, everyone can log into your app and do all kinds of crazy things, because all of the sudden, everyone is an admin. And so you don't want to do that.

23:30 Michael Kennedy: Yep, that's for sure. Yeah, you would basically be able to pass a flag that turns off all error handling, pre-condition checking. That might be bad. Do you know why they are statements and not functions?

23:42 Dan Bader: I remember reading about that. I hope I didn't actually mention this in the book, and now I forgot about it.

23:47 Michael Kennedy: I don't think you did. It seems to me like maybe one of the reasons is if you do turn them off, they can like literally vanish from the code and not just become operations that don't do anything. So maybe they're easier to like factor out. I actually have no idea. But it drives me crazy. Like, there's like weird conditions about the tuples versus parameters and all sorts of funkiness because they're not just like, actual calls, function calls.

24:11 Dan Bader: That's right. It's very easy to write assertion statements that never fail, because you would semi-intuitively think, this feels more like a function call, sort of like the print statement versus the print function. But if you write a syntax or if you use this like a function and, you know, you put parentheses around it, Python can actually interpret this as a tuple. So it's like the assert statement plus a tuple expression. And this will always be, so this will also be true, or truthy,

24:44 Michael Kennedy: Because it's not empty, yeah.

24:45 Dan Bader: Yeah. You can shoot yourself in the foot that way. But I think this is now actually a warning in Python 3.

24:52 Michael Kennedy: Yeah, you do get a warning in at least 3.6. I can tell you that for sure. So that's good.

24:56 Dan Bader: Yeah.

24:56 Michael Kennedy: Yeah. But I'd rather have just the language actually, you know, just not let that be possible in the first place. All right, so let's move onto the next one, the shocking truth about string formatting. It's not very zen-like, is it?

25:10 Dan Bader: Yeah, no, that's true. I mean, this is one of the realizations I had when the format strings or f-strings landed in Python 3.6. I was like, hold on a minute. There's like four different ways to do string formatting in Python now.

25:24 Michael Kennedy: And variations on each four of those.

25:27 Dan Bader: So which one should I actually use? And it seemed to go against the zen of Python as well. And, you know, I guess it's one of these things where, well, languages, they evolve over time, and we can't just throw out, you know, all of the legacy code that we have. So it's like a trade-off that we need to make or people need to make. But still, I just thought it was interesting, and yeah, so let's chat about that.

25:51 Michael Kennedy: Yeah, for sure. So we've got the old style formatting, the like string with a percent, a format character, so %s, %d and so on, %f, then mod, the values are a tuple of the values to be inserted, and that's kind of more or less not the way these days, right?

26:12 Dan Bader: Yeah, I mean it's called old style formatting for a reason, I guess. But yeah. So we have a newer way of doing string formatting. It's called, it's referred to as new style string formatting using the String.format function.

26:27 Michael Kennedy: But it's not actually the newest style.

26:30 Dan Bader: That's right.

26:31 Michael Kennedy: It's middle-aged.

26:32 Dan Bader: And the old style isn't actually...

26:33 Michael Kennedy: It's not deprecated, yeah.

26:34 Dan Bader: It's not going to go away, right? So I guess it's just a hint that we got from the CPython team that, you know, really, you should be using, maybe you shouldn't be using the old style string formatting. On the other hand, it is kind of nice to use.

26:48 Michael Kennedy: Yeah. And it's, you know, the new style is still in Python 2.7. So new style would be like, hello, curly, curly, all that in quotes, .format, name, or something like that. And that's pretty standard. That's really nice. Takes a lot of, sort of a ton of formatting mini-language to convert the values like numbers to digit grouping and things like that. And then we've got the f-strings, which are the new hotness in 3.6, right?

27:15 Dan Bader: Yeah, the f-strings are just awesome. They were added in Python 3.6, and I think it's a really, really great feature. The f-strings are also called formatted string literals and people also refer to this as string interpolation. So really, what this means is it's basically a new type of string. So you just put a lowercase f in front of your string. And then you can use curly braces to just have Python expressions right in your string, and when Python evaluates that string, it's also going to evaluate those expressions, convert them to a string, and then merge them with the surrounding pieces or parts of that string. So, this is a really, really nice and concise way to define strings in Python that are dynamic, and it does the same job than all of the other formatting tools that we talked about or formatting styles, but I find this to be very readable and just very, very nice to work with.

28:09 Michael Kennedy: Yeah, I really like it too. The more I use it, the more I really want to use it everywhere. It's great. And so, you would say f, quote, and then have, you know, regular text, and then curly, curly, but then like you said, you put these expressions. And they don't... It looks very similar to the dictionary version or the keyword value version of the old format or the new old format or middle-aged format, but you can do things that are unexpected, like you could actually call functions inside there. You can add up things. You can, what I didn't realize is you can actually put format descriptors, like some variable, colon, hash, x for hexadecimal representation. And that's pretty cool.

28:51 Dan Bader: It's pretty neat, and I think it's, it's also nice and readable, because you just have all of the context you want. Like, one of the challenges I felt with, you know, at least old style string formatting format is that a lot of times, you have to jump back and forth, and then, you know, if you have a complicated format string, you have to actually count out, like, okay, this %s placeholder, it refers to this other variable, and you just have to do all of that, that mental gymnastics to figure out what's actually going on. And with f-strings, well, you see it right in front of you, and it's kind of nice.

29:25 Michael Kennedy: Yeah, it eliminates a whole class of bugs. Like, it's totally possible to have too few arguments in like a format expression or too many arguments or get them in the wrong order, but when the actual variable goes in the spot, like, that's what in the spot. There's no more debating it. I guess the one thing that is kind of nice sometimes is if in the string you have to repeat something that's like sort of an expression that got evaluated. Like, that might be in the middle one. So I'm going to propose some nomenclature here, and we'll see if the world will take it. I don't know. So, let's call the old style, old style is fine, or it could be legacy style formatting. I think that's even a better word for it. The new style, what should we call that? What do you think? The traditional string formatting style? And then the interpolation, we could call that the new style.

30:16 Dan Bader: Yeah, let's do that. Let's go and submit a pull request on the Python, CPython docs.

30:23 Michael Kennedy: Exactly. Let's do a PEP to change the naming. Exactly, exactly. And then really quick, the last one is one that I really haven't played with, which is it's sort of the little bobby-tables equivalent of string. So, it's something called string templates, and it's really good for when you're taking untrusted input and jamming it into a string format.

30:44 Dan Bader: Yeah. This is great when you're dealing with user input, when you have something else that comes in externally where users can actually define those templates, because the problem is, if you just feed that into one of the other formatting techniques or methods, people can actually execute code into your interpreter environment. So there's a way to reach out and look at global variables, and I think there's a really good article on Armin Ronacher's blog where he demonstrated that technique. And basically, you can go in and there's a way to reach out from a format string and to reach into the dunder globals dictionary and just to look up any variable in the global, global variable environment. So, if you have your secret API key in there and this is a web app, then, well, bye-bye, API key. It's just going to, people can leak it out.

31:37 Michael Kennedy: It's like, what's my name? My name is global[aws.secretKey]. Oh, my goodness. Yeah, that's bad.

31:45 Dan Bader: Yeah.

31:46 Michael Kennedy: Okay, yeah. So you can use this template thing.

31:47 Dan Bader: And so the template strings, they prevent that.

31:49 Michael Kennedy: Yeah, nice. They're kind of like price queries in SQL expressions. Like, they don't really get evaluated. They just get put in there as raw strings. But they kind of signal that this part is input and it doesn't run. This just goes here as text. This portion of Talk Python to Me has been brought to you by Rollbar. One of the frustrating things about being a developer is dealing with errors. Relying on users to report errors, digging through log files trying to debug issues, or getting millions of alerts just flooding your inbox and ruining your day. With Rollbar's full stack error monitoring, you get the context, insight, and control you need to find and fix bugs faster. Adding Rollbar to your python app is as easy as pip install rollbar. You can start tracking production errors and appointments in eight minutes or less. Are you considering self-hosting tools for security or compliance reasons? Then you should really check out Rollbar's Compliance SaaS option. Get advanced security features and meet compliance without the hassle of self-hosting, including HIPA, ISO 27001, Privacy Shield, and more. They'd love to give you a demo. Give Rollbar a try today. Go to talkpython.fm/rollbar and check them out. So, the next one that you talked about is exceptions. And there's all sorts of stuff around exceptions. I mean, there's the whole embracing the look before you leap, rather, disregarding the look before you leap style from C and embracing the it's easier to ask for forgiveness than permission and just really using exceptions as, like, for exceptional cases, right? But then there's also making sense of the, consuming the APIs that have been really embracing that format. So, your recommendation is to define your own exception classes to make your code more readable, self-documented, and so on, right?

33:37 Dan Bader: Under some circumstances, I think that makes a lot of sense. I feel like a lot of times, you're totally fine if you just, let's say, raise a value or a key error. If it's just perfectly clear, you know, if you're working with a small function, and this is not some perfectly polished library that you're releasing to the world, you know, just go ahead and raise a value error. Don't bother with defining your own exception hierarchy. But I think as soon as you have a larger piece of code that you want to share, either across multiple applications or just, you know, kind of have it nicely sort of, in a well-defined package where it's its own package, it's its own module, then I think it makes sense.

34:14 Michael Kennedy: What do you think about this? What if you have your rule as, if it has a setup.py, it has its own exception hierarchy.

34:20 Dan Bader: I think that makes a lot of sense, yeah.

34:22 Michael Kennedy: If it's a package, people consume it externally, maybe it should be more formalized in this way, yeah?

34:28 Dan Bader: Yeah. At the very least, then, and also, if you're working in a bigger application, you know, where you have a lot of code, like, I don't do this in any little script that I write, but if you have, there's a larger body of code, I feel like these are the things that really make it easier for you also to work with your code in the future. Because it's just nicer, or it's easier for you to see, you know, what went wrong and where. Essentially, a value error can come from many, many places. But if it's a password validator exception that you defined, then, well, we know that this has something to do with password validation, or, you know, the next person working in the code knows that it has something to do with that.

35:05 Michael Kennedy: Another thing that I think is really nice that you talked about is actually having a base exception that defines like, these are all the errors that come out of this particular package. So, that makes me crazy. Sometimes I'll be handling errors out of different packages, and I'm like, what is this thing throwing? Is it throwing one of its own errors? Is it throwing like a system socket error? Like, what is this, and how do I find a way to catch it? You know what I mean? It'd be really nice to go, okay, I'm working with the SQLAlchemy library, so I'm going to catch, I'm going to start by catching SQLAlchemy errors, and like, dealing with that, and then I'll start to add on special cases of like, oh, yeah, this is when the network is down. This is when the database doesn't exist, or, you know, whatever.

35:48 Dan Bader: Yeah, exactly. And some people start using ValueError or something like that, like a built-in exception for that, and then just put that information in the exception message. And I mean, for debugging purposes, I think that's okay, because, well, we're just going to look at the stack trace, and you know, sort of know what's happened. But you really don't have the granularity when you're catching these exceptions, right, when you have a try-except block, and like actually the exception is part of your structuring the control flow of your program. Then that becomes very dangerous, because then, well, you kind of have to parse out that string. And I don't know, I don't like that.

36:25 Michael Kennedy: Yeah, well, and also, if you're trying to say, well, I'm trying to deal with a case when there's a value error, but is it, is it from this library or that library? 'Cause that means really different things. One is the network is down. The other is you don't have permission. And I don't really know what to do here, right? So you can totally help that. And, you know, another thing that I'm not sure people think very much about but I think would be really helpful is like, discovering the solution to some common problems. So if it's a generic standard library error, and you go search for that on, like, Google and Stack Overflow, I use Rollbar, and Rollbar has this sort of mechanism where you go into the error reporting for your web app, and it says, here are the common solutions or issues that are found associated with this error. But if it's super general, it's like, well, nothing, right? But if it's pretty specific errors, like, you could probably get help more easily on lots of levels that way.

37:19 Dan Bader: I really like your thinking there. It's like this whole meta game of programming, right, where you're just fending on your own, like, you're fighting the machine there, but a lot of times, it's just great if you can punch something in Google and just at least find some other people who have the same problem, and yeah, I think it really increases the googleability of your exceptions if you do that.

37:41 Michael Kennedy: Yeah, absolutely. I totally agree. All right, so let's talk about naming a little bit. I know that you've talked about this on your blog and some articles. I've talked about this with Brian Okken on Python Bytes. There's a special meaning for underscore and double underscore as in prefix underscores and suffix underscores in Python, and maybe we should just touch on those a little bit. There's five or so that you cover in here.

38:05 Dan Bader: So many underscores.

38:07 Michael Kennedy: Yeah, Python does love the underscore.

38:09 Dan Bader: I'm getting undersores from all these underscores.

38:12 Michael Kennedy: So, the first thing that we can start off with is like a semi-private variable, _var.

38:18 Dan Bader: Yeah. So the _var is just a naming convention. And I like that you called it semi-private, because a lot of times, when people come from, let's say a Java or C# background, they assume that private really means private in this context. Like, you cannot, you know, reach in and modify this variable, but in Python, if you do _var, it's a naming convention. You know, it's just something that the community agrees on. It's like a hint to the programmer, but it's generally not enforced. So that's something you got to keep in mind. Like, we're all consenting adults, or whatever Guido said about that.

38:51 Michael Kennedy: Exactly. I've heard him describe it that way as well. But then there's the yeah, but I still don't trust you, so I'll put another underscore in front of it.

39:00 Dan Bader: Yeah, and that's where it gets interesting, because that enables, or that kicks, that makes the name-mangling kick in. So if you do a double underscore in front of your variable name, the Python interpreter will actually mangle that name and prepend the name of the surrounding class to that variable. So this sort of kind of makes it like a private variable, but not really, because if you do the mangling yourself, you know, if you just do the string concatenation in your head or in your program, you can still reach in and modify that variable. So it's really more a mechanism for preventing naming collisions but not, you know, not a way to implement private variables.

39:42 Michael Kennedy: Not true private. I mean, it's not private in the sense that it's private. But there's still really not that much private in all the other languages. So if you think of, say, C#, it has a private, protected, internal, all these different mechanisms and scoping rules, but then on the other hand, you have reflection, and you could just go ask for it. For the private variables, Java has introspection. C++, they have private, but, you know, how many bytes offset is that? Okay, we just go to the front of the object and read that form over. And like in all of these languages, there's a backdoor to these privates, right? And so I feel like this is really no more or less than that. It's just maybe slightly easier if you know the convention. But it certainly screams to you, don't do this.

40:31 Dan Bader: Yeah, it's like you're digging a moat around your fortress, and of course people can still overcome it and destroy your beautiful programming fortress, but you're just making it a little bit harder for people to hurt themselves, so.

40:44 Michael Kennedy: Yeah. I mean, the whole point is like, people shouldn't mess with the internal implementation. They shouldn't depend on the internal implementation. And if they do, they're going to suffer, not you, probably, unless they file a bug, and you've got to deal with it.

40:57 Dan Bader: Yeah, you would hope so, right? It's not you as the maintainer who gets the flack for that.

41:02 Michael Kennedy: Yeah, exactly. But I do think like, that's kind of far enough. I use that every now and then. I also use it for when I want clean autocomplete. If there's some library that I'm like, I want to give this to somebody, I want them to say thing, dot, and the list of the stuff I want them to work with really clearly, I'll put that, either the underscore or the double underscore, not so much to make people stay away, but to like, make the API more discoverable in the tooling.

41:29 Dan Bader: Oh, that's nice. That's a good idea.

41:31 Michael Kennedy: Yeah, the double underscore like, takes it entirely out of the list, so I'm kind of like, eh, if it's not going to bother me too much, I might do that. But that's one of my motivations. I did want to ask you, with the single underscore, that means like, internal, private, but do you think that means protected, or truly private? Like, only in the class, or classes and its derivatives?

41:51 Dan Bader: That will come down to the individual developers' and their teams' agreement. Personally, I use it more like a protected variable, I think, where it's like, yeah, you know, if it makes sense to reach in and modify this and we're part of the same, you know, if it's related code, very, very closely related code, then I feel like it's okay. But it's definitely one of these things where it's like, A, it's a slippery slope, or it's like, oh, yeah, it'll be fine. And then it's like one of these famous last word things.

42:23 Michael Kennedy: It's this convention, and it's not super explicit. Yeah. But, I think it's good to know about it. So, just really quickly, maybe touch on the other underscores. There's var_, the suffix.

42:36 Dan Bader: Yeah, so that's mostly just used to get around, so, you know, you can't use a reserved word like a Python keyword like class, you can't use that as a variable name in your programs. But a lot of times it actually makes sense to have a variable name named class, if you're passing in some kind of class to a function.

42:54 Michael Kennedy: Yeah, the place where I've seen it is in Beautiful Soup. You can do a find and specify a class keyword, and then you'd say the name of the class you're looking for in the HTML, but you can't say class, right? So they have class_= as the keyword argument.

43:08 Dan Bader: Yeah, there you go. And then people sometimes do, they just kind of mangle the name themselves.

43:13 Michael Kennedy: Cls.

43:13 Dan Bader: So they maybe call it klass with a K, or change the, yeah, with a z. I actually went and drug through PEP 8 and saw what people recommend, or like the official recommendation, is to just end your variable name with an underscore. That disambiguates it and makes sure it doesn't conflict, or there's no conflict with the keyword or the built-in, reserved word. And I kind of like that, because I feel like, yep, it's pretty clear, pretty easy to remember, and we have more underscores that way. Isn't that wonderful?

43:43 Michael Kennedy: Yeah, with have more underscores. All these languages have ways of dealing with them, and none of 'em are really amazing. This seems just totally, totally decent. And then we have the dunder methods, which is really the Python data model magic methods, which I know you don't like that term.

43:58 Dan Bader: Do you like that? I was going to ask you, do you like the word magic methods?

44:02 Michael Kennedy: To me, they feel like operators. I don't know. I don't really like the term either. 'Cause magic seems like they're inaccessible and they're like, these internal black-box things. They're not really, right? And then the final thing around underscores, 'cause Python loves underscores is to use it as the, I don't care about this variable. Don't lint it. Don't make me name it. I don't care about it.

44:26 Dan Bader: Yeah, it almost has like a high school kind of feel to it, where you're like, oh, yeah, I'm so cool. I'm ignoring all these variables, and I'm deconstructing tuples and whatnot.

44:35 Michael Kennedy: Yeah, that's exactly. Yeah. So if you want to like loop over something like 10 times, you could say for n in range one to 11 or zero to 10 or whatever, but if you don't care about that n, you know, you might get warnings like that variable's not used and whatnot. And if you want to be explicit and say, I don't care about the loop variable, I just want to do this 10 times, just put an underscore instead of n, and then everything's good. Same thing for like parameters, yeah, in methods.

45:01 Dan Bader: Yeah. The only thing you got to be careful about is when you do that in a Python REPL session where the underscore, like a single, standalone underscore usually represents the result of the last expression that the interpreter evaluated, which is kind of nice if you're using, you know, if you're working within REPL, but if you redefine it, that can cause all kinds of trouble. So, you know, that's kind of the one caveat with using the single underscore.

45:28 Michael Kennedy: That's for sure. And it's nice because if you type an expression and you're like, oh, geez, I would love to use that as a variable, you can say, variable name equals underscore, and then roll on from there. That's nice in the REPL, yeah. All right, so let's move onto data structures. And this is like a pretty significant chapter that you talk about. And I kind of hinted at this at the opening, like, you can come in and find the really basic structures really easily, but there's actually a rich set of data structures that are special-made for certain circumstances and situations. So, let's dig into those. Maybe, yeah, let's start with the dictionaries, 'cause that's a pretty central thing in Python.

46:04 Dan Bader: Yeah, it's like the central data structure definitely I think in Python, and I feel like probably in programming in general. It's like, just a super versatile tool.

46:14 Michael Kennedy: Yeah, it's not just something you use in Python, right? Like, the variables and attributes in a class are defined by those. You have the stuff in the module, like global variables, like, all sorts of stuff, right? Like, it's the structure that stores and that holds the types of Python together, I guess.

46:30 Dan Bader: Like, the stuff the universe is really made of.

46:32 Michael Kennedy: It's the dark matter of Python.

46:38 Dan Bader: I like that. Yeah. And it's, I mean, it's everywhere, you know, but on the other hand, when we started, you know, the last place I worked at, we were interviewing pretty aggressively, and by aggressively, I mean, you know, interviewing a lot of people, and trying to hire Python developers. And so one of the things that came up over and over again in those interviews was that people were really struggling with when you're talking about like basic, like, hash table or hash map questions, people were like, hmm, yeah, I'm not sure if that's available in Python, and I don't really know how this concept would work in Python, but at the same time, they knew about dictionaries. So that was just... I don't know. I mean, obviously, I understand where this is coming from, and this is not necessarily like a terrible sign, but on the other hand, I was curious, or just confused, like, why was this happening? I mean, you know, like why is there this disconnect? Because in Java, well, if you want like a mapping structure or dictionary, you know that you're dealing with some kind of map or a hash map, right? You know what kind of concrete data structure you instantiate. Python is just like, yeah, it's a dictionary. It does its job. I don't really care how it works. And that's good, but it also means you're kind of, you're disconnected from a lot of the specific implementation details as a developer.

47:49 Michael Kennedy: Yeah, that's for sure. And until recently, dictionaries, and still under some circumstances, they're unordered, which can cause all sorts of issues. if you read a file in, make some changes, and save it back out in Python, you know, and you had that file in source control, it could look like 95% change, where you just changed like something from true to false or something silly like that, right? And so, there's actually, depending on the situations that you're in, there's all these other types of dictionaries that are pretty awesome. So we have ordereddict. I've seen the people that do data science and like, really chain together generator expressions and list comprehensions and make massive use of defaultdict. There's chainmap, which I don't think I had heard of chainmap before. Maybe I heard of it and forgot about it. And there's even MappingProxyType. So maybe take us through when you might want to use those other things, like when dict or curly, curly, is not enough.

48:41 Dan Bader: I was also really, really amazed or you know, really enjoyed learning more about chainmap. So, chainmap's really cool. Basically, you can chain together multiple dictionaries. You can treat them like a single dictionary, like a single mapping. So, you can group several dictionaries into one, and then when you do a lookup, actually the chainmap is going to go and it's going to search all of these underlying dictionaries one by one until it finds a match.

49:06 Michael Kennedy: Yeah, that's really cool. To me, it let like what I had previously done. If I wanted these behaviors, I would create a new dictionary, and I would merge all of them into it, and then I would ask that question. But this lets you basically create the same behavior without actually building that other dictionary, and potentially, you go create the chainmap, and then one of the underlying dictionaries could change, and then the answer coming back from the chainmap would adapt to that, which is pretty awesome, right?

49:31 Dan Bader: That's right. And you don't incur that memory overhead, because, you know, it's just like an abstract way to merge all of these dictionaries together, and it's just, the magic is in the search algorithm. But, on the other hand, of course, it can potentially be slower, because the search actually needs to happen. So depending on, you know, if you chain together thousands of dictionaries, maybe that's not a good idea, but if you, if you only have a couple of them and maybe you're combining a bunch of configuration dictionaries, then this could be really handy, and you could just make sure, you know, okay, we have like a default setting, dictionary with default settings, and you know, we can override them with more specific user settings.

50:08 Michael Kennedy: Right, put the user settings and stuff at the beginning of the chainmap, yeah. That's nice. Yeah, the place I see this happening all the time on the web is 'cause you've got a dictionary for like, the route URL data. You've got one for the post. You've got one for the query string. You've got one for, where else, somewhere else? Maybe the headers, all sorts of stuff. And you can like combine them together into one place and ask, like, did they give me this data somehow? I don't really care where it came from. Did they tell me their username or something, right?

50:35 Dan Bader: Where's the API key?

50:36 Michael Kennedy: Yeah, exactly. Then where's that print statement I'm looking for? All right. So another major class of data structures that I've found to be pretty interesting that you covered were arrays. So, you know, sort of the most common one of this is the list, but there's more than that, right?

50:51 Dan Bader: Oh, yeah. Yeah. It's really similar to, you know, what we just discussed with the dictionaries, where, if you're looking for an array, it's typically, people will just tell you, well, use a Python list. It does everything you want. And, you know, behind the scenes, Python lists are actually dynamic arrays, so mutable dynamic arrays. They will grow and shrink with the data that you put in. But there's actually other options as well. You know, you can, in a lot of ways, you can view a tuple as an immutable array, which means you can't change it.

51:20 Michael Kennedy: Right, you index into it. It has a length. You can say, give me the third, give me the fifth one, but it's totally static. Even the individual elements can't be changed, right?

51:28 Dan Bader: Exactly, yeah. And, I mean, the only problem you run into, if you have like a nested data structure, so if you have a dictionary in your tuple, you can't put something else at that position in the tuple and replace the dictionary, but you could still reach in and modify the dictionary, because the dictionary itself is not immutable. You can modify it. And so sometimes, you know, that causes trouble for people. But in general, you know, yeah, you can absolutely treat a tuple, like, as an immutable array.

51:54 Michael Kennedy: Let's go back just for a second. Like, one thing we kind of moved on from without talking much about is MappingProxyType in the dictionary space. And so if you really wanted like this truly immutable tuple, you wanted to put a dictionary there, you could wrap it in this MappingProxyType, right?

52:09 Dan Bader: Aw, yeah, that's a perfect segue. Yeah, exactly. So, you could use this MappingProxyType class to take any standard dictionary and then to provide a read-only view into that dictionary. So you can still access all of the variables and all the values and all the keys, but you cannot reach in and modify the dictionary. I mean, sure, you could just kind of pry that mapping proxy apart and then reach in and modify it, and it's probably not a good idea.

52:38 Michael Kennedy: But that goes back to one of the, like, you know you shouldn't be doing it, and if you mess it up, it's kind of your fault. Not the API's.

52:46 Dan Bader: It's like, yeah, you could install a kernel driver and patch the memory.

52:50 Michael Kennedy: I rooted my machine and then it crashed. Oh, this thing is a crappy OS. Yeah, so that would, I mean, you could combine that for like a greater functional program, immutable pass-by value type of behavior, right?

53:02 Dan Bader: Yeah, you could do that. I mean, personally, I'm always thinking, like, okay, at what point do we reach the level where we're really fighting against what the language wants us to do? But it is entirely possible, you know, to do what we just talked about. You know, you have a tuple, and then instead of putting dictionaries in there, I mean, you'd probably just go with the frozendict class for that, but you could totally also wrap it in this mapping proxy type thing, and that way, really enforce that immutability as best as we can. And I think that's so nice about Python.

53:32 Michael Kennedy: The frozendict is a little more obvious. But yeah, that is super nice. One of the things that traditional, let's say C, C++ arrays to a lesser degree, like some category of the C# arrays, have is they have the same type in them, and then they're rightly packed. And it's the tightly packed thing that I'm really thinking about, right. So, if I say list in Python and I put number one, number two, number three in there, that's really a tightly packed set of pointers that points out to three things that are not anywhere near each other, potentially. I mean, those are probably pre-allocated and put next to... Like I know one to 255 are pre-allocated, but, you know, in general, things in arrays are not nearby each other, which means you're following a lot of pointers, and there's a lot of GC-type behavior. But there are structures that if you really want homogeneous types tightly packed, you can get them, right?

54:26 Dan Bader: You can also get that in Python, because like you said, with your standard list, there's really no guarantees. Your list of a couple of elements, it could actually be much larger than just those like five or six elements, because it's pre-allocated to a bigger size and kind of shrinking and growing dynamically. And so if you want absolute control over how your data is laid out in memory, then you could use something like the array class from the array module that's in the Python standard library, and with this, you can actually say, hey, I want an array of C-style floats or ints, and you can just put in your values there, and they will be laid out in memory exactly the way you requested.

55:07 Michael Kennedy: But it won't accept types that don't fit the declared data type, right? So if you say it takes floats, it won't take a string, right? It enforces that as well as the packing.

55:16 Dan Bader: Yeah, that's right. So you get some type safety there. So it's going to make sure that you're not putting in, I don't know, something crazy. It needs to be consistent.

55:23 Michael Kennedy: Well, let's skip ahead a little bit. We're actually, I thought we'd get through more of these. I think we're going to run out of time pretty quick. But there's some stuff I really want to cover while we're here, because I think it's quite interesting. You talk about stacks, and you talk about queues, and I think this list that we were just talking about, it can be many things to many people. It can act as a queue, and you can, you know, take the thing off the front. It has a pop to take things off the end that you can append that's like, a synonym for push, sort of, but there's also other queues, and there's also some really terrible performance issues if you try to use, say, like a list for a queue. So, let's maybe touch on those things while we're on this topic.

56:02 Dan Bader: Yeah, for sure. So I think the list, again, like it's such a versatile data structure in Python, right? And it's, I feel like they made exactly the right performance trade-offs when they implemented how list is actually implemented in the CPython interpreter. On the other hand, it leads to those situations where you're like, oh, yeah, you know, I can push and pop from lists. So I can totally make a stack or I can totally use it to make a queue, and, I mean, you can, but the problem is, for example, if you use a list to make a queue, you can run into a situation where you'll just have terribly slow performance, because if you enqueue an element in a list and you add it, like, you insert it at the front, this will actually shift all of the other elements by one. So you're looking at a linear time operation just to insert one element, and that's really, really bad for a queue implementation.

56:56 Michael Kennedy: That's super bad, yeah, if you're doing like the first in, first out behavior, right. So append, that's fine, it goes on the end, but every time you take an element off the front, it has to entirely copy every element. If there's a million items in your queue, that's a heavy weight dequeue operation right there. So, ironically, you should use a thing called dequeue to fix it.

57:16 Dan Bader: Yeah, so there's this great, great little class in the collections module, and the collections module is just so awesome. If you're listening to this, for anyone who's listening to this show, and they haven't heard of played with the collections module, check out the collections module. It's just a treasure trove, and it's also fun reading, you know, just seeing what's available in there and the reasons for why it exists. But yeah, the collections module has this thing called dequeue, or deq, I think some people pronounce it, and this is actually implemented as a linked list. So you can append and remove from both ends really, really fast, because there's no shuffling around. It's just, okay, we cut off the head or add something to the end.

57:59 Michael Kennedy: It's a single pointer operation. It's super, super fast. That's right. And the D-E at the beginning is not the operation of removing the item from the queue, but that's the double-ended queue. D-E-queue, right?

58:11 Dan Bader: Yeah, that's right. It's fast from the front and fast from the back as well.

58:15 Michael Kennedy: Right. And we have things like queue.Queue and multiprocessing.Queue and some other ones, that special cases one. So, I think it's really interesting to think about these, 'cause a lot of 'em, like we said, you don't really discover them on your own. You have to kind of study them and seek them out, and you're like, oh, that's, I thought just Python was slow. List turned out to just be a really bad queue. So, maybe I'll do something different, right? Yeah, so I think these are all really great, but I don't want to give people the impression that your book is all about just data structures, right? There's actually, you know, just one chapter. It just happens to be a super rich chapter. So maybe, like, we have some other stuff that was on deck that we're not probably going to be able to talk about in terms of time, but maybe just run us super quick through like what else you wanted to highlight.

58:58 Dan Bader: When I wrote this book, really, the angle it's written from is that I don't want to create another technical reference, but I want to walk people through some actionable stuff and explain things step-by-step with examples that make sense. So I'm also covering kind of the classic features in Python that are often difficult to wrap your head around. And, you know, I don't know how much more time we have here, but, for example, I think if people understand decorators and like really understand how they work internally, that is a huge leap from a beginner level understanding of Python.

59:38 Michael Kennedy: I agree.

59:39 Dan Bader: Because you need to understand, you know, a couple of rather advanced features. Like you need to understand what first-class functions are, that all functions can be treated, are real, full-blown objects, and, you know, you can not just call them but also hand them around to other functions.

59:53 Michael Kennedy: Right. Closures, the concept of like, variable capture closures is front and center there.

59:58 Dan Bader: And that is all super important stuff, and just so by, you know, focusing, really understanding how this feature works, you can actually learn a lot about Python, and you can pick up a lot of other features and abilities that are going to help you in programming Python in general. And so a big part of the book is, you know, I really spent a lot of time on this, and it's been iterated over. I tested this with people, and just walking them through understanding how decorators, for example, work, or walking them through iterators and then going from there to generators and then later generate expressions, and really just explaining the logical connection between all of those and kind of structuring in a way where it not just like, oh, yeah, this is how it is, but kind of approaching the topic step-by-step.

01:00:46 Michael Kennedy: I think those are really, really nice things. Certainly building a simple decorator, like, come up with some concept for it that makes sense and go through the motions of creating it will really, like you said, expose you to it a lot. Some other things you talk about: you talk about the sushi operator, and I'm not going to give any more details. People have to check out your book to learn about that Python has a sushi operator. Design patterns like emulating a switch statement with dictionary dispatch, all sorts of stuff like that. So really just tons and tons of these little tricks, and you said it's like a buffet, right? So you don't have to take it in order.

01:01:20 Dan Bader: Yeah, that's right. So, most of these chunks in this book or the sections in this book, you can actually, you know, jump around and just kind of look for things that interest you. They're usually pretty short, and they all have a key takeaway section at the end so you can quickly review this. For example, I heard feedback on the data structures chapter where it's, you know, it's a great way to prepare yourself for a Python interview, just kind of, you know, get updated or get a refresher on how these standard data structures map to Python.

01:01:47 Michael Kennedy: I feel like people kind of, they take the easy way out, and they ask these algorithm questions and these data structure questions in interviews, and yes, it's important, but I don't necessarily feel like if I don't know the most efficient particular queue for this type of operation, that doesn't necessarily mean I couldn't write an awesome web app that has, you know, 100,000 users a month and has no real performance pressure on it, right?

01:02:11 Dan Bader: I agree with you. I mean, unfortunately, some people will really bust out the trick questions where it's like, aha, you didn't know about this like, special case. That means you're a fool and we're never going to hire you.

01:02:21 Michael Kennedy: The reason I don't like those is I feel like those tend to advantage the people that has CS degrees, but not necessarily better programming skills than people who are self-taught. So I really like that your book kind of backfills that information for people.

01:02:34 Dan Bader: Thanks. I mean, that was one of the goals, right, where it's like, yeah, a lot of this stuff, you don't need to spend like four years in university just to understand what a hash table is, right, and I don't like it when people create that impression. And so, you know, I feel like if you have a good practical understanding or practical programming skill, then it will be, in most cases, like pretty easy for someone to fill in those gaps in their knowledge and to satisfy the people who ask the more computer science-motivated questions, I guess.

01:03:06 Michael Kennedy: Yeah, absolutely. All right, well, let's leave it there for the book, 'cause we are definitely going to run out of time. So I do want to talk about a few other projects that you have going. First of all, you have this thing called Pythonista Cafe. Did you open, like, some kind of coffee shop or something?

01:03:22 Dan Bader: That's my lifelong dream, you know, open a coffee shop.

01:03:25 Michael Kennedy: Answer a Python question before you're permitted in.

01:03:28 Dan Bader: So, Pythonista Cafe, it all started when occasionally, I read Reddit or Hacker News. And it can be great, and it can also be, like, people have some weird personas on these forums sometimes. People kind of lash out at each other and get really worked up. And I remember this one time, like, somebody made a post on Reddit, one of the Python forums on Reddit, and said, hey, I'm this 14-year-old guy. I really love Python. I make these tutorial videos on my YouTube channel about, you know, building a flask app. What do you think about this? And like the first person to reply is like, oh, you're teaching it all wrong, and you should have never created these videos, and basically you'll never be a good programmer, and blah, blah, blah. Who do you think you are?

01:04:09 Michael Kennedy: He's probably like in the top five percent of like all 14-year-old programmers. I mean, come on.

01:04:14 Dan Bader: And that is nuts, right? Like, how can you, how can you, I was almost just like sitting in front of the computer shaking and getting angry at this person, and it just seemed so crazy. But the problem is, I mean, this is real, right? Like, this happens. I mean, most of the time, it's a great forum, it's a great community, but some people, they are just crazy with the knee-jerk reactions they get. And I really like that. And so I reached out to a bunch of people and said, hey, so here's an idea. We're going to do a Python forum, an online forum, where people can discuss Python, you know, can bring their code and get feedback on their code and we're going to talk about all things Python, but we're going to do it in private. So it won't be a public forum, and there will be an application process to get in. And I launched that at the beginning of this year, so 2017, I don't know when the show is going to come out, but so far, it's been a really, really cool experience. We have about 120 people in the forum right now, and it is, I don't like the word safe space, but in a way, it is a safe space, where you can just go and post some code, and you're going to get, like, really, really positive and encouraging, helpful comments, and there's no danger of someone going in and being like, oh, you're the worst like, 14-year-old programmer in the world, and you should go back to where you came from. That's Pythonista Cafe in a nutshell.

01:05:32 Michael Kennedy: Nice, that's awesome. So, yeah, you've got that going. Did you do a book or videos or something for Sublime for Python developers?

01:05:38 Dan Bader: Yeah, I also did that. So I have a couple of courses that I offer on my website, and one of them is also on Talk Python Training. And so one of them is a complete course on setting up Sublime text for Python development. So, like, an end-to-end, step-by-step course that consists of a couple of modules, you know, like workbooks and video lessons on how to get a really, really awesome and productive Python environment with the Sublime text editor. Because that's my favorite editor. I know you really enjoy PyCharm. I think that's a great option too. But for me, Sublime Text has just been a great setup. And you can script it with Python, which is nice, too.

01:06:14 Michael Kennedy: It's definitely a very pythonic place to be. That's awesome. So yeah, that's cool. If people want to learn more about like, getting the most out of Sublime, they can check that out. You've got the pip Dependencies course available, both on your website and Talk Python Training, which, I thought that was really awesome. I mean, people are like, well, I know Pip. No, this is like, really, really the right way to sort of manage dependencies on your app or other packages that you're using, and it's really great. So I definitely enjoyed checking that out as well. All right, so, I think you've given us a spoiler for the last two questions here, at least the first one of the last two. Favorite editor?

01:06:48 Dan Bader: No, it's Sublime.

01:06:49 Michael Kennedy: PyCharm! Yeah, that's awesome. Yeah, I like Sublime as well. I definitely use it some of the time. Very cool, and notable PyPI package, this time around?

01:06:57 Dan Bader: Yeah, so this time around, it'll be Django, because Django 2.0 just came out.

01:07:02 Michael Kennedy: It's big news. What's the biggest piece of news about Django 2.0?

01:07:05 Dan Bader: I think the new URL routing syntax is pretty cool. It looks really nice, and for me, Django, you know, it just, it just represents so well what Python stands for. It's really, this is a piece of software that you can depend on. It's mature. And it's for when you actually want to get work done and not be worried about having to rewrite your whole stack in six or eight months down the road. And I feel like, you know, the Django team is so awesome, and it's a huge achievement of what they've done over the last couple of years. And yeah, so my pick is Django 2.0.

01:07:37 Michael Kennedy: Awesome. Yeah, the shiny Django. And the other major news around there is that that's, they've dropped legacy Python support. No more Python 2 in Django 2.

01:07:46 Dan Bader: Oh, yeah.

01:07:46 Michael Kennedy: Ironically, given the numbers there. Yeah, I think that's a super positive move. Like, just the fact that Django switched their default behavior in their documentation from Python 2 to Python 3 made a huge dent in the consumption of Python 3 on PyPI, and now, I imagine that this is just going to take it up a notch. So, it's great.

01:08:06 Dan Bader: Yeah, totally. I mean, I saw that. Like, I've been, you know, I have a couple of videos on YouTube on should I be using Python 2 or Python 3, and just a couple of months ago, you know, people are like, oh, yeah, this is a valid question. And now it's like every time I tweet this video, someone is like, there's no question about it. Like, Python 3. Like, don't go with Python 2. So, that's great news, I think.

01:08:26 Michael Kennedy: I totally agree. I feel like the last six months or so, it really has become like, why would you even ask this question? Like, this is sort of a deprecated video. You should just take it down. But that's really positive for the whole community. Awesome. All right. So, final call to action. People can get your book. They can check out Pythonista Cafe. There's all sorts of stuff they can do to learn more about this, right?

01:08:44 Dan Bader: If you're listening to the show and you found interesting what Michael and I talked about, and you want to learn more and kind of go from beginner to intermediate Python skills and beyond without complex jargon and other crazy stuff, then you should download the free sample chapters from my book. You can just go to pythontricksbook.com, or also just go to Amazon and search for Python Tricks, and the book also comes with a free bonus toolkit where people can get more than two hours' worth of video material to help reinforce some of the main lessons in the book. So, yeah, go check it out, pythontricksbook.com.

01:09:19 Michael Kennedy: Awesome. All right, Dan, thanks for being here. It's fun as always to chat with you.

01:09:22 Dan Bader: Hey, thanks for having me on the show again. It was great.

01:09:24 Michael Kennedy: You bet. Bye. This has been another episode of Talk Python to Me. Today's guest has been Dan Bader, and this episode has been brought to you by Linode and Rollbar. Linode is bulletproof hosting for whatever you're building with Python. Get your four months free at talkpython.fm/linode. Just use the code PYTHON17. Rollbar takes the pain out of errors. They give you the context and insight you need to quickly locate and fix errors that might have gone unnoticed until your users complain, of course. As Talk Python to Me listeners, track a ridiculous number of errors for free at rollbar.com/talkpythontome. Are you or a colleague trying to learn Python? Have you tried books and videos that just left you bored by covering topics point-by-point? Well, check out my online course, Python Jumpstart By Building 10 Apps at talkpython.fm/course to experience a more engaging way to learn Python. And if you're looking for something a little more advanced, try my Write Pythonic Code course at talkpython.fm/pythonic. Be sure to subscribe to the show. Open your favorite podcaster and search for Python. We should be right at the top. You can also find the iTunes feed at /itunes, Google Play feed at /play, and direct RSS feed at /RSS on talkpython.fm. This is your host Michael Kennedy. Thanks so much for listening. I really appreciate it. Now get out there and write some Python code!