#141: Python tricks Transcript

00:00 How many Python developers do you know that learned Python quickly, but then plateaued

00:04 pretty quickly as well? Maybe this is someone you worked with, or maybe it's even you.

00:09 Python's clean and simple syntax can mean it's easy to learn, but hard to master. After all,

00:14 if you learned it in a week, what else is there to this language? How much more do you need to

00:18 dig into it? Well, plenty. And Dan Bader is here to share his very popular Python tricks with us.

00:25 You'll learn to look deeper for more than just the how, but the why and when of many of Python's

00:31 more subtle features. This is Talk Python to Me, episode 141, recorded December 4th, 2017.

00:51 Welcome to Talk Python to Me, a weekly podcast on Python, the language, the libraries, the ecosystem,

00:57 and the personalities. This is your host, Michael Kennedy. Follow me on Twitter, where I'm at

01:02 mkennedy. Keep up with the show and listen to past episodes at talkpython.fm, and follow the show on

01:08 Twitter via at Talk Python. This episode is brought to you by Linode and Rollbar. Thank them both for

01:14 supporting the show by checking out what they have to offer during their segments. Talk Python to Me is

01:19 partially supported by our training courses. Python's async and parallel programming support is highly

01:25 underrated. Have you shied away from the amazing new async and await keywords because you've heard

01:30 it's way too complicated or that it's just not worth the effort? With the right workloads, a hundred times

01:35 speed up is totally possible with minor changes to your code. But you do need to understand the internals,

01:41 and that's why our course, Async Techniques and Examples in Python, show you how to write async code

01:47 successfully as well as how it works. Get started with async and await today with our course at

01:53 talkpython.fm/async. Hey, everyone. Before we get to the conversation with Dan about his Python

01:59 tricks, I want to talk about Python bytes for a second. I know many of you who listen to Talk Python

02:05 also listen to my other podcast, Python bytes, but many of you don't, and it's been a while since I've

02:11 talked about it on the show. So if you're looking for almost the opposite of Talk Python, instead of

02:17 long form, deep conversations on one topic, you want to just catch up on the news, think newsletter and audio

02:23 forum for the Python space, head over to pythonbytes.fm and subscribe to the short

02:28 news focused 15 minute weekly podcast I do with Brian Okken. I hope you guys check it out and I hope you're enjoying both

02:35 of the podcasts. Now let's chat with Dan. Dan, welcome back to Talk Python. Hey, thanks, Mike. Thanks for having

02:41 me on the show again. Yeah, I love to talk with you about technology and programming, so I'm really

02:45 excited to do it once again.

02:47 You were previously on the show for our panel with Anthony and Ronald about

02:53 Contributed Open Source, and that was really a well-received episode. That was 132. Cool, yeah.

02:57 That was a really fun experience, and I really like the panel format. You know, I

03:01 listen to your show, and those are some of my favorite episodes when you bring on a bunch of

03:05 people, and it's just, you know, the amount of information you can soak up so quickly. It's, yeah, it's just great.

03:10 Awesome, thanks. The next one that I have scheduled, I don't know when it's actually going to air, like

03:15 four or five episodes out, is a catch-up for Python and machine learning at the Large Hadron

03:21 Collider and in particle physics, so that should be really fun. That sounds intense.

03:25 It's going to be intense. That means lots of research for me, so that'll be awesome. All right,

03:29 so let's talk about not contributing to Open Source so much.

03:33 Let's talk about making our Python better with Python tricks. So that's what we're going to talk about this

03:39 week. You know, I know you talked a little bit about it in 132, episode 132, but it was the panel format,

03:45 so you didn't get to go into it too much.

03:47 So let's talk just quickly, kind of like how you got into programming and what you

03:51 do day-to-day before we get into your tricks.

03:52 Sounds good, yeah. So, well, I guess how I got into programming was when I

03:57 finally managed to convince my parents to buy a used, old, dingy Commodore 64 for me.

04:03 And, you know, that was the end of the home computing era, I guess, and those machines

04:07 all booted into some kind of interpreter directly, so you wouldn't, you know,

04:11 you don't get, like, a graphical desktop or anything. You just get a basic, the

04:15 programming language.

04:16 Yeah, that was crazy because you would, like, you couldn't just interact with the

04:19 computer. You had to, like, converse with it to get it to do things, right? It was interesting.

04:24 Exactly. It was, like, a glorified calculator and you just turned it on and you booted

04:29 instantly and then you were just sitting there like, ah, okay, I'm getting this blue screen.

04:34 What am I going to do now?

04:36 And it really forced you to pick up some basic programming skills to do anything. You know, if you wanted

04:42 to play a game, well, tough luck.

04:44 You had to actually figure out, like, how to load the game from disk and

04:48 then run it or, you know, actually type it in.

04:50 Yeah, it was, like, a load command and stuff to actually do it, right?

04:53 Exactly. Yeah, the load 8,1 or something. It's like the drive ID.

04:57 And I think I'm kind of fortunate that I caught the tail end of that because

05:02 it was already, like, the PC era and, like, Windows 3.11 or something.

05:07 But because my parents were really, really against computers and technology

05:12 in some way, the only thing I managed to convince him was to get this, you

05:15 know, used old Commodore 64.

05:16 And I got it from this guy and it came with, like, stacks of disks and notes this guy took.

05:22 And so it was just a treasure trove for me.

05:25 And, you know, it hooked up directly to your TV screen and it was just, yeah,

05:29 just this amazing, like, miracle machine.

05:32 And that's how I got into programming.

05:34 Nice. And then how'd you get over to Python?

05:36 Oh, yeah, that's a long, that's a long jump, a big leap there.

05:39 Yeah.

05:39 Right.

05:40 I think that the basic programming and basic on the Commodore 64, I think it

05:44 just kindled some, some kind of passion in me.

05:47 And eventually I went on and got bachelor's and master's degree in computer science and just, you know,

05:52 wanted to become a professional programmer.

05:54 At some point in my university days, I think it was on a ski trip with some

05:59 friends and one person, he brought a book, a Python programming book, one of

06:05 these, like, learn Python in 48 hours.

06:07 One of those that never really work out that way. And this was the first time

06:10 I really.

06:11 Was that the Sam's Learn Python on a Ski Vacation or Your Money Back Guaranteed book?

06:17 Yeah, I should ask for my money back.

06:19 Remember the Sam's book?

06:20 It's awesome. So you had the book and you had some, like, kind of downtime

06:23 after you're skiing to chill and flip through it?

06:26 Yeah, it was a very geeky skiing trip.

06:28 But this is when, the first time when I saw the actual language, you know,

06:31 just learned a little bit more about it because my friend, he actually couldn't, hadn't used Python either,

06:36 but it just seemed like an interesting new language or, you know, I guess it wasn't new at the time, but

06:40 for us it was new.

06:41 And so I just really fell in love with the way the language looked, you know,

06:46 just seemed like the perfect blend of, well, this looks really appealing, you

06:50 know, like almost like pseudo code and like in some way, like really poetic

06:53 and nice, but also really, really powerful where it's not just your basic or basic programming language, but

07:00 it's actually something where you can take it to places and you can build

07:04 full-blown applications.

07:06 And so, yeah, that's how it started.

07:07 That's awesome.

07:07 There's plenty of simple programming languages, but they often have like a ceiling where it's like,

07:13 okay, you would build this and this with it, but you wouldn't really build

07:15 anything bigger than that, right?

07:17 Where it's pretty sweet that Python generally doesn't fall into that category.

07:21 Yeah, totally.

07:21 And I think it was you, you called it a full-spectrum language at some point or like on that interview that

07:27 we did on my blog.

07:28 And this is, I love this, you know, I bring this up all the time when people ask me about Python because

07:32 yeah, it is really a full-spectrum language where someone can dive in and

07:36 they can learn the basics of Python and know enough to be dangerous.

07:40 And then you have giant, very, very complex systems built in this language

07:45 and it can kind of scale that whole gamut.

07:47 And I think that's just really cool.

07:49 Yeah, it's amazing.

07:50 What I really like about that spectrum is a lot of languages make you take the

07:55 stuff you need for the advanced type of large applications.

07:59 you got to use that syntax and that structure and those design patterns right from the start, like Java or C

08:05 sharp static main void inside of a program class.

08:08 You're like, whoa, I just want to like do a few things.

08:09 Like what's all this about, right?

08:10 Whereas Python, you like as you need the features, you layer them in, but they're

08:14 not required until you need them, which is, I think, a part of the magic.

08:17 That's a huge part of the appeal of Python that I see, you know, when people

08:20 use Python or learn about Python, but it's also kind of the dirty little

08:24 secret where it's very easy to kind of get stuck at that beginner level.

08:28 And you're like, well, okay, so how would I actually go to the point where

08:31 I could build, I don't know, something like the Instagram backend or even, you

08:36 know, think about it.

08:37 Yeah, absolutely.

08:37 And that is one of the curses of its sort of success or its features is that

08:42 it's a lot of people quickly get into it and they just get comfortable and

08:45 they're just like, well, I just sort of, I knew C or I knew JavaScript.

08:48 So now I know Python.

08:50 I just don't do semicolons or curly braces, right?

08:53 That's okay.

08:54 And it works.

08:55 But, you know, I think our topic today is really about understanding like where

09:01 people get stuck and then going, okay, these are actually how you should go

09:05 farther and do it correctly or Pythonically or whatever, right?

09:08 Yeah, pretty, pretty much.

09:10 I mean, this is, I feel like this is the biggest challenge in taking the next

09:14 step with Python.

09:15 Like how, you know, what does it even mean to write Pythonic code and how,

09:19 how do you get there?

09:19 because it's such an opaque concept and yeah, and whatever can be done there,

09:24 I think it's going to make people's lives easier and it's going to make them more successful as developers.

09:28 Yeah, absolutely.

09:28 So let's talk about your book, Python Tricks.

09:32 It's a buffet of awesome Python features, which is really nice.

09:36 And you released it a little while ago in a digital only version on your site,

09:42 debater.org, right?

09:43 Right URL?

09:44 That's right.

09:45 Yeah.

09:45 Yeah.

09:45 And you also released it on Amazon and it kind of went a little bit crazy on

09:50 Amazon.

09:51 Well done, man.

09:52 Thank you.

09:52 How did it do?

09:53 Like right now, if I'm looking here, it's number 21 and all the books in

09:58 Python, like that's amazing for such a new book, but it was much higher, right?

10:03 Yeah.

10:03 So crazy enough, it actually hit the number one spot in Python programming

10:08 and in programming languages in general.

10:12 At some point it was even, it was book, it ranked number 250 in Amazon sales

10:18 rank across all books on Amazon.

10:20 And, you know, I actually, I took a screenshot of that and this is like.

10:24 Printed out, put it on the wall in frame.

10:27 This was like visible proof to my in-laws that I was just, you know, I was not,

10:31 not unemployed, not just unemployed and addicted to the internet.

10:35 It's like, here, I'm an author on the internet.

10:39 This is what I've been doing in my office.

10:41 I promise you I have a job.

10:43 Yeah, it was a cool experience.

10:44 That's really cool.

10:45 Congratulations.

10:46 Thank you.

10:46 Yeah.

10:47 Yeah.

10:47 And so it's, it's still going strong.

10:49 Like it's number 29 in web programming and stuff.

10:52 So yeah, right now.

10:53 Very cool.

10:54 So we'll definitely link to your book from the show.

10:56 So maybe give us the quick elevator pitch.

10:59 I mean, we've kind of been building up to it, but what's the, what's the

11:02 idea behind the book and what are you talking about in there?

11:04 Well, like you said, the, the subtitle for a book is a buffet of awesome Python features.

11:08 And so basically the book is a collection of actionable tips that will help you write clean,

11:15 professional and developer style, quote unquote, Python code.

11:20 And, you know, the feeling that I want to create for my readers is not like they're sitting through an

11:26 advanced computer science lecture where I'm throwing a bunch of jargon at you, but more like we're

11:30 on the same development team.

11:32 You know, we're sitting down together, you know, maybe know the basics of Python.

11:35 And I want to help you get up to speed as quickly as possible and help you grasp some of the

11:39 intermediate and more advanced Python topics that often seem weird and opaque, you know, things

11:45 like decorators, first class functions.

11:47 What's the difference between an iterator and a generator and generator expressions and list

11:51 comprehensions.

11:52 And just some of the really useful things in Python that border on the arcane, but they're really not.

11:59 If you, right.

12:01 If you approach them from the right perspective.

12:03 Sure.

12:03 It's a lot of these things that, you know, I feel like Python is, you can learn the language really quick,

12:08 but it takes a long time to really master it.

12:11 And part of that mastery is like discovering these little trade-offs like I could use a tuple, but oh, by

12:17 the way, did you know there's a name tuple?

12:19 And did you know in Python three, six, there's a new version of a name tuple that is more flexible, but

12:23 similar in performance and like, oh, wait, there's a new name tuple.

12:27 What is that?

12:27 Right.

12:28 You don't really easily fall into that discovery or things like, yeah, you can use list as a queue, but it's

12:34 like a thousand times slower than if you use this other proper thing for

12:39 queuing.

12:39 Oh, but don't use that for multi-thread and use this other one for, for parallelism and things like that.

12:44 Right.

12:44 Those types of things are somewhat arcane, but they're, they're not easily discoverable.

12:49 I don't feel.

12:50 Exactly.

12:51 you know, where it's a lot of times like when someone has some experience with other programming

12:55 languages and they're switching to Python, it can be really hard to understand how these different

12:59 concepts map to Python, because I guess Python uses a very like human friendly naming style for a lot of

13:06 things, you know, like you just mentioned the queue and list example, but if you're coming from,

13:11 let's say a Java background, how does a list map to what I know from, from the Java world, you know, what,

13:17 what kind of time complexity guarantees does it make and, and kind of, you know, touching on some

13:22 of these things as well.

13:23 Like how can you take some existing algorithm and actually bring it over

13:26 to Python and to make sure, you know, it doesn't just look pretty, but it actually also really works

13:31 and is fast enough.

13:32 Yeah, absolutely.

13:33 And one of the, you know, one of the things even more than like list versus say array list in Java is

13:38 array, right?

13:40 Like there's not really like the traditional C style array in Python in the built-in language sense.

13:46 There's the, you know, array dot array class, but there's not the like, square bracket means something

13:52 different than list, right?

13:54 Like those are actually the same thing.

13:55 And that just gets confusing, but I think we should probably just start going through some of your individual

14:00 tricks that you thought are particularly like noteworthy in at least a radio format is.

14:06 That'll be interesting.

14:07 Yeah.

14:08 It's so we'll try to not talk too much about code, but, you know, cover some of the ideas behind it.

14:13 All right.

14:13 So actually, you know, the first one you said, let's start with something

14:16 simple.

14:16 And I totally agree that it's a good place to start.

14:19 You talk about comma placement for dictionaries and lists and just like structuring those for basically for both

14:28 avoiding errors and making them source code control friendly.

14:31 What's the story there?

14:32 It started with sort of this being a pet peeve of mine.

14:34 Like if you use lists or any of the other built-in collection classes like a

14:40 dictionary or set, and you're trying to define a constant in your code, like,

14:45 let's say, you know, a list of names or a set of some other objects.

14:50 If those get too long and they don't fit into a line, like usually, you know, according to pep8, we're spacing

14:55 them out and we're kind of spreading them out across line breaks.

14:58 Now, when you do that, this usually looks pretty good.

15:02 But then when you add an element to the end of the list, you've got to make sure you have your trailing

15:08 comma right in that list, because if there's no trailing comma, then and

15:13 you're working with strings, for example, cpython, the interpreter or the parser is actually going to mash

15:18 two consecutive strings together into one.

15:21 So you're going to end up with a completely different value.

15:23 And it's not a it's not a interpreter error or parser error or syntax error.

15:27 But this you actually get the wrong output.

15:29 And so this if that makes sense so far.

15:33 So if the first if you had a list of four for names, name one, two, three,

15:39 and four, and you forget the comma because you added a new line, but you

15:41 forgot the comma, it's what's in the list is name one, name two, name three

15:45 and four as one string, like as if you would put a plus there.

15:48 And that drives me crazy about Python.

15:50 Like, I understand, OK, there's a few conveniences, but the ability to introduce bugs by having two string

15:57 variables next to each other and forgetting a comma or something is really frustrating.

16:02 It is totally frustrating.

16:05 Yeah.

16:05 And so this is something that I saw come up in code reviews.

16:09 you know, as I was, we were at the time like onboarding two new developers

16:14 and this actually came up in the code review and we're like, what's going on?

16:17 Like, I hate this language, right?

16:19 Like it was one of these things where you run into it and you're like, well,

16:21 why did this just happen?

16:22 And it's like a really nasty error or bug that you can introduce into your

16:26 programs.

16:27 And so basically we came up with this rule of like just ending every single

16:31 item with a comma because you can in Python, you can actually have a trailing comma, even in the last and

16:37 the final item in one of these collection constants.

16:41 Right.

16:41 The dictionary, the list, whatever, right.

16:43 It can always have that comma there.

16:45 The other benefit is if you don't put the trailing comma and you're going to

16:50 create new items in that list and you look at a diff in source control, it

16:55 looks like the previous item has changed as well because that line changed with a comma, whereas it only

17:00 changed so you could extend it.

17:02 But if you always put the comma there, like literally the lines that changed are the lines that changed.

17:06 Yeah, that's right.

17:06 You can keep your your diffs, your get diffs nice and clean.

17:10 And I mean, you know, I don't know, like this probably makes us sound like

17:13 super nitpicky.

17:14 You know, if you're like, oh, you're you're like the senior developer reviewing someone else's code, you're

17:19 pointing out stuff like that.

17:20 But honestly, I think a lot of times it's a little things, right?

17:23 Like if you see like it's a little bit hard to talk about this on, I guess, on a podcast where we can show

17:28 people code examples.

17:28 But if you see this in front of you, usually people go like, oh, yeah, that makes sense.

17:32 I'm just going to start doing that because you can just remove that class of errors or that potential

17:38 for errors there.

17:39 You can just remove that completely just by making one tiny, tiny, small change to your coding habits

17:44 or how you write your code.

17:46 It's super simple to do and understand how it works is great.

17:48 The other place that that's really helpful is for code generation.

17:52 So imagine you've got like a cookie cutter template and you're cookie.

17:56 you enter some values and that's powered by Jinja, too.

17:59 So you put some very like give it a list and it'll generate like it could

18:03 generate a bunch of code that's going to generate your starter project.

18:06 And because you can trail everything with a comma, you don't need some weird

18:09 if case to like filter out the last comma and just throw that puppy in there and it's all good.

18:13 Oh, that's nice.

18:14 Yeah, I see.

18:15 That would make sense.

18:15 Yeah.

18:17 This portion of Talk Python to me is brought to you by Linode.

18:20 Are you looking for bulletproof hosting that's fast, simple and incredibly affordable?

18:25 Look past that bookstore and check out Linode at talkpython.fm/Linode, L-I-N-O-D-E.

18:31 Plans start at just $5 a month for a dedicated server with a gig of RAM.

18:35 They have 10 data centers across the globe.

18:37 So no matter where you are, there's a data center near you.

18:40 Whether you want to run your Python web app, host a private Git server or a file server, you'll get native

18:45 SSDs on all machines, a newly upgraded 200 gigabit network and 24-7 friendly support, even on

18:51 holidays and a seven day money back guarantee.

18:53 Want a dedicated server for free for the next four months?

18:57 Use the coupon code Python17 at talkpython.fm/Linode.

19:01 Let's move on to sort of a more protective, defensive way of coding by covering our assets with asserts.

19:11 Yeah, the assert statement.

19:14 Yeah.

19:15 So you talked about the assert statement, which I don't see being used all that much in the code that I run

19:20 across.

19:20 Maybe it's just what I've been looking at, but I don't see it that often.

19:23 Yeah.

19:23 And I think that's a shame.

19:25 I don't know.

19:25 Like I always felt the assert statement in Python was really, really useful.

19:29 I feel like it's underutilized.

19:31 It can really help people write while making their code easier to debug.

19:36 First and foremost, you know, it's like a defensive programming technique.

19:41 And basically what an assert statement does, it's like a shorthand for testing a condition.

19:47 And then if the condition is false, it will automatically raise an exception.

19:51 It will just throw an assertion error.

19:54 So you can use assert statements in your code just to quickly test conditions.

19:58 If you don't want to write, you know, like a longer if statement and really, you know, maybe

20:03 define a custom exception or anything like that.

20:04 You just want to make sure that, for example, a product price, it shouldn't be negative or

20:08 something like that, where you don't really expect this to happen, but you kind of want

20:12 to make sure that it can never actually really happen in production because crazy things

20:16 sometimes happen in production.

20:17 Then this is just a great little tool or little feature in Python that you can use.

20:23 Yeah. And I really like the little examples that you give.

20:25 So like when you introduce assertions, you talk about like, let's imagine you have a store, say

20:30 for buying online classes or something, and you want to set up a discount coupon type of

20:34 mechanism.

20:34 And so you might assert that after applying the discount, the price is no less than zero and no

20:40 more than the actual maximum price.

20:43 So you don't get like a negative, negative coupon or something.

20:46 Charge them more than if they didn't enter it.

20:48 The worst coupon in the world.

20:50 Please don't put that in.

20:53 Yeah. So that's, those are all these little examples that you have in the book are really

20:56 great. And so like asserting after you apply the discount code that the price is zero to the full

21:02 price is pretty straightforward.

21:04 So you do talk about some really interesting things in here. One is that the idea behind assertions is

21:11 not the same as proper error handling.

21:14 And that these are really meant to handle conditions that are like bugs, not just invalid

21:20 input.

21:20 That's right. Yeah. You want to basically use an assertion. If you want to inform yourself or fellow

21:26 developers about an unrecoverable error in your program. So I like to think of them more as a

21:32 debugging aid rather than just, you know, a glorifier, like a fancy or weird way to throw an exception.

21:38 So this is really, you know, your app is probably still going to come down crashing, but at least you will know

21:45 exactly what cost your application to crash.

21:48 So it's usually going to make it easier to have your application, to have it fail fast, right?

21:54 When it hits that, that condition that it should not, or when it goes into a state that it should not be in rather than

22:00 having that carry on.

22:01 And then at a later time, you know, cause trouble further down the road. And so using an assert statement or using multiple

22:06 assert statements, that's a great way to just have these little checkpoints in your code that you can just sprinkle around these

22:13 little self checks and they can really save you a ton of time because they give you all the context you need.

22:19 They make it easier for you to have that context to debug your programs because of course, nothing is a silver bullet here.

22:26 So yeah, sure.

22:27 But I do think it's good to have, like you said, that context, because a lot of times maybe the website would start crashing

22:34 because Stripe is rejecting the price that you're trying to charge when the price is negative.

22:39 And you're like, what is going on here?

22:40 Why is this happening?

22:41 Right.

22:42 Whereas if you had an assertion that said, well, like when we applied the discount code, the price went negative.

22:46 Like, oh, I see.

22:48 I know what's going on here.

22:49 So it's really nice.

22:52 A final warning that you had in here is you should not use assertions for regular checks, right?

23:01 Like if you ask a person to enter a number between zero and 10 and they enter 11, you shouldn't like assert that that's

23:07 between zero to 10, right?

23:08 Because they can be turned off the dunder debug setting that you talk about.

23:15 And also they're, they don't, don't tell you anything about it from a perspective of the exception, right?

23:21 It's always an assertion error, not like a negative number custom exception thing you made or something.

23:26 Yeah.

23:26 You can add a message to the assertion that will also be part of the exception stack trace.

23:32 But, but yeah, like you said, it's, it's pretty dangerous to use assertions for actually handling conditions or checking for

23:40 conditions that you expect to happen as part of the normal program flow.

23:43 So for example, if you do your user validation with an assert statement, you know, if you go assert user is admin or something

23:51 like that, well, that's going to work great.

23:53 But as soon as these assertions are turned off and they can be globally disabled at the interpreter level, then all of a sudden, you know,

24:01 everyone can log into your app and do all kinds of crazy things because all of a sudden everyone is an admin.

24:05 And so you don't want to do that.

24:07 Yeah, that's for sure.

24:08 Yeah.

24:09 You would basically be able to pass a flag that turns off all air handling and preconditioned checking.

24:14 That might be bad.

24:16 Do you know why they are statements and not functions?

24:19 I remember reading about that.

24:21 I hope I didn't actually mention this in the book and now I forgot about it.

24:24 I don't think you did.

24:25 It seems to me like maybe one of the reasons is if you do turn them off, they can like literally vanish from the code and not just become

24:32 operations that don't do anything.

24:34 So maybe they're easier to like factor out.

24:36 I actually have no idea, but it drives me crazy.

24:39 Like there's like weird conditions about the tuples versus parameters and

24:43 all sorts of funkiness because they're not just like actual calls, function calls.

24:48 That's right.

24:49 It's very easy to write assertion statements that never fail because you

24:53 would somehow intuitively think this feels more like a function call, sort of like the print statement versus the print function.

25:01 But if you write a syntax or if you use this like a function and you put

25:07 parentheses around it, Python can actually interpret it as this as a tuple.

25:11 So it's like the assert statement plus a tuple expression.

25:17 This will always be so this will also be true or truthy.

25:21 And so, yeah, you can you can shoot yourself in the foot that way.

25:25 But I think this is now actually a warning in Python 3.

25:29 Yeah.

25:29 Yeah.

25:30 You do get a warning in at least 3.6.

25:32 I can tell you that for sure.

25:33 So that's good.

25:33 Yeah.

25:34 Yeah.

25:34 But I'd rather have just the language, actually, you know, just not let that be possible in the first place.

25:40 All right.

25:41 So let's move on to the next one.

25:42 The shocking truth about string formatting.

25:45 It's not very Zen-like, is it?

25:47 Yeah.

25:47 No, that's true.

25:49 I mean, this is one of the realizations I had when the format strings or F

25:53 strings landed in Python 3.6.

25:56 I was like, hold on a minute.

25:57 Like, there's like four different ways to do string formatting in Python now.

26:01 And variations on each four of those.

26:03 So which one should I actually use?

26:07 And it seemed to go against the Zen of Python as well.

26:11 And, you know, I guess it's one of these things where, well, languages, they evolve over time

26:16 and we can't just throw out, you know, all of the legacy code that we have.

26:20 So it's like a trade-off that we need to make or people need to make.

26:23 But still, I just thought it was interesting.

26:25 And yeah, so let's chat about that.

26:28 Yeah, for sure.

26:29 So we've got the old style formatting, the like string with a percent, a format character.

26:36 So percent S, percent D, and so on, percent F.

26:39 Then mod the values or a tuple of the values to be inserted.

26:44 And that's kind of more or less not the way these days, right?

26:49 Yeah.

26:50 I mean, it's called old style formatting for a reason, I guess.

26:53 But yeah, so we have a newer way of doing string formatting.

26:59 And it's called, it's referred to as new style string formatting using the string dot format function.

27:04 But it's not actually the newest style, but it's the middle age.

27:08 And the old style isn't actually...

27:11 It's not deprecated, yeah.

27:12 It's not going to go away, right?

27:13 So I guess it's just a hint that we got from the CPython team that, you know, really you should be using...

27:21 Maybe you shouldn't be using the old style of string formatting.

27:22 On the other hand, it is kind of nice to use.

27:25 Yeah.

27:26 And it's, you know, the new style is still in Python 2.7.

27:30 So new style would be like, hello, curly, curly, all that in quotes, dot format name or something like that.

27:36 And that's pretty standard.

27:38 That's really nice.

27:39 Takes a lot of sort of its own formatting mini language to convert the values like numbers to digit grouping and things like that.

27:47 And then we've got the f-strings, which are the new hotness in 3.6, right?

27:52 Yeah, the f-strings are just awesome.

27:55 They were added in Python 3.6.

27:56 And I think it's a really, really great feature.

27:58 The f-strings are also called formatted string literals.

28:01 And people also refer to this as string interpolation.

28:04 So really what this means is it's basically a new type of string.

28:08 So you just put a lowercase f in front of your string.

28:12 And then you can use curly braces to just have a Python expressions right in your string.

28:20 And when Python evaluates that string, it's actually going to evaluate those expressions, convert them to a string, and then merge them with the surrounding pieces or parts of that string.

28:30 So this is a really, really nice and concise way to define strings in Python that are dynamic.

28:37 And it does the same job than all of the other formatting tools that we talked about or formatting styles.

28:42 But I find this to be very readable and just very, very nice to work with.

28:46 Yeah, I really like it too.

28:48 The more I use it, the more I really want to use it everywhere.

28:50 It's great.

28:51 And so you would say F quote and then have a regular text and then curly curly.

28:57 Then, like you said, you put these expressions and they don't.

29:00 It looks very similar to the dictionary version or the keyword value version of the old format or the new old format or middle-aged format.

29:09 But you can do things that are unexpected.

29:10 Like you could actually call functions inside there.

29:14 You can add up things.

29:16 You can, what I didn't realize is you can actually put format descriptors like some variable colon hash X for hexadecimal representation.

29:27 And that's pretty cool.

29:28 It's pretty neat.

29:29 And I think it's also nice and readable because you just have all of the context you want.

29:35 Like one of the challenges I've felt with, you know, at least old style string formatting format is that a lot of times you have to jump back and forth.

29:43 And then, you know, if you have a complicated format string, you have to actually count out like, okay, this percent S, a placeholder, it refers to this other variable.

29:52 And you just have to do all of that, that mental gymnastics to figure out what's actually going on.

29:57 And with f-strings, well, you see it right in front of you and it's kind of nice.

30:02 Yeah.

30:03 It eliminates a whole class of bugs.

30:04 Like it's totally possible to have too few arguments in like a format expression or too many arguments or get them in the wrong order.

30:13 But when the actual variable goes in the spot, like that's what's in the spot, right?

30:18 There's no more debate in it.

30:19 Yeah.

30:19 I guess the one thing that is kind of nice sometimes is in the string, you have to repeat something that's like sort of an expression that got evaluated.

30:27 Like that might be in the middle.

30:28 And so I'm going to propose some nomenclature here and we'll see if the world will take it.

30:33 I don't know.

30:34 So let's call the old style.

30:37 Old style is finer.

30:38 It could be legacy style formatting.

30:40 I think that's even a better word for it.

30:42 The new style, what should we call that?

30:45 What do you think?

30:45 The traditional string formatting style.

30:49 And then the interpolation, we could call that the new style.

30:52 The new style.

30:54 Yeah.

30:54 Let's do that.

30:55 Let's go and submit a pull request on the Python, CPython docs.

31:00 Exactly.

31:01 Let's do a PEP to change the naming.

31:02 Exactly.

31:04 Exactly.

31:05 And so really quick.

31:06 The last one is one that I really hadn't played with, which is it's sort of the little Bobby tables equivalent of string.

31:13 So it's something called string templates.

31:16 And it's really good for when you're like taking an untrusted input and jamming it into a string format.

31:21 Yeah, this is great when you're dealing with user input.

31:24 You know, when you have something else that comes in externally where users can actually define those templates.

31:30 Because the problem is if you just feed that into one of the other formatting techniques or methods, people can actually execute code in your interpreter environment.

31:40 So there's a way to reach out and look at global variables.

31:47 I mean, I'm in Ronaker's blog where he demonstrated that technique.

31:51 And basically you can go in and there's a way to reach out from a format string and to reach into the Dunder globals dictionary and just to look up any variable in the global variable environment.

32:06 So, you know, if you have your secret API key in there and this is a web app, then, well, bye bye API key.

32:12 It's just going to, you know, people can leak it out.

32:15 What's my name?

32:16 My name is global square bracket AWS secret key.

32:19 Oh, look, there it is.

32:21 Yeah, that's bad.

32:22 Yeah.

32:22 Okay.

32:23 Yeah.

32:23 So you can use this template thing.

32:24 And so the template strings, they prevent that.

32:26 Yeah, nice.

32:27 They're kind of like parametrize queries in SQL expressions.

32:30 Like they don't really get evaluated.

32:32 They just get put in there as raw strings, but they kind of signal that this part is input and it doesn't run.

32:38 And this, it just goes here as text.

32:41 This portion of Talk Python to Me has been brought to you by Rollbar.

32:45 One of the frustrating things about being a developer is dealing with errors.

32:49 Relying on users to report errors, digging through log files, trying to debug issues, or getting millions of alerts just flooding your inbox and ruining your day.

32:58 With Rollbar's full stack error monitoring, you get the context, insight, and control you need to find and fix bugs faster.

33:05 Adding Rollbar to your Python app is as easy as pip install Rollbar.

33:08 You can start tracking production errors and deployments in eight minutes or less.

33:13 Are you considering self-hosting tools for security or compliance reasons?

33:17 Then you should really check out Rollbar's compliant SaaS option.

33:21 Get advanced security features and meet compliance without the hassle of self-hosting, including HIPAA, ISO 27001, Privacy Shield, and more.

33:30 They'd love to give you a demo.

33:31 Give Rollbar a try today.

33:33 Go to talkpython.fm/Rollbar and check them out.

33:37 So the next one that you talked about is exceptions.

33:41 And there's all sorts of stuff around exceptions.

33:44 I mean, there's the whole embracing the look before you leap, rather disregarding the look before you leap style from C and embracing that it's easier to ask for forgiveness than permission.

33:53 And just really using exceptions as, like, for exceptional cases, right?

34:00 But then there's also making sense of the consuming the APIs that have been really embracing that format.

34:07 So Crete, your recommendation is to define your own exception classes to make your code more readable, self-documenting, and so on, right?

34:15 Under some circumstances, I think that makes a lot of sense.

34:17 I feel like a lot of times you're totally fine if you just, let's say, raise a value error or a key error.

34:23 If it's just perfectly clear, you know, if you're working with a small function and this is not some perfectly polished library that you're releasing to the world, you know, just go ahead and raise a value error.

34:33 Don't bother with defining your own exception hierarchy.

34:36 But I think as soon as you have a larger piece of code that you want to share, either across multiple applications or just, you know, kind of have it nicely served in a well-defined package where it's its own package, its own module, then I think it makes sense.

34:51 What do you think about this?

34:52 What if you have your rule as if it has a setup dpy, it has its own exception hierarchy?

34:57 I think that makes a lot of sense.

34:58 Yeah.

34:59 I think that's a package.

35:00 That's a good idea.

35:00 People consume it externally.

35:02 Maybe it should be more formalized in this way.

35:05 Yeah.

35:05 Yeah.

35:06 At the very least then.

35:07 And also if you're working in a bigger application, you know, where you have a lot of code, like I don't do this in any little script that I write, but if you have a larger body of code, I feel like these are the things that really make it easier for you also to work with your code in the future, because it's just nicer or it's easier for you to see, you know, what went wrong and where.

35:27 Essentially, a value error can come from many, many places.

35:31 But if it's a password validator exception that you defined, then, well, you know, this has something to do with password validation or, you know, the next person working on your code knows that it has something to do with that.

35:42 Another thing that I think is really nice.

35:43 Another thing that I think is really nice that you talked about is actually having a base exception that defines, like, these are all the errors that come out of this particular package.

35:53 So that makes me crazy.

35:55 Sometimes I'll be like handling errors out of different packages.

35:58 I'm like, what is this thing throwing?

36:00 Is it throwing one of its own errors?

36:01 Is it throwing like a system socket error?

36:04 Like, what is this?

36:06 And how do I find a way to catch it?

36:07 You know what I mean?

36:08 It'd be really nice to go, okay, I'm working with the SQLAlchemy library.

36:13 So I'm going to catch, I'm going to start by catching SQLAlchemy errors and like dealing with that.

36:18 And then I'll start to add on special cases of like, oh yeah, this is when the network is down.

36:22 This is when the database doesn't exist or, you know, whatever.

36:25 Yeah, exactly.

36:26 And some people start using value error or something like that, like a built-in exception for that.

36:31 And then just put that information in the exception message.

36:35 And I mean, for debugging purposes, I think that's okay because, well, you're just going to look at the stack trace and you know, sort of know what's what happened.

36:43 But you really don't have the granularity when you're catching these exceptions, right?

36:49 When you have a try except block and like actually exception is part of your structuring the control flow of your program.

36:56 Then that becomes very dangerous because then, well, you kind of have to parse out that string.

37:01 And I don't know, I don't like that.

37:03 Yeah.

37:03 Well, and also if you're trying to say, well, I'm trying to deal with the case when there's a value error, but is it from this library or that library?

37:11 Because that means really different things.

37:13 One is the network is down.

37:14 The other is you don't have permission.

37:16 And I don't really know what to do here, right?

37:18 So you can totally help that.

37:20 And, you know, another thing that I'm not sure people think very much about, but I think would be really helpful is like discovering the solution to some common problems.

37:30 So if it's a generic standard library error and you go search for that on like Google on Stack Overflow, I use Rollbar and Rollbar has this sort of mechanism where you go into the error reporting for your web app.

37:43 And it says here are the common solutions or issues that are found associated with this error.

37:48 But if it's super general, it's like, well, nothing, right?

37:50 But if it's pretty specific errors, like you could probably get help more easily on lots of levels that way.

37:56 I really like your thinking there.

37:58 It's like this whole meta game of programming, right?

38:01 Where you're not just fending on your own, like you're fighting the machine there.

38:05 But a lot of times it's just great if you can punch something into Google and just at least find some other people who have the same problem.

38:11 And yeah, I think it really increases the Google ability of your exceptions if you do that.

38:18 Yeah, absolutely.

38:19 Totally agree.

38:20 All right.

38:21 So let's talk about naming a little bit.

38:23 I know that you've talked about this on your blog and some articles.

38:26 I've talked about this, Brian Okken on Python Bytes.

38:29 There's a special meaning for underscore and double underscores and prefix underscores and suffix underscores in Python.

38:37 And maybe we should just touch on those a little bit.

38:39 There's like five or so that you cover in here, right?

38:43 So many underscores.

38:44 Yeah, Python does love the underscore.

38:46 I'm getting undersource from all these underscores.

38:48 So the first thing that we can start off with is like a semi-private variable, underscore var.

38:55 Yeah.

38:56 So the underscore var is just a naming convention.

38:59 And I like that you called it semi-private because a lot of times when people come from, let's say, a Java or C# background,

39:06 they assume that private really means private in this context.

39:09 Like you cannot, you know, reach in and modify this variable.

39:12 But in Python, if you do underscore var, it's a naming convention.

39:17 You know, it's just something that the community agrees on.

39:19 It's like a hint to the programmer, but it's generally not enforced.

39:21 So that's something you got to keep in mind.

39:24 Like we're all consenting adults or whatever Guido said about that.

39:28 Exactly.

39:28 I've heard him describe it that way as well.

39:31 But then there's the, yeah, but I still don't trust you.

39:35 So I'll put another underscore in front of it.

39:37 Yeah.

39:37 And that's where it gets interesting because that enables or that kicks, that makes the name mangling kick in.

39:45 So if you do a double underscore in front of your variable name, the Python interpreter will actually mangle that name and prepend the name of the surrounding class to that variable.

39:58 So this sort of kind of makes it like a private variable, but not really, because if you do the mangling yourself, you know, if you just do that string concatenation in your head or in your program, you can still reach in and modify that variable.

40:12 So it's really more a mechanism for preventing naming collisions, but not, you know, not a way to implement private variables.

40:19 Not true private.

40:20 I mean, it's not private in the sense that it's private, but there's still really not that much private in all the other languages.

40:27 So if you think of say C#, it has a private protected internal, all these different mechanisms and scoping rules.

40:34 But then on the other hand, you have reflection and you could just go ask for it for the private variables.

40:39 Java has introspection C++ they have private, but you know, how many bytes offset is that?

40:45 Okay.

40:45 We just go to the front of the object and like read that form over.

40:49 And we, I mean, like in all of these languages, there's a backdoor to these privates.

40:54 Right.

40:55 And so I feel like this is really no more or less than that.

41:00 It's just maybe slightly easier if you know the convention, but it certainly screams to you.

41:06 Don't do this.

41:07 Yeah.

41:08 It's like you're, you're digging a moat around your, around your fortress.

41:12 And, you know, of course people can still overcome it and destroy your beautiful programming fortress,

41:18 but you're just making it a little bit harder for people to hurt themselves.

41:21 So yeah.

41:22 I mean, the whole point is like people shouldn't mess with the internal implementation.

41:26 They shouldn't depend on the internal implementation.

41:28 And if they do, they're going to suffer.

41:31 Not you probably unless they fire out, file a bug and you got to deal with it.

41:34 Yeah.

41:34 Yeah.

41:34 Yeah.

41:34 You would hope so.

41:35 Right.

41:36 That it's not you as the maintainer who gets the flack for that.

41:40 Yeah, exactly.

41:40 But I do think like that's kind of far enough.

41:43 I use that every now and then.

41:45 I also use it for when I want clean autocomplete.

41:49 If there's some library that I'm like, I want to give this to somebody.

41:53 I want them to say thing dot and the list of the stuff I want them to work with really clearly.

41:58 I'll put that either the underscore, the double underscore, not so much to make people stay away,

42:03 but to like make the API more discoverable in the tooling.

42:07 Oh, that's nice.

42:07 That's a good idea.

42:08 Yeah.

42:09 And the double underscore, like it takes it entirely out of the list.

42:11 So I'm kind of like, ah, you know, if it's not going to bother me too much, I might do that.

42:14 But that's one of my motivations.

42:17 I did want to ask you with the single underscore, that means like internal private.

42:23 But do you think that means protected or truly private, like only in the class or classes and

42:28 its derivatives?

42:29 That'll come down to the individual developers and their teams agreement.

42:34 Personally, I use it more like a protected variable, I think, where it's like, yeah, you

42:41 know, if it makes sense to reach in and modify this.

42:44 And we're part of the same, you know, if it's related code, very, very closely related code,

42:50 then I feel like it's okay.

42:52 But it's definitely one of these things where it's it's a it's a slippery slope, right?

42:55 Or it's like, oh, yeah, it'll be fine.

42:57 And it's like one of these famous last word things.

43:01 It's this convention.

43:02 And it's not super explicit, right?

43:04 Like, yeah.

43:04 But I think it's, I think it's good to know about it.

43:08 So just really quickly, maybe touch on the other underscores.

43:11 There's var underscore the suffix.

43:13 Yeah, so that's mostly just used to get around.

43:16 So, you know, you can use a reserved word, like a Python keyword, like class, you can use

43:23 that as a variable name in your programs.

43:24 But a lot of times, it actually makes sense to have a variable name, name class, if you're

43:29 passing in some kind of class to a function.

43:31 Yeah, the place where I've seen it is in Beautiful Soup.

43:34 You can do a find and specify a class keyword.

43:37 And then you say the name of the class you're looking for in the HTML.

43:40 But you can't say class, right?

43:42 So they have class underscore equals as the keyword argument.

43:45 Yeah, there you go.

43:46 And then people sometimes do, they just kind of mangle the name themselves.

43:50 So they maybe call it class with a K or with a Z.

43:53 Yeah, with a Z.

43:54 I actually went and dug through PEP 8.

43:58 And so what people recommend or what like the official recommendation is to just end

44:03 your variable name with an underscore.

44:06 And that disambiguates it and make sure it doesn't conflict or there's no conflict with

44:11 the keyword or the built-in reserved word.

44:13 And I kind of like that because I feel like, yeah, it's pretty clear, pretty easy to remember.

44:16 And we have more underscores that way.

44:19 Isn't that wonderful?

44:20 Yeah, we have more.

44:21 All these languages have ways of dealing with them and none of them are really amazing.

44:26 This seems just totally, totally decent.

44:28 And then we have the Dunder methods, which is really the Python data model magic methods,

44:34 which I know you don't like that term, but.

44:36 Do you like that?

44:36 I was going to ask you, do you like the word magic methods?

44:39 To me, they feel like operators.

44:41 I don't know.

44:42 I don't really like the term either because magic seems like they're inaccessible and

44:47 they're like these internal black box things.

44:49 They're not really right.

44:51 And then the final thing around underscores, because Python loves underscores, is to use

44:56 it as the, I don't care about this variable.

44:58 Don't lint it.

45:00 Don't make me name it.

45:01 I don't care about it.

45:03 Yeah.

45:03 It almost has like a Haskell kind of feel to it where you're like, oh yeah, I'm so cool.

45:08 I'm ignoring all these variables and I'm deconstructing tuples and whatnot.

45:13 Yes, exactly.

45:13 Yeah.

45:14 So if you want to like loop over something like 10 times, you could say for n in range

45:22 one to 11 or zero to 10 or whatever.

45:24 But if you don't care about that n, you know, you might get warnings like that variable is

45:29 not used and whatnot.

45:30 And if you want to be explicit and say, I don't care about the loop variable.

45:33 I just want to do this 10 times.

45:34 Just put an underscore instead of n and everything's good.

45:36 Same thing for like parameters.

45:38 Yeah.

45:38 And methods.

45:39 Yeah.

45:39 The only thing you got to be careful about is when you do that in a Python REPL session

45:43 where the underscore, like a single standalone underscore is usually represents the result

45:49 of the last expression that the interpreter evaluated, which is kind of nice if you're using,

45:54 you know, if you're working with the REPL, but if you redefine it, that can cause all kinds

46:00 of trouble.

46:00 So, you know, that's kind of the one caveat with using the single underscore.

46:05 That's for sure.

46:06 And it's nice because if you type an expression like, oh, geez, I would love to use that as

46:09 a variable.

46:10 You can say variable name equals underscore and then roll on from there.

46:13 That's nice in the REPL.

46:15 Yeah.

46:15 All right.

46:16 So let's move on to data structures.

46:18 And this is like a pretty significant chapter that you talk about.

46:22 And I kind of hinted at this at the opening.

46:25 Like you can come in and find the really basic structures really easily, but there's actually

46:29 a rich set of data structures that are special made for certain circumstances and situations.

46:36 So let's dig into those.

46:37 Maybe, yeah, start with the dictionaries because that's a pretty central thing in Python.

46:42 Yeah.

46:42 It's like the central data structure in definitely, I think, in Python and in, I feel like probably

46:47 in programming in general, it's like just a super versatile tool.

46:51 Yeah.

46:51 And it's not just something you use in Python, right?

46:53 Like the variables and attributes in a class are defined by those.

46:59 You know, the stuff in module, like global variables, all sorts of stuff, right?

47:02 Like it's the structure that stores, that holds the types of Python together, I guess.

47:07 Like the stuff the universe is really made off.

47:10 It's the dark matter of Python.

47:14 I like that.

47:15 Yeah.

47:16 And it's, I mean, it's everywhere, you know, but on the other hand, when we started, you

47:21 know, the last place I worked at, we were interviewing pretty aggressively and by aggressively mean,

47:25 you know, interviewing a lot of people and trying to hire Python developers.

47:29 And so one of the things that came up over and over again in those interviews that was that

47:34 people were really struggling with when you're talking about like basic, like hash table or

47:38 hash map questions, people were like, yeah, I'm not sure if that's available in Python.

47:43 And I don't really know how this concept would work in Python, but at the same time, they knew

47:49 about dictionaries.

47:50 So that was just, I don't know.

47:52 I mean, obviously, you know, I understand where this is coming from and this is not necessarily

47:56 like a terrible sign.

47:57 But on the other hand, I was curious or just confused.

48:00 Like, why, why was this happening?

48:01 I mean, you know, like, why, why is there this disconnect?

48:04 Because in Java, well, if you want like a mapping structure or a dictionary, you know that you're

48:10 dealing with some kind of map or a hash map, right?

48:13 Like, you know, what kind of concrete data structure you instantiate in Python is just like, yeah,

48:17 it's a dictionary.

48:17 It does its job.

48:18 I don't really care how it works.

48:19 And that's good.

48:20 But it also means you're kind of, you're disconnected from a lot of the specific implementation details

48:25 as a developer.

48:26 Yeah, that's for sure.

48:27 And until recently, dictionaries and still under some circumstances, they're unordered,

48:33 which can cause all sorts of issues.

48:35 If you read a file in, make some changes, save it back out in Python, you know, and you had

48:40 that file in source control, it could look like 95% change where you just change like something

48:45 from true to false or something silly like that.

48:46 Right.

48:47 And so there's actually, depending on the situations that you're in, there's all these

48:52 other types of dictionaries that are pretty awesome.

48:54 So we have ordered dict.

48:55 I've seen the people that do data science and like really chain together generator expressions

49:01 and list comprehensions and make massive use of default dict.

49:04 There's chain map, which I don't think I had heard of chain map before.

49:08 Maybe I heard of it and forgot about it.

49:09 There's even mapping proxy type.

49:13 So maybe take us through like when you might want to use those other things, like when dict

49:17 or curly curly is not enough.

49:18 I was also really, really amazed or, you know, really enjoyed learning more about chain map.

49:24 So chain map is really cool.

49:25 Basically, you can chain together multiple dictionaries and treat them like a single dictionary, like

49:31 a single mapping.

49:32 So you can group several dictionaries into one.

49:35 And then when you do a lookup, actually the chain map is going to go and it's going to

49:39 search all of these underlying dictionaries one by one until it finds a match.

49:43 Yeah, that's really cool.

49:44 To me, it felt like what I had previously done, if I wanted this behaviors, I would create a

49:50 new dictionary and I would merge all of them into it.

49:53 Then I would ask that question.

49:54 But this lets you basically create the same behavior without actually building that other

49:59 dictionary.

50:00 And potentially you could create the chain map and then one of the underlying dictionaries

50:04 could change.

50:05 And then the answer coming back from the chain map would adapt to that, which is pretty awesome,

50:08 right?

50:09 That's right.

50:09 And you don't incur that memory overhead because, you know, it's just like an abstract way to

50:15 merge all these dictionaries together.

50:16 And it's just the magic is in this search algorithm.

50:19 But on the other hand, of course, you know, it can potentially be slower because the search

50:24 actually needs to happen.

50:25 So depending on, you know, if you change together thousands of dictionaries, maybe that's not

50:28 a good idea.

50:29 But if you, if you only have a couple of them and maybe you're combining a bunch of

50:33 configuration dictionaries, then this could be really handy and you can just make sure,

50:37 you know, okay, we have like a default setting dictionary with default settings, and then we

50:42 can override them with more specific user settings.

50:45 Right.

50:46 Put the user settings and stuff at the beginning of the chain map.

50:49 Yeah.

50:49 That's nice.

50:51 Yeah.

50:51 The place I see this happening all the time on the web is because you've got to go, you've

50:54 got a dictionary for like the route URL data.

50:57 You've got one for the post, you've got one for the query string, you've got one for or else,

51:01 somewhere else, headers, all sorts of stuff.

51:04 And you can like combine them together into one place and ask like, did they give me this

51:08 data somehow?

51:09 I don't really care where it came from.

51:10 Did they tell me their username or something?

51:12 Right.

51:13 Yeah.

51:13 Where's the API key?

51:14 Yeah, exactly.

51:14 And then where's that print statement I'm looking for?

51:16 All right.

51:18 So another major class of data structures that I've found to be pretty interesting that you

51:23 covered were arrays.

51:24 So, you know, sort of the most common one of this is the list, but there's more than that,

51:29 right?

51:29 Oh, yeah.

51:29 Yeah.

51:30 And it's really similar to, you know, what we just discussed with the dictionaries where,

51:34 well, if you're looking for an array, it's typically people will just tell you,

51:38 well, use a Python list.

51:39 It does everything you want.

51:41 And, you know, behind the scenes, Python lists are actually dynamic arrays.

51:44 So mutable dynamic arrays, they will grow and shrink with the data you put in.

51:48 But there's actually other options as well.

51:51 You know, you can, in a lot of ways, you can view a tuple as an immutable array, which means

51:57 you can't change it.

51:58 Right.

51:58 You index into it.

51:59 It has a length.

52:00 You can say, give me the third, give me the fifth one, but it's totally static.

52:03 Even the individual elements can't be changed, right?

52:06 Exactly.

52:06 Yeah.

52:06 And I mean, the only problem you run into if you have like, you know, a nested data structure.

52:11 So if you have a dictionary in your tuple, you can't put something else at that position in the

52:17 tuple and replace the dictionary, but you could still reach in and modify the dictionary because

52:20 the dictionary itself is not immutable.

52:22 It's, you can modify it.

52:23 And so sometimes, you know, that causes trouble for people.

52:26 But in general, you know, yeah, you can absolutely treat a tuple like as an immutable array.

52:31 Let's go back just for a second.

52:33 Like one thing we kind of moved on from without talking much about it is mapping proxy type

52:37 in the dictionary space.

52:38 And so if you really wanted like this truly immutable tuple, you want to put a dictionary

52:43 there, you could wrap it in this mapping proxy type, right?

52:46 Oh yeah.

52:47 That's a perfect segue.

52:48 Yeah, exactly.

52:49 So you, you could use this mapping proxy type class to make, take any standard dictionary and

52:57 then to provide a read only view into that dictionary.

53:00 So you can still access all of the variables, all the values, all the keys, but you cannot

53:05 reach in and modify the dictionary.

53:07 I mean, sure.

53:08 You could just kind of pry that mapping proxy apart and then reach in and modify it.

53:14 And this is probably not a good idea, but.

53:15 That goes back to where the, like, you, you know, you shouldn't be doing it.

53:19 And if you, if you mess it up, it's kind of your fault, not the APIs.

53:23 It's like, yeah, you could install a kernel driver and patch the memory and.

53:27 I rooted my machine and then it crashed.

53:29 Oh, this thing is a crappy OS.

53:30 Yeah.

53:32 So that would, that, I mean, you could combine that for like a greater functional programming,

53:36 immutable pass by value type of behavior, right?

53:39 Yeah, you could do that.

53:40 I mean, personally, I'm always thinking like, okay, at what point do we reach the level where

53:45 we're really fighting against what the language wants us to do, but it is entirely possible,

53:51 you know, to do what we just talked about, you know, you have a tuple and then instead

53:55 of putting like dictionaries in there, I mean, you'd probably just go with a frozen dict class

53:59 for that, but you could totally also wrap it in this mapping proxy type thing.

54:03 And that way you really enforce that immutability as best as we can.

54:07 And I think that's so nice about Python.

54:10 The frozen dict is a little more obvious, but yeah, that is super nice.

54:12 One of the things that traditional, let's say C, C++ arrays to a lesser degree, like

54:20 some category of say C# arrays have is they have the same type in them and then they're

54:27 tightly packed.

54:28 And the tightly packed thing that I'm really thinking about, right?

54:30 So if I say list and Python and I put number one, number two, number three in there, that's

54:35 really a tightly packed set of pointers that points out to three things that are not anywhere

54:40 near each other.

54:40 Potentially.

54:41 I mean, those are probably like pre-allocated and put next to you.

54:45 Like I know one to 20255 or pre-allocated, but you know, in general, things in the arrays

54:49 are not nearby each other, which means you're following a lot of pointers and there's a lot of GC type

54:56 behavior, but there are structures that if you really want homogeneous types, tightly

55:01 packed, you can get them, right?

55:03 You can also get that in Python because like you said, with your standard list, there's really

55:07 no guarantees.

55:09 If your list of a couple of elements, it could actually be much larger than just those like

55:14 five or six elements because it's pre-allocated to a bigger size and kind of shrinking and growing

55:18 dynamically.

55:19 And so if you want absolute control over how your data is laid out in memory, then you

55:24 could use something like the array class from the array module that's in the Python standard

55:30 library.

55:30 And with this, you can actually say, Hey, I want an array off C style floats or ints, and

55:38 you can just put in your values there and they will be laid out in memory exactly the way you

55:44 request it.

55:45 But it won't accept types that don't fit the declared data type, right?

55:48 So if you say it takes floats, it won't take a string, right?

55:51 It enforces that as well as the packing.

55:53 Yeah, that's right.

55:53 So you get some type safety there.

55:55 So it's going to make sure that you're not putting in, I don't know, something crazy needs

55:59 to be consistent.

56:00 Let's skip ahead a little bit.

56:01 We're actually, I thought we would get through more of these.

56:03 I think we're going to run out of time pretty quick.

56:05 But there's some stuff I really want to cover while we're here, because I think it's, it's

56:09 quite interesting.

56:09 You talk about stacks and you talk about queues.

56:13 And I think this list that we were just talking about, it can be many things to many people.

56:19 It can act as a queue.

56:20 You can take the thing off the front.

56:22 It has a pop to take things off the end.

56:25 You can append that's like a synonym for push sort of.

56:28 But there's also other queues.

56:30 And there's also some really terrible performance issues if you try to use, say, like a list for

56:36 a queue.

56:37 So let's maybe touch on those things while we're on this topic.

56:39 Yeah, for sure.

56:40 So I mean, the list, again, it's such a versatile data structure in Python, right?

56:45 And it's, I feel like they made exactly the right performance trade-offs when they implemented

56:51 how list is actually implemented in the CPython interpreter.

56:54 But on the other hand, it leads to those situations where you're like, oh yeah, you know, I can push

56:58 and pop from a list.

56:59 So I can totally make a stack or I can totally use it to make a queue.

57:03 And I mean, you can, but the problem is, for example, if you use a list to make a queue,

57:10 you can run into a situation where you'll just have terribly slow performance.

57:16 Because if you enqueue an element in a list and you added, like you insert it at the front,

57:22 this will actually shifting all of the other elements by one.

57:26 So you're looking at a linear time operation just to insert one element.

57:30 And that's really, really bad for a queue implementation.

57:33 That's super bad.

57:34 Yeah.

57:34 If you're doing like the first in, first out behavior, right?

57:37 So append, that's fine.

57:39 It goes on the end.

57:39 But every time you take an element off the front, it has to entirely copy every element.

57:44 If there's a million items in your queue, that's a heavyweight DQ operation right there.

57:49 So ironically, you should use this thing called DQ to fix it.

57:53 Yeah.

57:54 So there's this great, great little class in the collections module.

57:57 And the collections module is just so awesome.

58:00 If you're listening to this, for anyone who's listening to this show and they haven't heard

58:04 or played with the collections module, check out the collections module.

58:07 There's, it's just a treasure trove.

58:08 And it's also fun reading, you know, just, just seeing what's available in there and the

58:12 reasons for why it exists.

58:14 And, but yeah, the collections module has this thing called a DQ or deck.

58:20 I think some people pronounce it.

58:21 And this is actually implemented as a linked list.

58:25 So you can append and remove from both ends really, really fast because, you know, it's no,

58:32 there's no shuffling around.

58:34 It's just, okay, cut off the head or add something to the end.

58:37 It's a single pointer operation.

58:38 It's super, super fast.

58:39 That's right.

58:40 And the DE at the beginning is not the operation of removing the item from the Q, but that's

58:45 the double ended Q, DEQ, right?

58:48 Yeah, that's right.

58:49 It's fast from, from the front and fast from the back as well.

58:52 Right.

58:53 And we have things like Q.Q and multiprocessing.Q and some other ones, special cases one.

58:59 So I think it's really interesting to think about these because a lot of them, you know,

59:03 like we said, you don't really discover them on your own.

59:06 You have to kind of study them and seek them out.

59:08 And you're like, oh, that's, I thought just Python was slow.

59:11 List turns out to just be a really bad Q.

59:13 So maybe I'll do something different, right?

59:17 Yeah.

59:17 So I think these are all really great, but I don't want to give people the impression that

59:20 your book is all about just data structures, right?

59:23 There's actually, you know, just one chapter.

59:25 It just happens to be a super rich chapter.

59:27 So maybe like we have some other stuff that was on deck that we're not probably going

59:31 to be able to talk about in terms of time, but maybe just run us super quick through like

59:34 what else you wanted to highlight.

59:35 When I wrote this book, really the angle it's written from is that I don't want to create

59:40 another technical reference, but I want to walk people through some actionable stuff and

59:47 explain things step-by-step with examples that make sense.

59:52 So I'm also covering kind of the classic features in Python that are often difficult to wrap your

59:58 head around.

59:59 And, you know, I don't know how much more time we have here, but for example, I think if

01:00:05 people understand decorators and like really understand how they work internally, that is

01:00:11 a huge leap from a beginner level understanding of Python, because you need to understand, you

01:00:17 know, a couple of rather advanced features, like you need to understand, you know, what

01:00:22 first class functions are that all functions can be treated are real full blown objects.

01:00:26 And, you know, you can not just call them, but also look like a hand them around to other

01:00:30 functions.

01:00:30 Right.

01:00:31 And closures, the concept of like variable capture and closures is front and center there.

01:00:35 And that is all super important stuff.

01:00:38 And just so by, you know, focusing on under really understanding how this feature works,

01:00:43 you can actually learn a lot about Python and you can pick up a lot of other features and

01:00:50 abilities that are going to help you in programming Python in general.

01:00:54 And so a big part of the book is, you know, I really spent a lot of time on this and it's,

01:00:59 it's been iterated over.

01:01:01 I tested this with people and just walking them through understanding how decorators,

01:01:06 for example, work, or, you know, walking them through iterators and then going from there

01:01:10 to generators and then later generate expressions and really just explaining the logical connection

01:01:15 between all of those and kind of structuring it away in a way where it's not just like,

01:01:19 oh yeah, this is how it is, but kind of approaching the topic step by step.

01:01:23 I think those are really, really nice things.

01:01:25 You certainly building a simple decorator, like come up with some concept where it makes sense.

01:01:31 And go through the motions of creating it will really, like you said, expose you to a lot.

01:01:35 Some other things you talk about, you talk about the sushi operator, and I'm not going to give any

01:01:40 more details.

01:01:40 People have to check out your book to learn about that Python has a sushi operator.

01:01:44 Design patterns like emulating a switch statement with dictionary dispatch, all sorts of stuff like

01:01:51 that.

01:01:51 So really just tons and tons of these little tricks.

01:01:53 And you said the, it's like a buffet, right?

01:01:55 So you don't have to take it in order.

01:01:57 Yeah, that's right.

01:01:57 So most of these chunks in this book or the sections in this book, you can actually,

01:02:02 you know, jump around and just kind of look for things that interest you.

01:02:05 They're usually pretty short and they all have a key takeaway section at the end.

01:02:10 So you can quickly review this.

01:02:11 And for example, I heard feedback on the data structures chapter where it's, you know,

01:02:16 it's a great way to prepare yourself for a Python interview and just kind of, you know,

01:02:19 get updated or get a refresher on how these standard data structures map to Python.

01:02:24 I feel like people kind of, they take the easy way out and they ask these algorithm questions

01:02:29 and these data structure questions in interviews.

01:02:32 And yes, it's important, but I don't necessarily feel like if I don't know the most efficient

01:02:38 particular queue for this type of operation, that doesn't necessarily mean I couldn't write

01:02:42 an awesome web app that has, you know, a hundred thousand users a month and has no real performance

01:02:47 pressure on it.

01:02:48 Right.

01:02:48 I agree with you.

01:02:49 I mean, unfortunately, some people will really bust out the trick questions where it's like,

01:02:54 ha ha, you didn't know about this like special case.

01:02:56 That means you're a fool and we're never going to hire you.

01:02:58 But the reason I don't like those is I feel like those tend to advantage the people that

01:03:03 have CS degrees, but not necessarily better programming skills than people who are self-taught.

01:03:08 So I really like that your book kind of backfields that information for people.

01:03:11 Thanks.

01:03:12 I mean, that was one of the goals, right?

01:03:13 Where it's like, yeah, a lot of this stuff, you don't need to spend like four years in

01:03:18 university just to understand what a hash table is.

01:03:22 Right.

01:03:22 And it's, I don't like it when people create that impression.

01:03:25 And so, you know, I feel like if you have a good practical understanding or practical programming

01:03:32 skill, then it will be in most cases, like pretty easy for someone to fill in those gaps in

01:03:38 our knowledge and, you know, to satisfy the people who ask the more computer science motivated

01:03:43 questions, I guess.

01:03:44 Yeah, absolutely.

01:03:44 All right.

01:03:45 Well, let's leave it there for the book because we are definitely going to run out of time.

01:03:48 So I do want to talk about a few other projects that you have going.

01:03:51 First of all, you have this thing called Pythonista Cafe.

01:03:54 Did you open like some kind of like coffee shop or something?

01:03:59 That's my lifelong dream, you know, open a coffee shop, write books.

01:04:03 Answer a Python question before you're permitted in.

01:04:05 Yes.

01:04:05 So Pythonista Cafe, it all started when occasionally I read Reddit or Hacker News and it can be great.

01:04:12 And it can also be like people have some weird personas on these forums sometimes, you know,

01:04:18 and people kind of lash out at each other and get really worked up.

01:04:23 And I remember this one time, like somebody made a post on Reddit, one of the Python forums

01:04:27 on Reddit and said, hey, I'm this 14 year old guy.

01:04:30 I really love Python.

01:04:31 I made these tutorial videos on my YouTube channel about, you know, building a Flask app.

01:04:35 What do you think about this?

01:04:36 And like the first person to reply is like, oh, you're teaching it all wrong and you should

01:04:41 have never created these videos and you'll basically, you'll never be a good programmer

01:04:45 and blah, blah, blah.

01:04:45 Who do you think you are?

01:04:46 He's probably like in the top, like 5% of all like 14 year old programmers.

01:04:50 I mean, come on.

01:04:51 And that is nuts, right?

01:04:52 Like, how can you, how can you, I was just almost like sitting in front of a computer

01:04:57 shaking and getting angry at this person.

01:04:59 And it just seems so crazy.

01:05:01 But the problem is, I mean, this is, this is real, right?

01:05:04 Like this happens.

01:05:04 I mean, most of the time it's a great forum.

01:05:06 It's a great community, but some people, they are just crazy when it, the knee jerk reactions

01:05:13 they get.

01:05:13 And I really liked that.

01:05:15 And so I reached out to a bunch of people and said, hey, so here's an idea.

01:05:19 We're going to do a Python forum, an online forum where people can discuss Python, you

01:05:24 know, you can bring their code and get feedback on their code.

01:05:26 And we're going to talk about all things Python, but we're going to do it in private.

01:05:31 So it won't be a public forum and there will be an application process to get in.

01:05:35 And I launched that at the beginning of this year.

01:05:39 So 2017, I don't know when the show is going to come out, but so far it's been a really,

01:05:44 really cool experience.

01:05:45 We have about 120 people in the forum right now.

01:05:48 And it is, I don't like the word safe space, but in a way it is a safe space where you can

01:05:52 just go and post some code and you're going to get like really, really positive and encouraging,

01:05:57 helpful comments.

01:05:58 And there's no danger of someone going in and be like, oh, you're the worst like 14

01:06:02 year old programmer in the world.

01:06:04 And you should go back to where you came from.

01:06:06 That's Pythonista Cafe in a nutshell.

01:06:08 Nice.

01:06:09 That's awesome.

01:06:10 So yeah, you've got that going.

01:06:11 Did you do a book or videos or something for Sublime for Python developers?

01:06:15 Yeah, I also did that.

01:06:16 So I have a couple of courses that I offer on my website.

01:06:20 And one of them is also on Talk Python training.

01:06:22 And so one of them is a complete course on setting up Sublime Text for Python development.

01:06:28 So like an end-to-end, step-by-step course.

01:06:31 And that consists of a couple of modules and, you know, like workbooks and video lessons on

01:06:35 how to get a really, really awesome and productive Python environment with the Sublime Text editor,

01:06:40 because that's my favorite editor.

01:06:42 I know you really enjoy PyCharm and I think that's a great option too.

01:06:45 But for me, Sublime Text has just been, it's just been a great setup.

01:06:49 And you can script it with Python, which is nice too.

01:06:51 It's definitely a very Pythonic place to be.

01:06:54 That's awesome.

01:06:54 So yeah, that's cool.

01:06:56 If people want to learn more about like getting the most out of Sublime, they can check that out.

01:06:59 You've got like the pip dependencies course available both on your website and Talk Python training,

01:07:04 which I thought that was really awesome.

01:07:06 I mean, people are like, well, I know PIP.

01:07:08 Like now this is like really, really the right way to sort of manage dependencies in your app

01:07:13 or other packages that you're using.

01:07:15 It's really great.

01:07:16 So I definitely enjoyed checking that out as well.

01:07:19 All right.

01:07:19 So I think you've given us a spoiler for this.

01:07:21 The last two questions here, at least the first one and the last two.

01:07:24 Favorite editor?

01:07:25 No, it's Sublime.

01:07:26 Bye, John.

01:07:26 Yeah, that's awesome.

01:07:28 Yeah, I think I like Sublime as well.

01:07:30 I definitely use it some of the time.

01:07:31 Very cool.

01:07:32 And notable PyPI package this time around?

01:07:35 Yeah.

01:07:35 So this time around, it'll be Django because Django 2.0 just came out.

01:07:40 That's big news.

01:07:40 What's the biggest piece of news about Django 2.0?

01:07:42 I think the new URL routing syntax is pretty cool.

01:07:46 It looks really nice.

01:07:47 And for me, Django, you know, it just represents so well what Python stands for.

01:07:52 It's really, you know, this is a piece of software that you can depend on.

01:07:56 It's mature.

01:07:57 And it's for when you actually want to get work done and not be worried about having

01:08:01 to rewrite your whole stack in six or eight months down the road.

01:08:04 And I feel like, you know, the Django team is so awesome.

01:08:07 And it's a huge achievement about what they've done over the last couple of years.

01:08:11 And yeah, so, you know, my pick is Django.

01:08:13 Awesome.

01:08:14 New 2.0.

01:08:15 Yeah, the shiny Django.

01:08:16 And the other major news around there is that they've dropped legacy Python support.

01:08:21 No more Python 2 in Django 2.

01:08:23 Oh, yeah.

01:08:23 Yeah, ironically, given the numbers there.

01:08:25 I think that's a super positive move.

01:08:29 Just the fact that Django switched their default behavior in their documentation from Python

01:08:35 2 to Python 3 made a huge dent in the consumption of Python 3 on PyPI.

01:08:39 And now I imagine this is just going to take it up a notch.

01:08:42 That's great.

01:08:43 Yeah, totally.

01:08:44 I mean, I saw that.

01:08:45 Like, I've been, you know, I have a couple of videos on YouTube on should I be using Python

01:08:49 2 or Python 3?

01:08:50 And just a couple of months ago, you know, people were like, oh, yeah, this is a valid question.

01:08:55 And now it's like every time I tweet this video, someone is like, there's no question

01:08:59 about it.

01:08:59 Like Python 3, like, don't go with Python 2.

01:09:02 So that's great news, I think.

01:09:03 I totally agree.

01:09:04 I feel like the last six months or so it really has become like, why would you even ask this

01:09:08 question?

01:09:08 Like, this is sort of a deprecated video.

01:09:10 You should just take it down.

01:09:11 But that's really positive for the whole community.

01:09:14 Awesome.

01:09:14 All right.

01:09:15 So final call to action.

01:09:16 People can get your book.

01:09:17 They can check out Python East Cafe.

01:09:19 There's all sorts of stuff they can do to learn more about this, right?

01:09:21 If you're listening to the show and you found interesting what Michael and I talked about,

01:09:25 and you want to learn more and kind of go from beginner to intermediate Python skills and

01:09:29 beyond without complex jargon and other crazy stuff, then you should download the free sample

01:09:36 chapter for my book.

01:09:37 You can just go to pythontricksbook.com or also just go to Amazon and search for Python

01:09:42 tricks.

01:09:43 And the book also comes with a free bonus toolkit where people can get more than two

01:09:48 hours worth of video material to help reinforce some of the main lessons in the book.

01:09:53 So yeah, go check it out at pythontricksbook.com.

01:09:56 Awesome.

01:09:56 All right, Dan, thanks for being here.

01:09:57 It's fun, as always, to chat with you.

01:09:59 Hey, thanks for having me on the show again.

01:10:00 It was great.

01:10:01 You bet.

01:10:01 Bye.

01:10:03 This has been another episode of Talk Python to Me.

01:10:06 Today's guest has been Dan Bader, and this episode has been brought to you by Linode and

01:10:10 Rollbar.

01:10:11 Linode is bulletproof hosting for whatever you're building with Python.

01:10:15 Get your four months free at talkpython.fm/linode.

01:10:19 Just use the code Python17.

01:10:21 Rollbar takes the pain out of errors.

01:10:24 They give you the context and insight you need to quickly locate and fix errors that might have

01:10:30 gone unnoticed until your users complain, of course.

01:10:32 As Talk Python to Me listeners, track a ridiculous number of errors for free at

01:10:37 rollbar.com slash talkpythontome.

01:10:39 Are you or a colleague trying to learn Python?

01:10:42 Have you tried books and videos that just left you bored by covering topics point by point?

01:10:46 Well, check out my online course, Python Jumpstart by Building 10 Apps at

01:10:50 talkpython.fm/course to experience a more engaging way to learn Python.

01:10:55 And if you're looking for something a little more advanced, try my Write Pythonic Code course at talkpython.fm/pythonic.

01:11:02 Be sure to subscribe to the show.

01:11:05 Open your favorite podcatcher and search for Python.

01:11:07 We should be right at the top.

01:11:09 You can also find the iTunes feed at /itunes, Google Play feed at /play,

01:11:14 and direct RSS feed at /rss on talkpython.fm.

01:11:18 This is your host, Michael Kennedy.

01:11:20 Thanks so much for listening.

01:11:21 I really appreciate it.

01:11:22 Now get out there and write some Python code.

01:11:24 See you soon.

01:11:24 I'll see you next time.