00:00 Are you fluent in Python or do you speak the language with an accent? Python's ease of learning can also lead to non-pythonic patterns for even experienced developers. Luciano Ramalho is here to give us a deeper understanding of this language we love.
00:00 It's episode number 24 recorded Thursday, August 6th 2015.
00:00 [music]
00:00 Welcome to Talk Python to Me, a weekly podcast on Python, the language, the libraries, the ecosystem, and the personalities.This is your host, Michael Kennedy. Follow me on twitter where I'm @mkennedy. Keep up with the show and listen to past episodes at talkpythontome.com and follow the show on twitter
00:00 via @talkpython.
00:00 This episode is brought to you by Hired and Codeship. Thank them for supporting the show on twitter via @hired_hq and @codeship
00:00 Before we get to the conversation with Luciano, he generously agreed to give away an electronic copy of his book, Fluent Python. All you have to do to be eligible to win is be a 'friend of the show'. Visit talkpython.fm and choose "friends of the show" in the navbar and sign up
00:00 I'll have a Python app randomly choose one lucky friend a few days after this episode is released.
00:00 Now let me introduce Luciano.
00:00 Luciano Ramalho is the author of Fluent Python. Ramalho was a Web developer before the Netscape IPO in 1995, and switched from Perl to Java and finally to Python in 1998. Since then he worked on some of the largest news portals in Brazil using Python. He has taught Python web development in the Brazilian media, banking and government sectors. He has spoken multiple times at OSCON, PyCon, PythonBrasil, FISL and RuPy. Ramalho is a fellow of the Python Software Foundation and co-founder of Garoa Hacker Clube, the first hackerspace in Brazil. He is a managing partner at Python.pro.br, a training company.
02:04 Luciano, welcome to the show.
02:06 Thanks Michael, thanks for having me.
02:08 Yeah, I'm really excited to talk to you about your new book, "Fluent Python".
02:12 Yeah, I'm excited about it too.
02:14 I bet. You've been working on it for a while. Before we get into talking about your book, which is really an amazing book, let's talk about how you got started into programming and in Python at all, like, you know so much, I can tell from your book, but at some point you had to start at the beginning like we all do, right? Where did you get started?
02:37 Ok, now, I am 51 years old, so when I was a teenager, in Brazil there were no micro computers, but I got a hand on a programmable calculator ti-58, and I learned to program by program in it. And actually my first interesting program was a part of a game from the HP 25 calculator to the ti-59. It was the Lunar Landing game that was famous.
03:07 Oh, that was a great game.
03:09 So that was my first significant program. And then, I was an exchange student in 1981 in the US, I lived in little Harrisburg Illinois, population 10 000 and there I got- there were in the library in the school there were a few Apple 2 computers that had just arrived when I arrived and nobody knew how to use them. The teachers were supposed to teach about them but hadn't yet taken the course to do it, so they were basically free for anyone to do whatever they wanted, and I learned I taught myself Basic programming.
03:46 Anyway, so that's a long time, and then I went through a lots of languages after Basic, Pascal, and Pascal was one of the most important ones for me, and then Delphi and then Visual Basic and I studied a little bit of Smalltalk. And then, when the web started here in Brazil again we had a kind of a different situation, because there was all telecom companies in Brazil were state owned at that time, later they were privatized, but they were state owned in 1994, and so I was hired by a large publishing company here to start developing their strategy for online and I taught myself pro programming, because everybody did everything on the web, that was automated using Perl at the time.
04:38 I loved Perl, I thought it was awesome, but I also found it- well I just said I loved it but I loved it but I also hated it at the same time, because it was very powerful and quick, but it was also full of craps. And, it was difficult to read afterwards and so on. Then I tried Java, and I probably was one of the first people in Brazil to do server side Java, because when Java was first released, some was trying to market it as a tool for client side programming with the applets and so on. It took a while for them to realize that server side was where Java was going to thrive.
05:24 But I didn't enjoy the verbosity of Java. I liked the fact that it was really object oriented which Perl was not at the time, but it was too verbose and then after a while I went back to Perl again, and Perl had acquired object oriented features around Perl 5. And then, something happened, as the Perl community was discussing how to do things in a dynamic language but also in an object oriented way they kept repeating the sentence, "Here is how they do it in Python. In Python this is like that..." and that's how I first heard of Python.
06:06 It was discussions on the Perl maybe about how to do things in an object oriented way, but in a language there was a scripting language and not a dynamic language. So, after reading mentions of Python few times, I decided to go study it, and then I read the tutorial, which was shorter at the time than it is now, and I just fell in love immediately, because for me, it had the best features of Python, of Java and Perl. It was really object oriented with an object oriented standard library like Java, and had exceptions and other things that I liked about Java, but it was very concise and to the point and agile like Perl. And, so it was for me the combination of the best features of both languages, and then I basically never looked back.
07:06 That's a great story. I think we've all had those sorts of feelings about some of our first programming languages, we love them because we really got into programming, but they are not necessarily the best-
07:20 Yeah, exactly.
07:20 But Python is where you sort of settled into, I could tell you've been doing it for the long time from your book.
07:28 Yeah, so I started doing it in 1998, it was when I found the tutorial that I mentioned, and then I immediately pitched a Python solution to our client, at the time for an online news site, and then I discovered Zope which was just then released, as open source, and I started using it like a couple weeks after it was released as open source, and then a few months later, we actually launched in Brazil a new site that was based on Zope and it was one of the first big Zope cases around the world.
08:10 Wow, it's excellent.
08:11 Yeah, and so I owe a lot to Zope, I don't use Zope anymore these days, but I owe a lot to it because it was because of Zope I was able to deliver the kinds of products that the clients were looking for me to deliver, like content management systems, and so I was able to get paid for writing Python pretty early, at least in the context of Brazil, since 1998.
08:40 That's a great feeling, right? To find this thing you love and wow, people pay me to do it, that's really cool. It's a good time to be a programmer indeed. So let's talk a little bit about your book. The title is "Fluent Python" and it is coming out under- it's been published by O'Reilly and it's coming out I think September 4th is what Amazon.com tells me anyway, is that right?
08:59 Yeah, to be honest I don't know exactly when the print book is going to come out, because different pages that I look at give different dates, I've seen August, I've seen September, and I've seen October all in the last two weeks.
09:12 Amazing.
09:12 So, yeah.
09:15 But soon, right, pretty soon, just about the same time that this podcast comes out I think, should be the time that the book is released.
09:22 Wow, that's excellent. But let me say, the e-book is now complete in the final form so people who buy the e-book or who have bought it previously can now download the final version of first edition first revision, and it's also available from Amazon.com for the kindle, in the final form. So we are only waiting for the print book at this time.
09:51 Right, ok, excellent. I think having electronic book works really well. I was reading on my Kindle Paperwhite and it was really nice you know, the code samples came through really well, sometimes that doesn't work so well-
10:05 Wow, that's cool, nice to know.
10:06 Yeah yeah, it came out really, really well. And you know, so I've only had the book for a few days, so I've only been able to read maybe the first five chapters. But my impression of the first five chapters is this book is a masterpiece, it is really, really good.
10:24 Thank you very much.
10:26 I think it is going to go down as a classic book in the Python community.
10:31 Thank you very much.
10:32 Yeah, I don't just say that because you are on the show, but I was reading and it is a little hard for me to put in words I think for people to really understand, but it seems like everything that you covered- we'll talk a bit more about that, but it seems like everything you covered I'm kind of like "I mostly know this- oh wait, there is this like really cool piece of information or motivation that I was never really aware of", and just underneath the surface of the knowledge that I do have, and so and that was like all the time happening to me while I was reading the book and so I think it's a great contribution.
11:09 Thank you.
11:10 So, let me read a quick quote right from the beginning to get people a sense of I think what you are going for with this book. So, in the introduction, one of the first chapters you say "Python is an easy to learn, powerful programming language- those are the first words of the visual Python tutorial; that's true, but there is a catch, because the language is so easy to learn and put to use, many practicing Python programmers leverage only a fraction of its powerful features". Does that kind of sum up the problem you are trying to solve with this book?
11:40 Exactly, and that's why I called it Fluent Python, you know. As somebody who is fluent in two languages in Portuguese and English, and that knows how difficult it is to become fluent in a language because it is easy to become fluent in the language in your native language, right, you grow with it and people do not even notice. But getting fluent in another language is hard. And so, I've been teaching Python to lots of people; when I had a company whenever we hired people we never would hire people that knew Python because nobody knew Python at the time, so we hired people who knew Java and then taught them Python, but I also worked as a Python instructor and that's what I do now, these days.
12:29 So I realize that this thing that I say over there that is that the language is easy to use as practical so people start solving real problems really fast but that means that sometimes they are already using it for several years and don't know why some things are the way they are, and another thing that I say right after that in the second paragraph is that when you are learning a new language you look for things that are familiar to you, like for instance, "Maybe this language has a way to use a regular expression, let's see how that works".
13:10 But if there is something in a language that is not mentioned by something that you know from another language like for instance the concept of generators that many languages do not have, then you are unlikely to go look for it, right, because you don't even know the thing exists. A simple example in generators is like tuple unpacking, it's just a nice syntactic sugar but if you have never seen that in a language, you won't look for it, and maybe you are going to spend years working with Python without ever using it. But, it's a really nice thing to use because makes code more readable, even safer in some ways Python- tuple unpacking has this characteristic because it actually forces interpreter to count the number of items so if there is one more, one less an error will be raised, so it is a very nice feature but it is something that people who come from other languages may never use because they won't think about it.
14:10 Right, if you came from something like C Sharp for example, that doesn't have this concept of tuple unpacking, and so you just wouldn't even- you might never find it and that's, that kind of gets at the broader issue or concept of pythonic versus non pythonic code...
14:29 Yes.
14:32 It's pretty easy to come over to Python and learn it really quickly so that you can do the same thing that you do, but it is not always the best, is it? Some of your mental models don't really carry over so well.
14:43 Exactly. Yeah, it's very elusive thing to define what is pythonic, I actually mention it several times in the book and tried to address it in the afterwards, but basically I just send people links to other people that I respect a lot in the Python community who have a tend to define it like directly. But it's a good definition which doesn't really explain what is Pythonic, is to say idiomatic Python. Right?
15:21 Right.
15:20 But Ok, but so what is idiomatic Python? It's hard to define.
15:29 Yeah, I think your best bet is to show people a bunch of examples say, all of these things are pythonic- all of these things are not pythonic, because they carry over concepts that work well in other languages but are not the proper way to do them here, and once you get enough experience of going bad example- good example, then you kind of get this sense like almost a smell for like yeah it seems like this is right, seems like that's not right but it is very hard to be very concrete, you know...
15:58 Exactly. I tried to do that in the book and actually a decision that I made early on was to be as Pythonic as I could in the program examples, even when being Pythonic might not have provided the clearest source code forms from something that's not from mean within language. Like for instance, writing the first example in the book, I use name tuple, right. And then, I give it just like a couple of sentences to explain it's probably not enough if the person has never seen it, but it's actually something useful that I started using all over the book, and if that picks up the person's curiosity then that person can go and look for it.
16:48 This is what I decided, like for instance, before I actually have a chapter that formally discusses generator expressions, I use them in the book whenever it was natural to do so, even before I actually go over it, because of what I just said, you know, I wanted to write idiomatic Python whenever I could, even if that may- because the thing is when you are speaking to a native speaker of English, for example the person won't break down his or her speech for you, right? And that's cool, sometimes maybe you are going to understand everything but you are catching that there is something new about the way this person is expressing herself. And then you are going to try and find out what that means. If you always talk to people that spoke easy English you would only learn easy English and not real English.
17:50 You wouldn't get that chance to grow into a fluent speaker, kind of like you said.
17:52 Exactly.
17:54 So, your example with the name tuple, kind of hits on the point that I was talking about like I've used name tuples, I'm familiar with them, I think they are great, but you know, you go a little bit deeper and you are like, "Hey, it's really cool to use a name tuple, because the actual dictionary that stores attributes is stored in a class object which is a singleton in a runtime" not individual instances so storage is actually cheaper than regular custom classes.
18:21 Yes.
18:23 So if all you want to have is just a bunch of pieces of data that have names and an object, but not functionality, maybe you should use these named tuples.
18:23 [music]
18:23 This episode is brought to you by Hired. Hired is a two-sided, curated marketplace that connects the world's knowledge workers to the best opportunities.
18:23 Each offer you receive has salary and equity presented right up front and you can view the offers to accept or reject them before you even talk to the company. Typically, candidates receive 5 or more offers in just the first week and there are no obligations, ever.
18:23 Sounds pretty awesome, doesn't it? Well did I mention the signing bonus? Everyone who accepts a job from Hired gets a $2,000 signing bonus. And, as Talk Python listeners, it get's way sweeter! Use the link hired.com/talkpythontome and Hired will double the signing bonus to $4,000!
18:23 Opportunity is knocking, visit hired.com/talkpythontome and answer the call.
18:23 [music]
19:41 Exactly, yeah. And another thing- because they are inexpensive you can use it, they are inexpensive and they are completely compatible with tuples, right, you can unpack them, you can iterate over the items and so on. So for instance, whenever you need to return more than one value from a function, I recommend you return it in tuple. Because it will make for instance understanding what the function does easier because there is going to be the definition of this name tuple. If the person is looking at the result of the function in the consoled or in the debugger, the person is going to actually see the names of their parts of the tuple of the function is returning, so this is our recommendation, if you are going to return more than one value from a function, usually people do it with a tuple and that's fine, but with the named tuple is even better, and is cheap like you said, because of the way it is implemented it doesn't require more memory.
20:44 Yeah, you don't have to store all the keys, copies of the keys for every instance, right, like you would with the regular classes or objects.
20:52 Yeah.
20:53 And of course, when those come back if you are unsure what is coming back from that method, you could print it out and you get a nice listing of actually not just what's in there but the meaning of them by the name, of if you are in a debugger like PyCharm you could hover over and actually see it, it's excellent.
21:11 Yeah.
21:13 So one thing we said you used in your book is something called doc test to check the code in the console- can you talk about that a little bit?
21:20 Yeah, so that's something that I think was created by the people in the Zope community. And so the Zope community was always very big, so the main idea of doctest is to make sure that whatever code snippets appear in documentation, particularly listings of console sessions demonstrating stuff, that those actually match the behavior of the program at that time. So this was the starting point for doctests. In the Zope community, which is too active, there is a lot of development going on still in Zope, there is a lot of also files that are testing files that are not strictly documentation that are written in doctest formats.
22:17 Because it is really easy to- I find it excellent when I have to design a new API because it's sort of a BDD thing you know, you start typing those fake console sessions when you are thinking how it would feel to interact with these API. And the result would be look back that and so on, and then it's really good to do TDD, because you write a couple of stances of doc test and then you implement the code that makes it pass and so on and on and on...
23:00 You know the whole idea of trying to invision what your API looks like from the outside and then implement it generates much cleaner APIs in the end and this is one flavour that, right, that's cool.
23:14 Yeah, one thing I have to say about doc test is that after a while you realize there are some limitations to it. And even recently like last week it was- I had to I come out with the difficult solution to a waiting problem, and I started implementing some tests using doc tests, but then for instance doc test doesn't let you easily run just one part of the tests, it runs everything from top to down and sometimes when you are debugging it is used for to be able to run just a specific parts.
23:48 Anyway, the unit testing frameworks not only unit test but pytests which is my favorite much more powerful than doc tests and they can actually also run doc tests, so there is a way to integrate them. So you can have- I think the best idea these days is to really use doc test just for the documentation and just the straightforward cases, not corner cases, not weird outcomes, not testing whether an exception is raised or not because although of that can be done in doc tests in the end it becomes difficult to maintain the tests.
24:31 Right, maybe it's better done in py test, something like that.
24:33 Exactly, yeah. I use py doc tests, I prefer much better than unit tests because I think unit test has a very strong Java flavor of forcing you to do everything inside classes, and I much prefer the syntax that Pytest introduce, but I prefer py doc tests of having tests as functions- you can create classes if you want but you can also have module-- I prefer the syntax.
25:02 Yeah, cool. So one of the very first things that you talk about in your book is the Python data model. And you talk about it's sort of the foundation for making your own classes fit this pythonic programming model, and you have a quote in there, it's something like the- you know, talking about this data model or just learning about it is the tip of iceberg is just the iceberg is called the Python data model and it describes the API that you can use to make your own objects play well with the most idiomatic language features.
25:34 Yes.
25:36 Do you think people who have sort of come to Python from other languages ones that add functionality to their classes very much by overwriting virtual methods and those kinds of things sort of miss out on some of the Python data model like _ _init or the iter or str methods or things like that?
25:58 Yeah, for instance, this is something that happened to me many years ago when I developed the French deck, the card deck example that I use in that chapter, so many years ago when I have the idea of implementing that, and so the idea was to show how easy it was to create a new class that behaves like a sequence, right, so you implement _len, and you implement _getitem, and basically you are done. But then something really interesting happened, because the idea was to implement the card deck, right. So what is an obvious thing you want to do with the card deck is to shuffle it, right? So the first time that I created this example I actually had a method called shuffle, which was not a special method, it was just shuffle, right; so the idea was so you have this card deck and you can look at the cards and you can pull the shuffle method and see the cards shuffle.
27:01 Ok. But then, a few weeks after that, I realized that when I looked back at the example giving the class, I saw- wait, wait a minute, there is already a shuffle function in the random module, that shuffles any mutable sequence that you give it. So it actually doesn't make sense to implement the shuffle if the whole point of my exercise was to show how easy it was to create a sequence even if the sequence represented the card deck of a specific domain, specific thing, it didn't make sense for me to implement my own shuffle method because there was already a shuffle method and it is designed to work with mutable sequences.
27:55 Right, and it's already tested, probably optimized, all that kind of stuff, right?
27:59 Exactly, so that was an insight for me, an advantage of being pythonic is the fact that when you are pythonic, your objects suddenly become more compatible with things that are already out there in the standard library, and in other packages that you can download from PyPi. Yeah, I think this is a lesson that people need to learn, because for instance, I don't know about C Sharp, I studied Java a lot for a while and I even today sometimes I spend sometimes every year I try and sit down and read something about Java because I get a lot of clients in my course that studied Java before and is useful to make analogies. But for instance a big advantage of Python over Java is the fact that we have these operators that are more generic, like for instance the square brackets operator, which is the operator for to get an item or to set an item, right?
29:06 And they don't have that in Java. That operator in Java is a magical thing that only works with the primitive arrays. Anything else, you have to use some other you have to use method syntax. Also, these days I'm studying Go language and I really like it very much but one of the things that I most dislike about it is the fact that it has it follows this Java philosophy of having some data structures that are magical and you cannot create your own data structures that emulate them precisely, like for instance in Go there is a way to you can iterate over some primitive types like maps and arrays and slices, but you cannot create your own type that is iterable using the for range construct of the language.
30:03 Right, that's too bad, unlike in Python which is totally sensible, yeah?
30:08 So I think Python is actually better than most languages in this regard of enabling you to create structures that behave exactly like the most fundamental data structures in the language and the way to accomplish that is through the data module. So I think it's very nice to have that documented and exposed and I wanted to give it more visibility so that's why I started the first chapter with that even if in the following chapters my strategy then is to say ok, but let's look at what is already implemented in Python, because I didn't want people to go out re-implementing new sequence types when there is maybe they don't even leverage everything that the built in sequence types can offer.
30:56 Right, I think that's a good point. People coming from other languages they don't necessarily know what is available so they will find themselves re-implementing stuff... a really simple example would be I would like to randomly get an item out of a list so I might import the random module get a random index based on the length and then go index that out or you could just do random.choice- oh, I didn't know random.choice existed.
31:19 Exactly.
31:20 So one thing that you talk about around the Python data model is the len method. And, being such an object oriented language, I always felt that it should be collection.length or length or something like that, maybe even as a property, and the sort of special function on the outside of the classes seemed a little bit odd, but you had a good conversation around that, you want to elaborate on that?
31:48 Yeah, sure. So that is also something that bothered me for a while, of course you get used to it because it is really easy to get used to it, and it is really easy to type, is actually one of the arguments that Guido uses to defend that decision is that it's easier to type than having- if it was a method call, you would have to write like you know, x.len() so there is one dot extra there. And it's also something, and here is another thing that's funny; because I think it's of the human nature, when people come to learn a new language, sometimes they are not pleased, sometimes their boss told them to learn it, right. And then, some people try to resist by pointing out things that they don't like or things that they think are inconsistent, and bugs although bugs are extremely difficult for a beginner to find in Python, I've never seen a beginner find a bug in Python, every time a beginner thinks-
32:57 I've seen them write bugs...
32:59 Yeah yeah sure, yeah. But most of the time, in my life every time I've seen a beginner complains about a Python bug it was unbearable because the person didn't understand what was exactly going on. It was not a real bug. But anyway, about inconsistency- so this is inconsistent because it should be expelled as a method. Ok, now let's think about how this problem is solved in Java: in Java there is actually an inconsistency because there is the array type there is like a len as a property as it were, is actually like a public field, that you read, so you just write myarray.length without parenthesis, and that's how you get the length of the array, because Java is just basically giving you the value that's it's internally stored in a structure that represents the array, so that's very cheap and that's how they do it and it is important to be cheap because it is a very common operation. Then for other types that don't have the structure, there is a method called length. But also in the Java API, there are other classes, that could perfectly have methods named length, but have methods name size.
34:23 Size, or count or something, right.
34:24 Yeah, so it is at least three different ways that are common in Java called to do the same thing. And in Python we have only one way, it's the len of the thing. Call this a function. And so, first of all it's more consistent. Second thing is- it's pragmatic. Because the way it works is if you say len of x and x is a built in type, this built in type if the built in type has many elements there is actually a struct I forgot the name but it is spelled out in the book, there is actually a c struct that represents the type and it has the field that has the count of the numbers, of the number of items. So, that's the same thing that Java does, except that it doesn't expose a special field but uses the len function.
35:15 So the len function the implementation of the len function in the Python interpreter does that, it just goes to look at if there is a built in type and it's a multivalued built in type, then return the value of this field in the struct. So that's very cheap, and it is actually faster than doing the resolution of an attribute, right, so because if you had to write x.length this dot requires some advanced machinery in Python because of the dynamic nature of Python is a price we pay for all these dynamic behaviors that the 35:51 bot operaters kind of expensive.
35:52 Right. Basically a dictionary lookup and then the function calls themselves are also kind of expensive.
35:58 Exactly. And it's more expensive than a dictionary lookup, because there is all that machinery that I explain later in the book, basically in the last part of the book, the method programming part is where I explain actually the how descriptors work. And descriptors, the infrastructure, below properties, but also below another thing that's really fundamental that everybody uses in Python which are methods. Because what makes a method behave as a method, as an instance method, as a bound method or as a unbound method has to do with the way that descriptor mechanisms resolves the dot operator. So, it's an expensive thing.
36:48 So by making it be written as len(x) then interpreter can resolve this much faster. And, then if the structure is not a built in structure, then the interpreter falls back to looking up and then there is the cost of the attribute lookup, but if you look up under len method. So I think this is actually very elegant, because in the end the user interface is the always the same, is always len(x) but it gives us creators of API a way to provide a consistent interface, an interface that's consistent with the built in method, with the built in types, while at the same time ensuring that the built in types have the fastest possible performance for doing this operation which is crucial, right.
37:45 Yeah.
37:45 It's often something done in loops, right, and you want to optimize everything that's done in the loop, so getting the len of something is important to be fast.
37:45 [music]
37:45 This episode is brought to you by Codeship. Codeship has launched organizations, create teams, set permissions for specific team members and improve collaboration in your continuous delivery workflow. Maintains centralized control over your organization's projects and teams with Codeship's new organizations plan.
37:45 And, as Talk Python listeners you can save 20% off any premium plan for the next 3 months. Just use the code TALKPYTHON.
37:45 Check them out at codeship.com and tell them "thanks" for supporting the show on Twitter where they are at @codeship.
37:45 [music]
38:45 Yeah, that's excellent. And you talked a lot about the internals, which I think can be kind of tricky to discover in Python, the previous show that I recorded but is not yet released is with Philip Guo, all about this university class he did and then put online on the internals of CPython, like 10 hours series of walking through CPython, and-
39:12 I want to watch that-
39:12 Yeah, I just want to recommend the listeners like if that kind of stuff that we were just talking about sounds a little like, "Oh I'm not sure where this is going," watch Philip's videos. Maybe I will put them in the links here, because all of a sudden it clicked together, Oh I see exactly. And you talk a little bit about the same kinds of things that he does by importing the dis assembly module, the dismodule, and disassembling functions that actually looking at what it is you are talking about, to see what is really happening.
39:38 Yeah. I am not an authority on the internals of Python, but sometimes I was curious about some things and I went to look for it, but I also want to tell your listeners that the book is not about that kind of things, it has some moments that has these kinds of look into under the covers, but most of the time it's not like that. It's something that I enjoy but I also find it challenging, to read the source code for CPython, so I would love to see Python developing- here is a wish that I have: I would love to see some time in the future perhaps, a Python interpreter written in Go, because Go is much more readable than C, it's simpler also, the syntax is simpler, but it has the problem of concurrency resolved in a very good way and it will be awesome to see a Python interpreter written in Go, it would make it also much easier to understand what is going on.
40:49 Yeah, how about something a future Python the runtime parts are easier to read, because it is Go and it doesn't have the global interpreter lock-
40:59 Exactly.
40:59 Maybe because you can somehow resolve that with simpler threading models in Go.
41:03 Yeah, it's important to, since you mentioned the global interpreter lock, the Jython and the IronPython implementations of Python do not have it. Because they already can count on the underline implementation of objects and threads that is thread safe, and so something that happens with CPython and also with PyPi but IronPython and Jython do not have the global interpreter lock.
41:33 Oh, it's interesting. Yeah, so basically there is two cases of this is a possible thing, it's just we are not there yet, right?
41:40 Yeah, no, not at all.
41:42 Let's move on to maybe like the array section, the sequences. You did some really interesting things there, a lot of which was kind of familiar to me like list comprehensions and generators expressions, those are- you know, if you have been doing Python for a while it makes a lot of sense but then you also have like little gems in there like one of them that really stuck with me basically if you implement the iter method, you can automatically do iteration. But if the collection you have doesn't have 42:19 contains, then any sort of is check will actually be a sequential scan
42:26 In check, not is
42:27 I'm sorry, yes of course, in check, so you say object in sequence and it implements iteration but not the contains it can answer the question but it is a huge performance problem potentially.
42:41 Yeah, and that's because if you look on the right of the second page of chapter two, there is a diagram of the mutable sequence and sequence ABCs and then container 42:55 and so those ABCs pretty much define the standard API for mutable sequences and in the sequence ABC contains is a concrete method. So even though that is an abstract class the only abstract method init is the _getitem, and but the contains is already implemented and the way it is implemented is using iteration. This is one thing that makes me really like Python is the fact that as a teacher I found it easier to explain over the years. Although it has also grown more features that take a lot of time to explain, but for a certain specific set of features- for instance, if you are talking about sequences, these days is easier to talk about sequences I think than it was before. Because, although there is more content, you have to talk about comprehensions and so on, the basic concept of a sequence was not very well defined earlier in the language, because there were no ABCs in the documentation you read about, "Oh so this inspects a sequence..." but there was not strict definition of what a sequence was. After a while-
44:13 Like a vain concept of well a sequence is kind of these- yeah what does that actually mean precisely
44:17 Exactly. So people who were comfortable with the language intuitively knew what a sequence was, people who knew about the implementation of the language in C knew exactly what a sequence was but in a way that was very complicated to convey, and now with introduction of ABCs it is easier, because you can say, "So here is what a sequence is" A sequence has to have a way of-- is a container and an iterable size it container, so it has to be a way to get the len, to iterate over it and whether something is or is not in it. And then it has to have- there is a definition that people can look up, in the documentation, and in the book I drew some UML diagrams which I also think they make it easier to see the relationship between those ABCs because in the documentation there is no diagram so-
45:18 Yeah, those pictures are really nice, I like them.
45:20 Yeah.
45:20 So, one thing that I thought was really helpful, and I think probably I learned a decent amount from it was, when you are talking about this whole thing on data structures, you have this part that is called that you entitled "when a list is not the answer". Like you can almost always use a list, for everything and it does have effectively a pop method and pushes just a 45:50 and things like that, but it is maybe not the best choice for that, so you talk about things like arrays and the Deque class or sequence for double-ended queues- and I thought that was really interesting; do you want to just quickly talk about your message there?
46:10 Yeah, so that was the idea, because it's something that I've already caught myself doing in the past was to use a list when I should be using an array. An array is really easy to use and it is much more efficient in terms of memory, and also in terms of execution because it's a packed data structure where one number is right after the other, and-
46:41 It might be worth just like expanding on it just a little bit, because in normal Python, even if I say x=7, that allocates under down in the runtime, like a full on object which has all sorts of additional information and it is a pointer out to that thing whereas array allocates literally four bytes for answer whatever in a huge long array.
47:06 Exactly, that's why when you create- I think the first stumbling block to create an array is that you have to look up in the documentation because when you create an array you have to write this type code that tells the array what are the size of the numbers that I'm going to work. The size and the type of numbers that are going to be there because you can also have an array of floats, right, so you have to look up in the documentation to remember what are the letters that you used to specify the numeric type that goes into the array, but then when you do that, you are all set.
47:42 One thing that I try to do like for instance in the discussion about arrays there is that and these other places is to compare, I wrote two tables with like all the methods and operations of a list and array and so you can just briefly look and realize what the difference is, "oh, here is something that I can do with an array that I cannot do with a list, or vice versa". So, that is why I decided to add those tables, because there is a lot of functionality, and I wanted to pretty much graphically show people that they are almost the same, but there are some differences when you look at the tables it is easy to tell really quickly what the differences are.
48:28 Yeah, it was really helpful for me, so I appreciate that you had that, that's cool. So we are kind of coming up towards the end of the show, maybe you want to touch on just a couple more points that you think are important parts of the book, that maybe I haven't gotten to.
48:44 Yeah. So, basically the book is divided in six parts, right, we talked about the part one which is just one chapter where I present data module in a very introductory way, but the idea of the data module and demonstrate how useful it is with the card deck example and then there is a vector class with operators; anyway, and then the second part is data structures, the part that we were talking about, that part has a chapter, I think the longest chapters in the book are there.
49:20 The one about all the sequence types, the one about dictionaries and sets and the one about texts versus bytes. About the text versus bytes chapter I was very happy because one of my favorite programming book authors Bruce Eckel who wrote the famous "Thinking in Java" book these days he is doing some Scala writing, but also he loves Python and he was reading my book and he said that my chapter about texts versus bytes which covers unicode, what unicode means and all the different ways of converting unicode, from and to bytes, he thought they were very useful for him.
50:04 Anyway, so this was the data structures part of the book, then part three is about functions as objects. I decided to introduce that before classes, also because I think it's a pythonic way of thinking. We start by doing, by coding stuff and functions and we usually only create classes when realize that just playing functions won't do.
50:27 Right, maybe some data need to come along with those functions or something, right?
50:32 Exactly, exactly. If you were creat50:35 in Java, you are trying to do everything in classes because that's the only way to do it. So that's why I decided in the book to give a thorough explanation of functions and design patterns using functions, how you can simplify some design patterns by getting rid of classes just using functions, and then I wrap up with decorators and closures.
50:57 And then the part four is about object oriented programming, it starts with an explanation of how references work and how the whole thing about mutability works, and then I give some examples of implementing what I call Pythonic objects which are first a vector class, two dimensional vector class and then multidimensional vector class. And those examples are excuses to show the use of many special methods in practice-
51:34 I thought your comment on how you don't like to call the dunder methods magic methods. But special methods was interesting. There is nothing magic about these, this is how you do it.
51:47 Exactly. And the thing is, when you call something magical, you are saying, "Ok, so this is not for 51:55" but that is exactly the opposite like I was saying before about the criticism that- I was criticizing Java and Go because of the fact that it has some data structures that are magical and that you cannot really emulate them in your own code. And so, exactly. Anyway, I saw this is the part of the book where I explain how to do that and also there is some discussions about the use of interfaces and something that's a first and I am very proud that Alex Martelli who was one of the reviewers of the book one of the tech reviewers of the book, wrote an essay introducing after he introduced Duck typing the Wikipedia credits him with the creation of at least with the first use of expression Duck typing in the context of discussing programming languages, Alex Martelli now invented the concept of Goose typing.
53:03 Goose typing? Ok, what is Goose typing?
53:06 So, Goose typing is a variation of Duck typing where you have some type hints, this new feature of Python 3.5. But it doesn't require actually the type hints, the idea is to use ABCs as a way to specify types but here is the thing- Python uses ABCs in a way that's very influenced by its origins as a dynamic typed language with Duck typing. The main reason for that is that it allows you to actually create a class that does not inherit from an ABC but then you register it as a virtual subclass of that ABC so you actually promising, "I do implement that thing, although I do not inherit from it, I do implement that".
54:00 And you can actually do that after the fact for instance, maybe because it's a third party library that implements something that you at and say, hey, this actually looks like sequence and it will be used for it might called to be able to ask if that is the sequence simple have the answer yes. So in your own code, you can affirm that these autodata structures that somebody else did is in fact a virtual subclass of a sequence. So this is a facility that Python has with the way that ABCs were implemented and then there is register method of ABCs that allow you to register some other type as a virtual subtype of this type.
54:44 That's cool.
54:44 Yeah, so that's the idea of Goose typing, a different kind of waterfowl, but it is related to Ducks in some way.
54:54 Yes. Nice.
54:58 And then there is a chapter about inheritance where I discuss multiple inheritance, which again is something that a lot of people do not have contact with because many of languages do not implement that. And then there is a chapter about Control Flow, this was the hardest chapter for me to do, because- basically because Asyncio was very new-
55:26 Yeah, you have a lot of stuff on basically parallelism or Async you've got concurrency with Asyncio, concurrency with futures, what is the story with the futures part?
55:36 Yeah, so the thing is, there is a module called concurrent features, right, where this module actually works with threads and processes under the covers. But it was the first package in the Python standard library that implemented this idea of futures, which is something that is gaining a lot of concurrency. So futures are basically objects that represent pending computation. So the idea is that when you submit to some concurrent API a task to be done, usually you submit please run this function, right, so the API immediately in a non blocking way returns to you a future that represents that pending computation, and then you can query this future to know whether the thing that you asked to be done is actually done or not.
56:39 And when it is done, you can actually get the result from the future as well. And another thing that the future does, is that is extremely useful whenever you are doing Async work or threaded work is that it encapsulates some exceptions that was raised when the system was running, that function let you 57:01 to run. And this is really important, because what happens is when you are running code asynchronously and an exception happens, it is happening in another context, right. So how do you catch it?
57:14 And it is really interesting, the way that it works is the future object catches it, and stores it for you. And then later when you actually go look at the future object and for instance if you request the result from the future, if instead of an exception was raised, the exception is raised at this point in your program. So it teleports in a way the exception that happen in this other context that you have where you have no control to a context where you actually either you want to look at the result of the computation but actually there was an exception and this is where the exception is raised and you get to handle it.
57:54 Yeah, that's excellent. Because normally you think that the exception is going up the call stack, until somebody catches it but on other thread you didn't start or manage, right, that's not what you need exactly, cool.
58:07 Yeah.
58:07 Awesome. And then in the last section you have metaprogramming, right?
58:11 Exactly. So the metaprogramming part is about how attributes actually work in Python. So I start with some simple examples using the 58:23 and get attr special method for example that allows you to emulate like a virtual attributes and I use it, I have a real data stream from OSCON last year, they had all the talks so they had a few thousand records in JSON format and I implement a few classes to make it easy to navigate through this OSCON data and then what else- then I talk about properties and then the next chapter is about descriptors which is the infrastructure for properties.
59:00 And then I developed an example that actually evolved from a talk that I gave at PyCon at 2013, in Santa Clara, it was called "Encapsulation with descriptors", the talk, but in the chapter I can spend much more than 45 minutes with that, or 30 minutes actually, anyway, and the last chapter is about class metaprogramming. And I didn't call it metaclasses, although metaclasses is the most famous way of doing metaprogramming with classes in Python and other languages that allow it. We now have an easier way of metaprogramming with classes which is using class decorators. So then I show some examples with that, and a real metaclass, and that wraps up the book.
59:56 Yeah, that's really cool. There is a bunch of stuff that I am looking forward to dig into, if I have a few more days, especially the concurrency looks really interesting. So, congratulations on your book, it's very well done and people can get a lot out of it.
01:00:13 Thanks a lot. I will just mention one thing that I am proud of in the concurrency part of the book, actually the control flow part. The coroutines chapter; I wanted to show people the coroutines used in an interesting way which was not linked to asynchronous programming which is the main usecase for them these days. And so I decided to implement this 60:37 and simulation of taxi cabs. So this is an example that I am proud of, that I came up with this example that shows and gives you when you run the simulation if you actually run the script, it gives you an intuition of how actually you can use coroutines to manage many things happening at the same time in the example there are different cabs that are on the street catching passengers and dropping off passengers, and so but it doesn't involve an event loop, but it is a simulation, and so there is a very simple scheduler implemented in about ten lines of Python, and makes all those cabs run, and actually the cabs are implemented as coroutines. So,...
01:01:25 That sounds really cool. I definitely want to check that out. So before I let you go, there are two questions I always ask my guests on the way out the door. The first one is what is your favorite code editor?
01:01:36 I use Sublime text.
01:01:38 SUblime, that's a good one.
01:01:41 Yeah, although my- I have a partner in my training company, that uses PyCharm and sometimes I get I think of moving to PyCharm but Sublime is... I prefer simple things, and right now I am very happy with Sublime text.
01:02:02 Yeah, excellent. And other question is what is your favorite PyPi package? There is all these different things out there, 60 000 plus whatever, and there is just so much raise people awareness of, so what is your top list?
01:02:17 Well, Requests is probably a common answer.
01:02:23 That's one of my favorites.
01:02:23 I think it's really cool, and I actually use it in the book as one of the few external libraries that I use even though that there is a library that does pretty much the same in the standard library, and I use it because it is much simpler, but also because it runs the same on Python 2 and Python 3. Our HTTP client functionality in the standard library has always been not very good, and because it's not been very good, it's being changed a lot in Python 3, without getting much better, but enough to becoming compatible.
01:03:04 Lovely.
01:03:04 So, I think Requests if you want to write an HTTP client is something that you really have to go for because then your code will be easier, nicer and it will run in Python 2 and Python 3 at the same time.
01:03:23 Yeah.
01:03:25 The only thing that I wish Request had, and it doesn't is support for Asynchronous requests. One of the things that you learn in my book is that you can actually do very effective network code in Python despite the GIL, with threads because of the fact that every single function in the standard library of Python that does Blocking I/O releases the GIL. So, if you test it, if you run it you will see, actually accepting very extreme cases, the performance of thread in applications and asynchronous applications in Python is pretty much the same. So you can use Request with threads for doing very fast HTTP clients.
01:04:11 Yeah. That's great, because most of your blocks are waiting on the network anyway, right?
01:04:15 Exactly.
01:04:17 Cool. Yeah, we had Kenneth Reitz on the show, the sixth episode and we talked a lot about Request, it was very cool.
01:04:23 I was very glad I think Kenneth gave some manifestation that was positive about my book on Twitter, maybe he favored somebody else's tweet that was positive about my book but I was very proud of that because he is really an example of a guy who can do things in a very pythonic way. If you want to learn what pythonic is, study Kenneth's code.
01:04:51 Yeah, he is very good at creating APIs.
01:04:54 Yeah.
01:04:55 All right, any final shout outs or call to actions? People should go read your book, they should go check out your book, right?
01:05:05 Yeah, sure. It was a huge investment for me, I don't know if most of your readers know but it's pretty impossible to make a living writing books. It's sort of like how to make a living playing guitar.
01:05:22 Yes.
01:05:25 There is about very few people in the world that do it, that manage that. Anyway, so I have to recover the investment in time that was huge, I spent 18 months writing this book and the second half of that time was full time because my editor started complaining it was not in the schedule. There were few postpones but then she noticed that I was writing a lot, the book grew, It started as a project for create 200 pages book, but I am really happy with the result and really proud and I am really happy with this chance that I had to work with Meghan Blanchette my editor, she was awesome, and the support that I had from the technical editors, Alex Martelli, and Leonardo, Lynda, Victor 66:27 who was a special guest tech editor for asyncio chapter, they were awesome and it has really been a great experience and I am really happy to see how people are enjoying the book.
01:06:41 Yeah. It was quite an investment but it hopefully pays off for you because it is definitely good work.
01:06:47 Yes, thanks a lot.
01:06:48 Yeah, you are welcome, thanks for being on the show, really appreciate it.
01:06:52 Thank you very much Michael.
01:06:54 You bet, bye.
01:06:55 This has been another episode of Talk Python To Me.
01:06:55 Today's guest was Luciano Ramalho and this episode has been sponsored by Hired and CodeShip. Thank you guys for supporting the show!
01:06:55 Hired wants to help you find your next big thing. Visit hired.com/talkpythontome to get 5 or more offers with salary and equity right up front and a special listener signing bonus of $4,000 USD.
01:06:55 Codeship wants you to ALWAYS KEEP SHIPPING. Check them out at codeship.com and thank them on twitter via @codeship. Don't forget the discount code for listeners, it's easy: TALKPYTHON
01:06:55 You can find the links from the show at talkpython.fm/episodes/show/24
01:06:55 Be sure to subscribe to the show. Open your favorite podcatcher and search for Python. We should be right at the top. You can also find the iTunes and direct RSS feeds in the footer on the website.
01:06:55 Our theme music is Developers Developers Developers by Cory Smith, who goes by Smixx. You can hear the entire song talkpython.fm/home/music.
01:06:55 This is your host, Michael Kennedy. Thanks for listening!
01:06:55 Smixx, take us out of here.