#117: Functional Python with Coconut Transcript
00:00 Michael Kennedy: One of the nice things about the Python programming language is it's at least three programming paradigms in one, there's the procedural style, the object oriented style and the functional style of programming, this week, you'll meet Evan Hubinger who is taking Python's functional programming style and turning it to 11, we are talking about Coconut, a full, functional programming language that is a proper superset of Python itself, this is Talk Python, episode 117, recorded May 22nd, 2017. Welcome to Talk Python to Me, a weekly podcast on Python, the language, the libraries, the ecosystem and the personalities, this is your host, Michael Kennedy, follow me on Twitter where I'm @MKennedy, keep up with the show and listen to past episodes at talkpython.fm, and follow the show on Twitter via @talkpython, this episode is brought to you by Rollbar and Hired, thank them both for supporting the show, check them out @rollbar and @hired_hq on Twitter and tell them thank you. Before we jump into the main part of the show, I wanna let you know that we're adding a couple of new courses over at Talk Python Training, within the next few weeks, we'll be releasing Restful HTTP services with Pyramid and Python, including a SQLAlchemy data driven backend and another course, MongoDB for Python developers, if you're interested in either of these, be sure to create a free account over at training.talkpython.fm to get notified. Evan, welcome to Talk Python.
01:55 Evan Hubinger: Thank you, thank you for having me.
01:56 Michael Kennedy: Oh, it's an honor to have you, I'm really excited, you've got this really cool project that I think is doing some interesting stuff in the Python space and so I'm pretty excited to share this new functional superset of the language that you've created, but of course, before we get into the details of Coconut, let's talk about your story, how'd you get into programming and Python?
02:17 Evan Hubinger: So I started programming actually as a sort of summer camp thing between middle school and high school. It was not a very well taught summer camp, actually, it was really just we're gonna throw some Python at you, I took a look at it, I sort of experimented with it and it was interesting. What really got me excited about it was just the things that I started to be able to do once I just was exposed to it and then started learning about Python online, going through the documentation and building up, I started writing text games and worked my way up to more complicated mathematical stuff to follow along with what I was learning in school and as I went through I sort of worked my way through the various different things in Python, working my way to know the language better and better, all from this start just as what is this thing, this thing that actually lets you take a computer and get a command and actually output something that's interesting, useful and powerful, so that was my beginning, my introduction, the way that I got started.
03:25 Michael Kennedy: That's cool, and the first language was Python that you learned?
03:27 Evan Hubinger: My first language was Python, yeah, and then I used, yeah. I've used Python pretty exclusively for, like, many years after that, my second language that I really got into after that was Haskell.
03:40 Michael Kennedy: Okay, yeah, Haskell is definitely one of the more popular functional programming languages, right, and so did that kind of blow your mind, you're, like, whoa, Haskell's different?
03:48 Evan Hubinger: Absolutely, yeah, so, I at that point had been pretty used to Python, that was sort of two years after I had originally learned about Python and since that time in doing a lot of Python like I was saying, and Haskell was this new experience, this experience of there's all these new concepts, all these new features, all these new powerful things I've never seen before, I was blown away by how powerful pattern matching was, I was blown away by how powerful all of the sorts of different constructs that I had available to me, the different things that I could do in Haskell that I had never seen before and one of the first things, the sort of mythical origin story of sorts was one of the first things I did when I tried to learn Haskell is I went back and I tried to, like, do all of those things in Python and of course, that didn't go as well as I was hoping and that was the sort of basis and jumping off point for what I would later do.
04:43 Michael Kennedy: That's really cool, is that the thing you called Rabbit?
04:47 Evan Hubinger: Yes, so originally, so the very first programming language that I wrote was Rabbit, that was just a sort of project that I was making, like, I'm going to take all of these cool Haskell features that I think are awesome, all of these new things that I'm learning and I'm gonna put them in Python, I'm gonna write an interpreter on top of Python, of course, that's a terrible idea for what should have been obvious reasons.
05:12 Michael Kennedy: How long did it take, like, a week?
05:14 Evan Hubinger: Yeah, so I wrote that, it was fun, it was a cool, interesting project, would not recommend trying to write code in Rabbit but it was a big learning experience for sure and a lot of the lessons that I learned in that process I would later apply.
05:34 Michael Kennedy: Yeah, that's cool, and Rabbit is actually on GitHub so I'll put it in the show notes if people wanna check out your first attempt, they can do that, right?
05:41 Evan Hubinger: Yes, that's on my GitHub.
05:42 Michael Kennedy: Nice, all right, so what do you do day to day? You're still in college, right?
05:47 Evan Hubinger: Yes, so I'm a student at Harvey Mud, I'm a rising junior.
05:51 Michael Kennedy: And for people listening in other parts of the world where's Harvey Mud?
05:54 Evan Hubinger: Harvey Mud is in Claremont, which is a suburb of LA in California.
05:59 Michael Kennedy: Yeah, excellent, okay, and then you also did some intern work at some pretty interesting companies, right?
06:04 Evan Hubinger: Yeah, so I am currently an intern at Google, last summer I interned at Yelp and before that I was an intern at Ripple.
06:12 Michael Kennedy: Oh, that sounds, that's pretty awesome, man, what do you think of that experience?
06:16 Evan Hubinger: It's been a blast, I've really liked all of the companies that I worked for, I've done a lot of really interesting things as part of those jobs and Yelp, of course, that was really exciting last year because that was all in Python.
06:29 Michael Kennedy: Oh, yeah, nice, so you're a Math CS grad, right?
06:32 Evan Hubinger: That's correct, I'm a Math CS major.
06:34 Michael Kennedy: Nice, so yeah, I also studied math, not so much CS, I guess I did a bit of a minor type thing, it wasn't called that but more or less and so how do you feel like that's prepared you for this? For these internships and some of these experiences?
06:50 Evan Hubinger: Obviously, the computer science portion of it is there, but the math I think really is also there as well. I know that in my work at Yelp, there were points where I used graph theory, points where I used some number theory, so the stuff really does come up where you want to have that background to be able to fall back on to help you handle weird, interesting, challenging problems.
07:16 Michael Kennedy: Right, more on the algorithmic side, yeah, nice, I also feel like math, if you get really into math, like advanced math, proofs and real analysis and things like that, it teaches you how to think and solve problems and it doesn't really matter the rules around it, if they're programming rules, they're programming rules, right, I think it's pretty good for a foundation.
07:37 Evan Hubinger: Yeah, I totally agree with that.
07:38 Michael Kennedy: Awesome, so you've had this Haskell experience and of course there's a number of different functional programming languages, why don't we talk a little bit about, like, what is functional programming, I mean, it's not just using functions, right? It's entirely different than that, isn't it?
07:55 Evan Hubinger: Yeah, so the sort of textbook classic definition of functional programming is that it's all about pure functions, it's all about immutable data structures, and all of that is true, functional programming does involve pure functions, it does involve immutable data structures, but one of the things I think a lot of people get frustrated by with that definition is that it's sort of exclusive, it says what functional programming is not, it isn't about mutability, and it isn't about functions with side effects, but it doesn't say what functional programming is, and what functional programming is is really those are the parts that Coconut is really trying to nail down because Coconut isn't a pure functional programming language, but what it brings to the table and what functional programming brings to the table and what Coconut brings to the table with it is the way that you work with those pure functions and those immutable data structures, having them, the things that that lets you do, that lets you have things like higher order functions so you can take a paradigm standard practices and turn them into functions that you can then apply in general concepts, it lets you have pattern matching, instead of having to explicitly define how you're going to match or take a look at a particular incoming data type, it lets you just define a general pattern for what the data type looks like and how it gets deconstructed.
09:10 Michael Kennedy: Right, maybe the best way to, like, conceptualize that is it's a little bit like function overloading but way more general, right?
09:19 Evan Hubinger: Yeah, yeah, that's definitely a good way to think about it because when you're overloading a function, I guess, in like C or C++, you're doing it based on just types, and pattern matching is sort of what if you could do that not just on types but on all sorts of different attributes of the data, what if I could have a different function call for when it was an empty list versus when it wasn't an empty list and that's yeah, that's one of the big things that pattern matching lets you do.
09:44 Michael Kennedy: Awesome, let's dig into that, yeah, and you also have lazy evaluation, there's some other follow ons or corollaries from this that are very, I think one of the reasons driving functional programming is with this immutability and no side effects, paralellization becomes what they call embarrassingly parallel, right, where it's just, like, well you just break it into pieces and you just do that, right? Because you don't have to worry about one piece getting ahead of another, right?
10:15 Evan Hubinger: Yep, exactly, exactly, paralellization becomes really trivial, really easy to do when you have your functions written, your data structures written in a functional style.
10:25 Michael Kennedy: Nice, nice nice, so give us some examples of existing functional languages.
11:07 Michael Kennedy: Sure, and if we think about Python, Python itself has some functional components, right, we could go like full in OOP programming in Python, we could go quite deep into the functional programming, you can, or just straight imperative code, right, we can sort of pick some pieces up and choose a style for Python, right?
11:29 Evan Hubinger: Yeah, absolutely, I mean, that's one of the things that I love about Python, Python really is great at being a multi paradigm language, you really can do all of the functional programming stuff, one of the things of course that I would say attests to that is that Coconut can compile into Python and make it all work, even when you try to import it from Python, but there's a lot of things that while they technically work, really don't work nicely so when Python is designed, the sort of whole philosophy around it, the use cases that are considered really aren't generally functional programming use cases, you know, the syntax is designed, when the various different use cases are considered, it's mostly thinking about and considering object oriented use cases, and that's fine, Python's a great object oriented programming language but it means a lot of the functional programming use cases will get left behind, so you have things like Python's lambda, which is notoriously ugly as well as functools.partial, which is difficult to use, you get a lot of things that don't really, features you would really like to see from a functional standpoint that you really don't see at all, features like tail call optimization or tail recursion elimination, features like pattern matching and features like immutable algebraic data types, a lot of that stuff is missing.
12:43 Michael Kennedy: Right, the very dynamic nature of Python does make it a little bit harder to have immutable data types, for sure, and yeah, its primary thing is not in its functional style, I would say, whatever it is, maybe it's OOP, maybe it's not quite, maybe it's something slightly different, but functional is not its primary focus, do you think that the adoption of things like Scheme and lists in Haskell and whatnot is somewhat limited by its pureness?
13:15 Evan Hubinger: I would say that's definitely true because it really is much more difficult to try to learn a pure functional programming language than it is to try to get, especially if you're coming from an imperative, non functional background, than it is to try to go with a language that is partially functional, Haskell is great but getting started with Haskell isn't easy, there's a lot to learn, there's a lot of new concepts that you may not be familiar with, even if you've programmed for a long time in other programming languages and so that really does create a barrier to entry.
13:48 Michael Kennedy: I guess so, it definitely seems like it would be a challenge to me, so you decided to add one more language to this list of, this spectrum of functional programming languages, yeah, so what's this project called Coconut?
14:02 Evan Hubinger: Yes, so that's true, so Coconut is a functional programming language that compiles to Python so some of the basic design decisions of Coconut, Coconut's syntax is a strict superset of Python 3 syntax, so Coconut takes Python 3 and adds on top of that new features for functional programming, that includes things like algebraic data types, tail call optimization, pattern matching, and then Coconut takes all of that and compiles it to universal version independent Python.
14:34 Michael Kennedy: Yeah, the version independent part is pretty awesome. And it's version independent in certain ways, right? Like the syntax that you write into your Coco files, that's Python 3, right?
14:45 Evan Hubinger: Exactly, so Coconut syntax is based on Python 3, it's a strict superset of Python 3, but then that gets compiled to Python code that is compatible with 2.6, 2.7, 3.2, and above.
14:58 Michael Kennedy: Right, so the execution engine can be either Python 2 or Python 3 but you input Python 3. Or pure Coconut, right?
15:06 Evan Hubinger: Yes, that's exactly correct, Coconut code you input is a strict superset of Python 3, but the compiled output is universal or you can specifically target one Python version or another, there are some cases where you are forced to target a Python version, one example of this is if you wanted to do something like 3.5 matrix multiplication there's no way to back port that to Python 2 and so you're going to have to pass a flag that tells the compiler, I only care about 3.5.
16:14 Evan Hubinger: Yep, that's exactly it, so that was a purposeful design decision on my part from the very beginning, one of the really core things that, like I was saying earlier, learned from Rabbit was I wanna make sure to have a coherent story for how people are gonna adopt this, why people are gonna use this, and how even just as an existing Python developer, I can make something that is useful to you and one of the really fundamental ways that I try to achieve that is via this we're gonna make the whole syntax a superset of Python 3, that way you already know, if you know Python 3, most of the core syntax of the language, you don't have to learn anything new to be able to do things you already know how to do, the only things that you get to learn new are new features, new things that you can add to your repertoire, so the point is not to relax, reduce the amount that a Python developer is capable of but only to expand and to allow you to be able to do new things in Coconut that you wouldn't otherwise be able to.
17:21 Michael Kennedy: That's really cool, and I can tell you from going through the tutorial, there's some really mindbending stuff in there that we're gonna get to, it's pretty awesome, so but before we get into the details, why did you choose Coconut as a name?
17:32 Evan Hubinger: Oh, as a name. So that's a reference to Monty Python and the Holy Grail.
17:38 Michael Kennedy: And the Knights Who Say Ni and those guys clicking along?
17:40 Evan Hubinger: Yeah, they've got to follow behind the knights of the round table, smashing the coconuts together, so yep.
17:47 Michael Kennedy: That's awesome, yeah, yeah, if I was gonna guess that's what I thought, but yeah, perfect perfect. It's really funny that a lot of people think Python is some kind of snake reference and of course it's not, right?
17:58 Evan Hubinger: Yeah.
18:00 Michael Kennedy: Hey everyone, Michael here, let me take just a moment and thank one of our sponsors who makes this show possible, this portion of Talk Python to Me has been brought to you by Rollbar, one of the frustrating things about being a developer is dealing with errors, relying on users to report errors, digging through log files, trying to debug them or a million alerts just flooding your inbox and ruining your day. With Rollbar's full stack air monitoring, you get the context, insight and control you need to find and fix bugs faster, adding the Rollbar Python SDK is just as easy as pip install rollbar, you can start tracking production errors and deployments in eight minutes or less. Rollbar works with all the major languages and frameworks, including the Python ones, like Django, Flask, Pyramid, as well as Ruby, .NET, Node, iOS, and Android, you can integrate Rollbar into your existing workflow, send error alerts to Slack or Hipchat or automatically create new JIRA issues, Pivotal Tracker issues and a lot more, they have a special offer for Talk Python to Me listeners, visit talkpython.fm/rollbar, sign up and get the Bootstrap plan free for 90 days, that's 100 thousand errors tracked for free, but you know, just between you and me, I hope you don't encounter that many errors, give Rollbar a try today, just go to talkpython.fm/rollbar. Yeah, so you've covered some of the reasons why you've built it, but, like, maybe give us some of the technical underpinnings of, like, why do we need this new language instead of just different PEP practices or whatever to work in this way.
19:25 Evan Hubinger: Sure, so I think Coconut really does add a lot of very fundamental changes to Python syntax, so it's a very hard to do a lot of that stuff via just, like, the PEP process, I don't even think that that would necessarily be a good thing if sometime along the road all of that, all of Coconut's syntax was implemented in Python, it really is trying to be a separate language because it adds a lot of new stuff on top of Python that is specifically for being able to do a lot of these more advanced functional things to add to some of the more basic things that you can do in Python, as well, I would say, that as far as like why use Python, why get into it, what is the more technical underpinnings as well, first, Coconut really is built to be useful to existing Python programmers. You should be able to just pick it up, write a Coconut file and immediately as you work your way through the tutorial, as you work your way through the documentation see new things that you can just add in to your existing knowledge of Python just to make your code more powerful, just to make it so that you have new access to new features as well as make the version compatibility story really easy for you.
20:37 Michael Kennedy: Yeah, and I really like the fact that you take one input language variation, Python 3, but then it becomes execution independent.
20:47 Evan Hubinger: Yeah, exactly, one of the really nice things about compiling to Python source code is that it really is truly independent, you can run Python 3 even without adding new Coconut features or if you do add new Coconut features, run it through Coconut, get a compiled source that because it's still Python you can even run on PiPy, you can run on Jython or Iron Python, you don't even need to, like, if it compiles a CPython bytecode for example, then it'd be very restricted, but it really is just pure Python and so wherever you wanna be able to use Coconut constructs you can.
21:49 Evan Hubinger: There are no plans to do something like that right now, that would be a pretty massive undertaking but certainly possible, I suppose sometime very far down the road, I could see Coconut going that direction, but that's.
22:04 Michael Kennedy: I'm not proposing that it should, I'm just kind of curious what your thoughts on that were.
22:08 Evan Hubinger: I think that for a lot of that stuff, there are a lot of utilities that already exist for taking Python code and making it available in other frameworks and other places, and Coconut can run right on top of those things because it's just producing Python source code.
22:22 Michael Kennedy: Right, so you really wanna just stay focused on enhancing the functional programming story within the Python space purely?
22:28 Evan Hubinger: Right, Coconut has a pretty singular mission that is really focused on I'm gonna make Coconut be able to allow you to do functional programming really well in Python, you don't have to deal with any of the weird syntax ugliness, any of the, like, missing features, everything's gonna work, you're able to just take all of these powerful functional programming features and just run it in Python.
22:49 Michael Kennedy: Nice, so what if I don't know functional programming, right, maybe I'm self taught and I've learned Python and whatever, like, should I bother to learn or play with Coconut, like, what do you think?
23:01 Evan Hubinger: So I would simply definitely say yes, my argument would actually be that Coconut is a really nice way to learn functional programming, because it means that you can start with everything you know about Python already being applicable so you don't have to, like, if you're trying to learn functional programming through Haskell, for example, you have to rebuild up all of that knowledge about programming in general and all of the different ways to perform all of the different operations, whereas with Coconut, all of that basic syntax is just Python, and it's only building functional stuff on top of that that you have to learn, and so I think that trying to learn Coconut really is a good way to start getting into different functional programming features, and the tutorial for Coconut is I think a great way to do that.
23:50 Michael Kennedy: Yeah, that's really cool, it's definitely a way to expand your mind of what's possible. So how do you install it, is there, like, some download I gotta do or what's the story?
24:01 Evan Hubinger: Pip install Coconut, that's it.
24:03 Michael Kennedy: Beautiful, that's the way it should be, right? And then once you pip install Coconut, you have a new CLI, you just say Coconut thing and then stuff happens, right?
24:12 Evan Hubinger: Yeah, most basic command line interface is just Coconut name of a .coco file and then it'll compile that into a Python file, there's of course more complicated stuff than that, there's, like, command line flags and whatnot, but the most basic syntax is just that.
24:25 Michael Kennedy: Yeah, so you basically run it through this compile step and then you've got a set of Python files laying there and you just execute them like you would anything, right? Any Python program.
24:35 Evan Hubinger: Coconut can compile and execute if you want it to take care of both of the steps to get past the -r flag, but it can take care of a lot of different things, it's got a lot of different features but most fundamentally it's a compiler, transpiler, whatever you wanna say, you give it Coconut code, it turns it into Python code.
24:50 Michael Kennedy: Sure, okay, so one of the pieces of syntax, like, it's a little, it's beyond a little, it is hard to talk about syntax in audio format so I don't want to do it too much but I want to touch just a little bit on a few things because a little bit of it will give people a strong idea so one of the things that's kind of annoying is if I wanna take one function and pass its value to the next function and pass its return value to the next function, I basically have to do that inside out, the innermost, if I wanna do it in one expression and not multiline, right, I take the first function and then I wrap it in a call to the second function and I take that and wrap it in another one and so you don't read it normally, you read it, like, inside out, which is not great so you have this syntax in Coconut that says left to right, take this and flow it to, almost like a Unix pipe sort of thing, right?
25:46 Evan Hubinger: Yeah, pipeline operators in Coconut are borrowed from F#, they exist in a lot of other languages as well, like Elm, but yeah, the basic idea is it's a much more intuitive way to read and think about function calls, it really is one of the more syntactic sugary features, it's not like something like pattern matching or tail call optimization that is sort of a more powerful thing behind the scenes but I really do think it's a good example of a way in which just from the most basic level, before you even start delving into the more complex functional programming features, it just immediately makes your code easier to read easier to understand, easier to follow because like you were saying, that is how we more intuitively natively think about functions as you take some value, you apply a transformation to it then you get something new and you apply another transformation to and when you write it with that syntax that's exactly the way that it looks and so I do really think that it makes it look nicer.
26:48 Michael Kennedy: Yeah, it is pretty interesting, let's talk about some of the other features, some of 'em are pretty straightforward, so you have an alternate lambda syntax.
26:57 Evan Hubinger: Yes.
26:58 Michael Kennedy: So give us a sense of what that is and also why did you create that, it's not that different.
27:03 Evan Hubinger: No, so the alternate lambda syntax is just an arrow, it's the hyphen and then the greater than sign which is already used in Python 3 for example for the return type annotations, but the basic idea operating in other programming languages, you use lambdas a lot in functional programming and the word lambda is somewhat ungainly, you don't really wanna have to write it everywhere, of course you still can in Coconut because it's a strict superset of Python 3, but it gives you this new syntax that lets you work with it more nicely, make it easier to read, again, another one of the more syntactic sugary features, Coconut does have a lot of nice sugar to play with.
27:44 Michael Kennedy: Yeah, there are some special cases where there's, like, super short versions of these lambda expressions where they basically expand out in the arguments, right, like X comma Y, do, like, star, parentheses and that'll do, like, an expanding multiplication or something like that, right?
28:03 Evan Hubinger: Yeah, I think you're referring to a couple of different things, the sort of star and parentheses, that's a syntax that is somewhat borrowed from Haskell, but you can get an operator function, which in Python you would have to do normally via import operator and then operator.add or something, you just put the operator in parentheses which is another nice syntactic sugary thing, as well as you can then like compose that with other functions via the function composition operator, partially apply that via Coconut's partial application operator, and so you can work with these operator functions, work with the built in functions to build up sort of more complex composite functions out of those in a very functional manner.
28:45 Michael Kennedy: Yeah, that's very cool, all right, let's dig into some of the things that are, like, farther out there, let's say. Let's start with algebraic data types, those seem pretty easy to describe on air.
28:58 Evan Hubinger: All right, I'll give it a shot. So algebraic data types are fundamentally first of all immutable and so that means that they've just got, they're sort of like classes, but they've just got that one constructor at the very beginning, you pass in the data that's gonna be inside of it, and then it's done, that's, you have a new data type, sort of like a tuple.
29:19 Michael Kennedy: Yeah, to me it looks like you take the class word and you change it to data and then it has a sort of convention for the initialization because it can't be changed once it's initialized, right?
29:29 Evan Hubinger: Yep, yep, you change class to data and then you put the arguments to the constructor effectively, the different things that will be inside of it as instead of, like, the inheritance right there in the syntax, but then there's a lot of things that are more complex, more beyond just the fact that okay, it's an immutable data type and it's got this syntax, some of the things that are really cool that you can do with algebraic data types are first of all is pattern matching, you can pattern match against an algebraic data type because you can say in the pattern, like, for example, let's say I have a two vector that I've defined as an algebraic data type and then I want to write a function that just gives me that first element, if it's a two vector and it's got an X position and a Y position, and I can pattern match and I can say, like, two vector of X comma Y in my function definition, so, like, F of, def F(x,y) and then it'll actually match any incoming data that is this two vector, decompose it into the components based on that pattern, say okay, this value matches the pattern and I know from the pattern, I see that there's this X and this Y that I need to bind to the argument, the values inside of the algebraic data type, and it deconstructs it into its component parts in the pattern and then you can take out and work with those components, so the pattern matching component of algebraic data types is really powerful, really useful stuff, there's some other cool stuff you can do with algebraic data types, too, Coconut provides an map function that lets you map over the contents inside the algebraic data type to produce a new algebraic data type, and there's a bunch of other stuff as well along those lines that you can do with algebraic data types, that are more powerful than just, okay, it's immutable, but it is immutable and that is important.
31:17 Michael Kennedy: Right, and do you use something like slots or something internally and properties to say, there's a property with the name of the thing you added but in fact it's, like, a hidden field, how do you make it immutable?
31:29 Evan Hubinger: Internally it's slots and named tuple are very sort of heavily relied upon, there's some other magic that has to go on sometimes, if you are somewhat creative with the way you define the constructor, like, if you use a star for example, but in general, yeah, it uses named tuple and slots pretty heavily behind the scenes.
31:53 Michael Kennedy: Yeah, interesting, maybe tell people what slots are, I know some people know, but not everybody, it's kind of an underadvertised feature of classes in Python, not that you should use it often but maybe you should know about it, right?
32:06 Evan Hubinger: Happily, yes so slots, specifically it's __slots__, it's a very useful attribute that you can set on any class, and if you set __slots__ to be some container like I said, __slots__ would be a tuple of some strings, then each of those strings, those become the only valid attributes for that class from that point so they're the only instance attributes that you can have and so for every instance that you make of that class, only the things in __slots__ are allowed to be changed, modified, reassigned, added to it.
32:40 Michael Kennedy: Right, even in __init__, in your own class, you can't add, like, another thing if it isn't pre specified in __slots__, right, so that's a good way to get a lot of constraints for immutability, there's also some interesting memory performance tradeoffs that you get with that as well, but let's not get sidetracked. You talked about algebraic data types and how they support pattern matching, let's talk about pattern matching, there's, like, a couple ways which you have this concept which obviously I never really done much with, I know Scheme, but that's about it for functional languages, so tell us what pattern matching is and like, how it works.
33:20 Evan Hubinger: Yeah, so I sorted of introduced us a little bit when I was talking about pattern matching against algebraic data type, but in its general form, pattern matching says I have a specific type of data that I'm interested in working on this function, so I wanna maybe define a function specifically for dictionaries, that have the form or one of the keys corresponds to a list of these parts, and that's pretty common, when you have functions that operate on specific types of data, and pattern matching lets you define the specific type of data that you want the function to operate on and then it lets you deconstruct the parts of that match that you actually wanna use in the body of that function, it gives you bindings to them and then it also lets you in Coconut you can use add pattern for this, lets you dispatch to multiple functions based on which patterns match and so you can say, for this type of data that matches this pattern, I wanna do this thing, but for this other type of data that matches this pattern, I wanna do this thing instead so pattern matching is a way to both check against your data as well as deconstruct it and make assignments to it.
34:31 Michael Kennedy: Yeah, one of the things I like in the function part of it, not necessarily the sort of error handling or case matching component, is what might have been one big function with many if statements now becomes separate functions, one for each particular case, right, so you were having an example with factorial and you say, literally the syntax is def factorial(0) = 1, what the heck does that mean?
34:59 Evan Hubinger: Hahah, yes, so that is valid Coconut, so what's happening there is you are defining a factorial function, and you are saying the argument is the pattern (0), (0) is a valid pattern and it means it matches only the very specific data that is the number 0 and then the =, that's just a little bit of syntactic sugar for an implicit return and so the last expression after an equals in a function definition is automatically returned, and so.
35:28 Michael Kennedy: Yeah, it doesn't have to be static, right, it can be computed?
35:31 Evan Hubinger: Yeah, it's just whatever the last expression is, that gets returned.
35:34 Michael Kennedy: Okay, cool, and then you have this @addpattern, and you can take and create another function called factorial and instead of having, like, n for the number being passed in you have def factorial(n is int if n > 0), all right, tell us what this does.
35:53 Evan Hubinger: Okay, yeah, so also valid Coconut, so what's going on there is you have a little slightly complex pattern, but that's still what that is, that (n is int if n > 0), that's another pattern, and that's doing a couple of things, it's first n is int, that's a type check, that says I only wanna match n if it's an integer, and then the if n > 0, that's a guard, and that says if my pattern matches, perform this extra check to make sure that it is > 0 before going in it, and say that the pattern doesn't match if that extra check at the end fails.
36:29 Michael Kennedy: Yeah, that's really cool, so basically the way you implement the factorial is you've got this one more complicated one that we just talked about and it just more or less calls itself, right, over and over with one fewer values in there and then eventually that pattern doesn't match, it falls back to the one that returns one and that's, like the degenerative case of the recursion and boom, you're done, right, but there's no if statements, you've written factorial with no if statements recursively.
36:59 Evan Hubinger: Yep, that's exactly correct.
37:00 Michael Kennedy: You know, that's actually, that reminds me of being confused in college. I remember I'd kind of played around with languages like C++ and stuff, and then I took the Scheme course, a derivative Lisp, and there were a lot of things that were, like, okay, we're going to implement this with no if statements, or, you know, the Scheme itself doesn't even have, like, a looping construct in the language, there's no loops, so loops are actually done with recursion and these sorts of crazy things, right, and I just remember, thinking wait, do this with no if statements, that's just kind of crazy, man, that can't work, can it?
37:41 Evan Hubinger: Right, well, so, as a functional programming guy, that doesn't worry me as much though it doesn't have to, even if it does worry you, that's not a problem in Coconut because you've still got your loops there, you've still got your if statements there, but you get to use all of these other very powerful features that sort of can in some ways replace a lot of that as well, and so it's, I hope, more friendly than trying to jump headfirst into Scheme.
38:08 Michael Kennedy: Yeah, you know, it wasn't that it worried me so much as it just kind of blew my mind, like, I didn't think of programming, like, I had this one view of programming which was very imperative and you know, looping and ifs and whatnot, and conditionals, and then they're just like, no, there's this completely different way of writing code and I think that's one of the values of even if you don't ever wanna use Coconut, I would encourage people to go through the tutorial because it's quite involved and you really do, you do get to think, or let's say challenge your own paradigms of programming, right?
38:43 Evan Hubinger: Yes, definitely, if you, functional programming really is a new interesting powerful paradigm that if you haven't seen it before really is nothing like what exists in a lot of other programming languages.
38:57 Michael Kennedy: Yeah, I feel like functional programming is one of those languages that you read more and you write less and by that I mean, you look at it and you go, okay, I know it's two lines, but they're very deep, I'm gonna have to contemplate them for a little bit, right, do you find that?
39:12 Evan Hubinger: Yeah, I mean, it's definitely true where it's, like, you can be very succinct and to the point in functional programming because, like, you know, in the factorial example, it lets you define a lot of the logic without being as, without having to go through the motions of how you would actually implement it, it lets you just say factorial of zero is one, you figure out how to actually check to see if the value is zero and then return the right thing, I'm gonna tell you what you need to know, the most basic information, which is zero should map to one, and you can take care of the rest and that's sort of how functional programming works in a lot of different cases and it's that nature of it that is one of the things that makes it so powerful.
39:54 Michael Kennedy: This portion of Talk Python to Me is brought to you by Hired, Hired is the platform for top Python developer jobs, create your profile and instantly get access to thousand of companies who will compete to work with you. Take it from one of Hired's users, who recently got a job and said, I had my first offer within four days and I ended up getting eight offers in total. I've worked with recruiters in the past, but they were pretty hit and miss, I tried LinkedIn, but I found Hired to be the best, I really like knowing the salary upfront and privacy was also a huge seller for me. Well, that sounds pretty awesome, doesn't it? But wait until you hear about the signing bonus. Everyone who accepts a job from Hired gets a $300 signing bonus, and as Talk Python listeners, it gets even sweeter, use the link talkpython.fm/hired and Hired will double the signing bonus to $600. Opportunity is knocking, visit talkpython.fm/hired and answer the door. So let's talk about some of the other features, another one, if you haven't done a lot of functional programming you're probably not familiar with it, but one of the problems you can have with recursion is the call stack only has so much space on it, right. If you need to do what would effectively be many many many loops or very very deep recursion, you can end up with an error, right, so you have a special way to solve this, really cleanly, and I'm not sure I totally understand how you determine how to solve this, you have to tell me, but ...
41:17 Evan Hubinger: All right.
41:18 Michael Kennedy: Tell me about your tail call optimization that you do.
41:21 Evan Hubinger: I'll go for it, yeah, so in Python that manifests via maximum recursion depth exceeded error and what happens is every time Python sees a function call, it effectively creates a new stack frame. It says, you know, pops all the existing or pushes all the existing variables onto a stack and then enters this new frame with all of these new variables and it does that execution but what that means is every time you call a new function you're building up more memory, you're building up more function calls and it can get very intensive, doing that a lot, and so what Coconut does is it actually looks at every single function you write and it says, any point, if you directly return a call to another function, then we don't actually need to keep around the stack frame that corresponds to your current function call because all of the data that is necessary to figure out what the return statement with the actual return value of that function is going to be, is all contained just in this other function call because you've directly returned that function call, and so whenever Coconut sees something of that form, it automatically optimizes it away to get rid of the necessity of, like, making the, adding this new stack frame, there's actually two ways in which Coconut does that for two different cases, the most extreme, most powerful case is tail recursion elimination, if you write a function that returns a call to itself, then you can refactor that into a wall loop which is really nice and very fast, if it returns a call to another function, that's a little bit slower but you can still prevent the new stack frame from being created, but it's a slightly more complex, more involved mechanism.
43:05 Michael Kennedy: Yeah, it works really nicely and with the pattern matching, with the tail recursion optimization and stuff like that, one thing that I really liked was I was playing with the different versions of your implementation, and the pure Python, nearly pure Python version was slower than the highly Coconut style, the ones that leverages these special features the most, and so I thought that was pretty cool, actually, that it's not, like, well, you have this better syntax but you're gonna suffer for it, but eh, who cares, you don't need that much performance anyway, that you actually didn't, at least didn't pay a price for using it but it seems like it actually might even be less of a price than not.
43:51 Evan Hubinger: Yeah, in most cases, Coconut's performance is gonna be very similar to Python's because of course, it compiles to Python, but yeah, in some cases Coconut really can perform optimizations that will save you, one of the biggest of those optimizations is the tail recursion elimination, tail call optimization where it'll take a look at every function you write and just see, are you returning a call to another function directly, implicitly or explicitly, in any way, and if so, that gets optimized away, always, and that's really nice for adding, like, you were saying, it means that in some cases, Coconut code actually can be significantly faster than the equivalent Python code, in most cases, at the very least, it's usually not slower just because it really is Python, it just compiles to it.
44:37 Michael Kennedy: Yeah, that's really cool, and you could of course write those optimizations yourself, because after all this does compile to Python, you could write what the compiler wrote but that's a much harder deal, the goal is to have super simple code that is actually optimized.
44:51 Evan Hubinger: Yeah, exactly, the nice thing about the compiler, you can look through the compiled code, it'll just be a file right there but you'll see that a lot of it really does get very complex, very hairy, very, especially with pattern matching, a lot of it gets really weird and hard to read because pattern matching can get very complex very quickly, and having a compiler figure out the sort of ways to implement all of that for you is really nice.
45:20 Michael Kennedy: Yeah, the way I think of it is people are very bad at writing state machines, compilers are really good at it, sometimes you end up with stuff like that, right, so speaking of debugging and trying to understand, like, the thing that you're going to actually debug when you run a debugger, be that some CLI one or something like PyCharm, you're going to see the transpiled compiled output, right, so how do I know if that maps back, like, where does this map back to this funky syntax that I maybe have in my coco file?
45:50 Evan Hubinger: Yeah, so that's it, absolutely, great question, debugging is of course a huge massive part of what every programmer does all the time, so Coconut has a really clean simple story for that, so you can pass the line numbers flag, which is just -l and what that does is it appends to every line in the compiled Python a comment that says, take a look at this line in the source code that produced this line in the compiled code, then you do pdb.settrace, you get an error and you see the lines that error is referring to and you'll see on those lines, comments that are telling you what line in your source code caused that exception, what line in your source code are you tracing through? All of that stuff you can see, and you can just look at that line number, you can go right back to the source code without having to really worry about and deal with the compiled code at all so that way, when you're debugging, you don't have to do any context switching, now there's also another option, the keep lines options, which actually instead of a line number will just put the entirety of the line, it will just paste it right in there for you as well, so that's another option.
47:01 Michael Kennedy: Yeah, those are both pretty good stories, that's cool, so what about parallelization primitives?
47:07 Evan Hubinger: Coconut tries to make parallelization as easy as possible because you can have all of these nice immutable data structures and ideally you wanna be able to use them in a nice way, of course, in CPython at least, Python fights back against you a little bit there, you've got the global interpreter lock, but what you can do, what Coconut provides, is two primitives, there's a parallel map and a concurrent map which are multiprocess and multithreaded respectively, which take the most basic of the functional programmers' operations and the one that is most easily parallelizable, the map operation, which just says apply this same thing, the same function to every element of this sequence, this data, and it allows you to really really easily do that operation automatically in multiple processes or in multiple threads if you're doing IO bound stuff, such that you don't have to mess with any of the libraries, you don't have to mess with anything fancy whatsoever, you just swap out wherever you're doing the map operation, that really parallelizable operation, with the parallel version and you're automatically getting all of the benefits of running that in parallel and of course, behind the scenes, it's using multi processing and it's pickling everything ...
48:16 Michael Kennedy: Yeah, that's really cool, so basically it's still a blocking function as far as your code is concerned but the internal representation maybe changes the order in which it processes them and it just fans it out, right, all right, awesome, and then one final feature we should talk about, maybe, is partial application, is this your $ operator?
48:38 Evan Hubinger: Yes, but the $ operator's a little bit more than that, so the $ operator in Python in Coconut just means lazify this, it means take this operation in Python and make it lazy instead of doing the Python thing and so that means different things in different contexts, but there are sort of two contexts where it's used, so the big one, yeah, you put it right before function call and that means instead of actually doing this function call, I want you to do it lazily and don't do it until I actually tell you I want it and so that is partial application, that says, here are the arguments that I'm gonna pass to this function when I actually call it, but I might pass more arguments later and so for right now you can just throw these arguments in it and hold them there and that's that partial application, the other usage of the dollar sign is for indexing slicing with brackets and in that case it's a lazy form of that as well that doesn't actually perform the operation and until you ask for the data, like an iterator and lets you do it on arbitrary iterators, so you can for example slice, do sort of complex slicing operations on iterators instead of just on lists or tuples which lets you make use of the laziness of iterators in a much more effective manner.
49:51 Michael Kennedy: Yeah, that sounds really pretty awesome, you were definitely have some cool uses and examples in your tutorial, so one of the things that you talked about in the tutorial is the integration or the capability to use Jupyter or IPython notebooks and so when I was playing with your projects, I'm, like, oh, I was just at Pycon, I just saw Jake VanderPlas's really cool, science-y keynote and got me excited about Jupyter and I had of course played with it before but not a lot and, like, I'll just explore your library through Jupyter and it was really fun so you have pretty good support for that, right?
50:28 Evan Hubinger: Yeah, so Coconut has support for Jupyter through both a kernel, which means instead of running your code through IPython, you just select Coconut instead and it works exactly like IPython, including, and this is actually somewhat of a new feature, including support for things like IPython magic and all of the sort of the nice fancy stuff that IPython throws in there and then, sort of on top of that, you have support for all of the Coconut utilities, all the Coconut language features as well and so it's basically just like an IPython notebook but instead of writing Python, you're writing Coconut. If you only wanna do a little bit of Coconut, you can also use Coconut as an extension inside of IPython and then it'll just take all of Coconut's built ins, give you access to those and then whenever you wanna use Coconut specific syntax, then you just do the align or block magic specifically just to run the Coconut code.
51:26 Michael Kennedy: That's really cool, and of course, data science type of work is a place where functional programming would really be applicable, right?
51:35 Evan Hubinger: Absolutely, yeah, the stuff that people do in data science and in machine learning, it really does apply to the functional paradigm really well because a lot of that stuff is, first of all, it's data transformation, instead of, like, a lot of the very heavily stateful stuff that happens in other areas of programming and a lot of that stuff, I would argue, still can benefit from a lot of the features of functional programming, when you're doing something that really is fundamentally, it's just got the one data that you're passing along, this is the data that I collected, I wanna do some statistical transformations on it, I wanna get insight into what it's like, what you're doing is you're passing it through transformations at each point where the functions themselves are also stateless transformations and that is exactly the paradigm, exactly the approach that functional programming is built for and so it really does lend itself well to being able to make use of a lot of these constructs from functional programming, especially like easy parallelization, if you just want a down and dirty really easy way to just, like, process this data faster and a lot of the Coconut's very powerful utilities for working with iterators so that you can construct your data and work with it in a lazy fashion so you only have to deal with what you really need.
52:53 Michael Kennedy: Right, when you have billions of rows but you only wanna find a few of 'em, lazy is good.
52:57 Evan Hubinger: Yes, exactly.
52:58 Michael Kennedy: Another area, it sounds like, this might be really helpful is in finance, I feel like functional programming gets applied in that area a lot as well.
53:06 Evan Hubinger: Yeah, OCaml is really big there for example and so that's totally true, it's another area where we really want to be able to do a lot of the transformations on the data, to be able to see what's happening, as well as one of the nice things about using immutable data structures is you can always really easily just take a snapshot, just, like, okay, give me whatever the immutable data structure was at that point, and you know, once you have a reference to it that it's never gonna change and so it's very easy, like, if you're doing finance and you wanna see, like, okay, what was it like at this point, you can really easily just store references to immutable data structures because you know once you've got a reference to it, it'll never change.
53:44 Michael Kennedy: That's really interesting, yeah, that's cool.
53:46 Evan Hubinger: Yeah, and so that's another feature that's really useful, particularly in finance.
53:49 Michael Kennedy: Where else do you see functional programming really, people benefiting from adopting it?
53:54 Evan Hubinger: I'm a believer in functional programming being very broadly applicable in almost all cases, like I said I worked at Yelp and while I was there, one of the things that I did in my spare time was I went through some of the code base and tried to rewrite some of the things in Coconut to try to get it working and even in, like, a web application, even in something that doesn't seem functional at all, a lot of those constructs, just being able to make use of them gives you a lot more power in being able to define the logic in a simpler, easier manner that I think is really basically always applicable.
54:32 Michael Kennedy: Yeah, awesome, if I go to PYPI and randomly grab a package could I use it in a Coconut app, like, are there restrictions where this doesn't work?
54:41 Evan Hubinger: Yes, absolutely, in fact you could use it regardless of whether that package was built for Python 2 or Python 3, because Coconut of course can compile to code that works on either version, but yeah, because Coconut compiles just to Python anything that you would be able to access, anything you would be able to do from Python, you can do from Coconut, so that includes importing any package that you wanna import, also the vice versa is true as well, if you want to from Python code import and make use of anything written in Coconut code, that is also really easy to do, because the Coconut code ends up as Python code and so it's really easy to import that as well. You basically ship the compiled Python files as a deliverable, right, very cool, I think we're getting close to the end of the show so we should probably start to wrap it up and I'm asking about your favorite editor, but before we do, let me ask you an editor question about Coconut. So one of the things that I really like about editors is when they remind me what features are on a class or in a module and tell me when I have it right or wrong or give me auto completion and things like that. Do I give that up if I take on Coconut? So that's a good question, so the answer is we're working on it, so right now you can, there is one method that I have tested that does work with that, if you use Jupyter console, I know I think the notebook as well, you can get the Coconut Jupyter or IPython kernel to do some auto completions for you, I think as far as I'm aware, that's the only one we've got so far, but I'm working on it, that's definitely, you know, there's other people that's working on it as well, that's definitely something that should be coming out in a more broader way pretty soon, I'm gonna try to add it into Coconut's just basic command line interpreter as well.
56:30 Michael Kennedy: Yeah, it would be great, I mean, it's already really cool, but it would be even more cool if it could help us along the way, right, if you could get into Sublime, VS Code, PyCharm, those sorts of things, right?
56:43 Evan Hubinger: Yep, definitely a cool feature, being worked on, for right now, try using the Jupyter IPython.
56:50 Michael Kennedy: Nice, okay, and so then let me ask you, if you're gonna write some Coconut or Python code, what editor do you open up?
56:58 Evan Hubinger: So I use Sublime, I'm a Sublime fanatic. I love Sublime, I use Sublime for everything, I write my school notes in Sublime, there are now a bunch of highlighters for Coconut but the Sublime one, that was the one that I wrote, that was the sort of first one to be made and I'm the one that maintains that, I of course use that whenever I write Coconut code so yeah, I love Sublime, that's definitely my editor of choice.
57:21 Michael Kennedy: Nice and so there's also many many many PYPI packages, over 100 thousand now, which is totally awesome, let me go ahead and throw out Coconut as one of the really cool packages there, but maybe give us another one.
57:35 Evan Hubinger: Yeah, yeah, there's pip install Coconut, but yeah, so my favorite PYPI package would definitely be pyparsing, so pyparsing is awesome, it's a really cool utility for being able to define very simple grammars, really complex grammars as well just in pure Python and then parse against them, Coconut's compiler is pretty much entirely written in pyparsing, so you can tell that I really like it, it's really powerful, it works really well, Coconut pushes pyparsing almost to its limits in a lot of cases, to the point where I had to submit a bunch of issues in the development of Coconut to pyparsing, but all of that's been fixed, the newest version of pyparsing is awesome, it works really well, would definitely recommend checking out pyparsing, I actually used it for another project while I was at Yelp, Undebt, which allows you to automatically refactor code by defining pyparsing pattern.
58:31 Michael Kennedy: Yeah, it looks for, like, code debt and duplication type things, something like that?
58:35 Evan Hubinger: Yep, exactly, lets you define patterns using pyparsing grammars, as well as a bunch of other sort of Undebt specific stuff, and then you can define a like replacement function and it'll scan for the grammar and then replace it with that and you can use that to really easily refactor code.
58:52 Michael Kennedy: Cool, yeah, it's a great recommendation, and I think it's really neat that your work actually expanded pyparsing a little bit, but also its existence made your work pretty much possible already.
59:04 Evan Hubinger: Yep, it's a symbiotic relationship.
59:07 Michael Kennedy: It definitely is, it definitely is, okay, so this brings us to the end of the show, final call action, or do you have quite a few people who've starred this and are playing with it on GitHub or are you looking for people to kick the tires to contribute, what's the story?
59:23 Evan Hubinger: So yeah, so Coconut is built on GitHub, everything's on GitHub, definitely check out the GitHub, star it on GitHub for sure. I would definitely recommend trying to at least take a look, go through the tutorial, take a look at the documentation, take a look and see if you know, maybe this is something that you could really make use of because I of course think it's really cool and that people can really benefit from it and if you have any questions, if you wanna look around, there's a lot of resources for that, you can check out Coconut's Gitter, which is a chatroom where you can just ask questions and then there's a bunch of Coconut developers there including me that can just answer questions, check that regularly, you can also check out all of the sort of documentation resources, there's a frequently asked questions, thorough documentation for all of the specification on features and a tutorial to sort of give you basic introduction, and then of course, if you are interested in contributing to Coconut I would love to have more people get involved, more people take a look, it's all open source, just take a look at the issues, fork it, make changes, submit a pull request, if you wanna ask any questions about it, if you're trying to get started, definitely goes back to the Gitter, just ask around on Gitter and I'll be happy to help anybody out there, so yeah.
01:00:43 Michael Kennedy: Yeah, yeah, awesome, and if people are thinking of contributing, like, what kind of crazy language do they have to learn to like work with this, what's it implemented in?
01:00:50 Evan Hubinger: Python!
01:00:51 Michael Kennedy: Yeah, that's awesome, I noticed the Gitter, if GitHub is to be believed, click that little bar, and it'll tell you the languages used, it's 99.7% Python and .03% makefile or something like that.
01:01:06 Evan Hubinger: Hahah, yes, there is one makefile, so I hope you don't shy away too much for having to deal with that, but yeah, no, it's all Python. Coconut, the compiler just like the compiled source is written in Python and is intended to be universal, so it works on 2.6, 2.7, 3.2 and above, but yeah, it's all just Python, pure Python.
01:01:25 Michael Kennedy: All right, well, this is definitely an interesting project you have going and I'm happy to see it succeeding and thanks for sharing it with everyone.
01:01:31 Evan Hubinger: Absolutely, thank you so much for having me, it was really fun to be here.
01:01:34 Michael Kennedy: This has been another episode of Talk Python to Me, today's guests was Evan Hubinger and this episode has been brought to you by Rollbar and Hired. Rollbar takes the pain out of errors. They give you the context and insight you need to quickly locate and fix errors that might have gone unnoticed until your users complain, of course, as Talk Python to Me listeners track a ridiculous number of errors for free at Rollbar.com/talkpythontome. Hired wants to help you find your next big thing. Visit talkpython.fm/hired to get five or more offers with salary and equity presented right up front and a special listener signing bonus of 600 dollars. Are you or your colleagues trying to learn Python? Well, be sure to visit training.talkpython.fm. We now have year long course bundles and a couple of new classes released just this week. Have a look around, I'm sure you'll find a class you'll enjoy, be sure to subscribe to this show, open your favorite podcatcher and search for Python, we should be right at the top, you can also find iTunes feed at /itunes, Google Play feed at /Play, and direct RSS feed at /RSS on talkpython.fm. Our theme music is Developers, Developers, Developers by Cory Smith, who goes by Smix. Cory just recently started selling his tracks on iTunes so I recommend you check it out at talkpython.fm/music, you can browse his tracks he has for sale on iTunes and listen to the full length version of the theme song, this is your host, Michael Kennedy, thanks so much for listening, I really appreciate it, Smix, let's get out of here.