WEBVTT 00:00:00.001 --> 00:00:04.140 We all know that Python is a major player in the application of machine learning and AI. 00:00:04.140 --> 00:00:08.440 This often involves grabbing Keras or TensorFlow and applying it to a problem. 00:00:08.440 --> 00:00:13.280 But what about AI research? When you're actually trying to create something that has yet to be 00:00:13.280 --> 00:00:19.360 created, how do researchers use Python here? Today you'll meet Alex Levin, a Python developer 00:00:19.360 --> 00:00:24.020 and research scientist at Vicarious, where they're trying to develop artificial general 00:00:24.020 --> 00:00:31.600 intelligence for robots. This is Talk Python To Me, episode 124, recorded May 31st, 2017. 00:00:31.600 --> 00:01:01.280 Welcome to Talk Python To Me, a weekly podcast on Python, the language, the libraries, the 00:01:01.280 --> 00:01:05.900 ecosystem, and the personalities. This is your host, Michael Kennedy. Follow me on Twitter, 00:01:05.900 --> 00:01:10.820 where I'm @mkennedy. Keep up with the show and listen to past episodes at talkpython.fm 00:01:10.820 --> 00:01:16.400 and follow the show on Twitter via at Talk Python. This episode is brought to you by Linode and 00:01:16.400 --> 00:01:20.700 Talk Python Training. Be sure to check out what the offers are for both of these segments. 00:01:20.700 --> 00:01:24.620 It really helps support the show. Alex, welcome to Talk Python. 00:01:24.620 --> 00:01:25.860 Hi, thanks for having me. 00:01:25.860 --> 00:01:31.540 Yeah, I'm pretty excited to talk about AI, researching AI, and all the stuff that you guys 00:01:31.540 --> 00:01:35.800 are up to. It's something we haven't really touched that much on the show, and it's 00:01:35.800 --> 00:01:42.120 both interesting at a low-level, technical level, but also at a philosophical level, almost. So 00:01:42.120 --> 00:01:43.540 looking forward to talking to you about it. 00:01:43.540 --> 00:01:48.180 Yeah, but before we get into the details, let's start with your story. How did you get into programming 00:01:48.180 --> 00:01:48.680 in Python? 00:01:48.680 --> 00:01:53.980 Well, my background is a bit untraditional for what I do. I actually studied mechanical 00:01:53.980 --> 00:02:00.700 and aerospace engineering at school, and I was determined to make a career as a spacecraft 00:02:00.700 --> 00:02:09.200 engineer. And even some of the Mechie assignments where we used MATLAB, I hated those. But that 00:02:09.200 --> 00:02:12.660 eventually led me to Carnegie Mellon to study space robotics. 00:02:13.420 --> 00:02:13.920 Oh, nice. 00:02:13.920 --> 00:02:19.980 And the atmosphere there is incredible. It's hard not to get caught up in software and AI. And 00:02:19.980 --> 00:02:21.380 that was really my first exposure to AI. 00:02:21.380 --> 00:02:28.880 That's cool. When I think of universities that are cool and amazing in programming and robotics 00:02:28.880 --> 00:02:35.580 and things like that, Carnegie Mellon is right up there. Maybe Stanford, MIT, Carnegie Mellon. Those 00:02:35.580 --> 00:02:37.340 are the top three, at least in my mind. 00:02:37.500 --> 00:02:42.440 Yeah, absolutely. And I was fortunate enough to work with this incredible roboticist, Red 00:02:42.440 --> 00:02:48.940 Whitaker, who was one of the, I guess, founders of the DARPA Grand Challenge of autonomous vehicles. 00:02:48.940 --> 00:02:52.560 Oh, I was going to ask you about the DARPA Grand Challenge. Tell people about what that is. That 00:02:52.560 --> 00:02:54.060 is such an amazing thing. 00:02:54.060 --> 00:03:17.140 Yeah, so that started back in 2003 or 2004. And it's just a challenge to create an autonomous vehicle that drives across the desert down in Mexico. And the first year was, you ask most people, a complete failure, but it really kicked off all this research. And then the next year, it was won by, I believe, Sebastian Thrun and his team from Stanford. 00:03:17.140 --> 00:03:18.060 Yeah, I think so. 00:03:18.780 --> 00:03:22.940 Now it's just almost a piece of cake for these autonomous vehicles to complete the course. 00:03:22.940 --> 00:03:30.840 Didn't the Stanford team eventually more or less move over to Google for their Google car stuff? Do I have that put together right? 00:03:30.840 --> 00:03:39.780 Yeah, I believe a lot of them went over there. And Sebastian Thrun, I think he is very involved in the Udacity effort. They have a self-driving cohort. 00:03:40.820 --> 00:03:58.080 Okay. Yeah, yeah. That's really cool. So when you talk about the car driving through the desert, it's not like it just drove a little ways. There's basically a dirt path through the open plains as well up into the mountains for like 100 miles, right? 00:03:58.080 --> 00:04:13.300 Oh, yeah. You can check out some of the videos from these races. And these are like very intense off-road vehicles. It's through the desert in Mexico, I think on the Baja Strip. And I want to say it's like hundreds of miles. 00:04:13.300 --> 00:04:21.240 Yeah, yeah, that's really cool. So yeah, there's a great Nova on this called the Great Robot Race. I'll be sure to include it in the show notes. 00:04:21.440 --> 00:04:21.920 Oh, cool. 00:04:21.920 --> 00:04:42.360 Yeah, it's really, really neat. But it's super inspiring. And this is not the first year, but the second year that they did the documentary. So it's not where they all failed. And they had different devices, different machines. People had robotic motorcycles. So people had huge trucks. And it seems like the SUVs were the real winners, though. 00:04:42.360 --> 00:04:53.040 Yeah. And it's also worth checking out the DARPA challenge for autonomous human robots. There are some pretty funny fail videos up there on YouTube. 00:04:53.040 --> 00:05:04.620 Okay, so I derailed your story, but I definitely wanted to just talk about that because that's like that was the world that you were in at Carnegie Mellon, right? And you're like, well, okay, this requires programming, right? 00:05:04.620 --> 00:05:21.040 Yeah, yeah. So I was studying mechanical engineering and doing things like CAD and FEA analysis and all that. And my last semester there, I ended up taking one software, well, it was a software slash AI course. And it was all in MATLAB, of course. 00:05:21.880 --> 00:05:35.420 And that really just got me hooked. I loved it. And I figured after graduation, I would just figure out a way to just pivot my career. And I wanted to do AI and software and AI research specifically. 00:05:37.320 --> 00:05:56.200 And then, well, Python seemed to be a natural fit. And I started teaching myself Python. And then, at the same time, dove into any AI textbook and paper I could get my hands on. And then, as soon as I could, started implementing some of these algorithms and models in Python. 00:05:56.200 --> 00:06:16.340 And some of the motivation for that came from things like Kaggle competitions and such. Eventually, I felt comfortable enough as a programmer and AI researcher to seek a position with a few different companies. And that led me to Numenta. And I worked there for a few years. 00:06:16.340 --> 00:06:17.960 Cool. What kind of stuff did you do there? 00:06:17.960 --> 00:06:29.700 So, Numenta is interesting. They run the full gambit from theoretical neuroscience all the way into machine learning models that are implemented for production code. 00:06:29.700 --> 00:06:38.740 So, it was an incredible learning environment, both, I mean, from software engineering perspective and about machine learning algorithms. 00:06:38.740 --> 00:06:44.600 But I've since moved over to Vicarious, and I've been here for about eight months now. 00:06:44.600 --> 00:06:46.640 Okay, nice. So, what kind of work do you do now? 00:06:46.640 --> 00:07:07.580 At Vicarious, we're working to build general artificial intelligence. So, that's AGI. Any problem a human can do, we want our AI to do. And in the past, we've pointed to Rosie, the robot from the Jetsons. But I can say that we're not building exactly Rosie, per se. 00:07:07.580 --> 00:07:09.180 More like R2-D2? 00:07:09.180 --> 00:07:10.580 Oh, sure. 00:07:11.760 --> 00:07:23.720 No, okay. That sounds really interesting. And, you know, when you talk about AI, there's usually some kind of split around specialized AI or general AI, right? 00:07:23.780 --> 00:07:37.680 Like, a specialized AI might be Siri or a self-driving car or something, like, or the thing that solves Go. It's, like, incredibly good at driving the car, but there's no way that it could tell you, you know, what's the weather. 00:07:37.680 --> 00:07:40.280 Or, like, something that's unrelated to what has been taught, right? 00:07:40.280 --> 00:07:41.380 Exactly. 00:07:41.380 --> 00:07:43.720 Then you've got the more general one that you're working on. 00:07:43.980 --> 00:07:44.600 Yeah, yeah. 00:07:44.600 --> 00:07:53.140 There's been a lot of press lately about deep learning models and frameworks, and they've been incredibly successful and useful for some of these narrow AI problems. 00:07:53.140 --> 00:08:03.140 A lot of times in research development of these models and algorithms, we'll use video game environments, specifically the Atari environments. 00:08:03.140 --> 00:08:06.820 Just kind of good test beds for the approaches. 00:08:07.540 --> 00:08:19.420 And you take, for example, a breakout or a space invaders environment, and deep learning can blow away any human player. 00:08:19.420 --> 00:08:20.720 It's solved. 00:08:20.720 --> 00:08:31.560 But you add some minor tweaks to the game, like change the pixel values ever so slightly or just move some blocks around. 00:08:31.560 --> 00:08:36.180 And things that are just imperceptible to a human, they pick that up easily. 00:08:36.780 --> 00:08:39.620 The same deep learning model fails. 00:08:39.620 --> 00:08:45.380 So that is kind of exemplifying the narrow component of those models. 00:08:45.380 --> 00:08:47.060 Yeah, that's pretty interesting. 00:08:47.060 --> 00:09:03.280 So one of the challenges with neural networks and deep learning, is that basically like taking multiple neural networks and sort of treating them as a group of things that then answer questions? 00:09:03.280 --> 00:09:05.360 Or is there more to it than that? 00:09:05.360 --> 00:09:06.360 In a way. 00:09:06.520 --> 00:09:07.920 Let me dial back real quick. 00:09:07.920 --> 00:09:08.740 Yeah, okay. 00:09:08.740 --> 00:09:13.680 Artificial neural networks are a model that dates back to... 00:09:13.680 --> 00:09:15.040 Yeah, at least the 90s for sure. 00:09:15.040 --> 00:09:15.280 Yeah. 00:09:15.280 --> 00:09:19.300 And even earlier than that, I feel like I'm offending some people out there by saying... 00:09:19.300 --> 00:09:22.080 I did that in small talk in 1974. 00:09:22.080 --> 00:09:22.860 What are you talking about? 00:09:22.860 --> 00:09:23.180 Exactly. 00:09:23.180 --> 00:09:32.960 But nowadays, in the past five years and so, we've seen that we can stack layers and layers and layers of these neural networks and make them very deep. 00:09:32.960 --> 00:09:37.540 That's where the name deep comes from because we have the computation power to do so. 00:09:37.540 --> 00:09:42.320 So these deep neural networks are essentially function approximators. 00:09:42.320 --> 00:09:44.920 And you can feed them a ton of data. 00:09:44.920 --> 00:09:48.100 It could be videos from your self-driving car. 00:09:48.100 --> 00:09:53.180 It could be text in English and French if you want to do machine translation. 00:09:54.100 --> 00:10:04.520 And these deep function approximators are able to learn from all that data how to take your input as, for example, English and give you the output in French. 00:10:04.520 --> 00:10:05.060 Right. 00:10:05.900 --> 00:10:07.400 Yeah, that's really amazing. 00:10:07.400 --> 00:10:14.420 I feel like AI and machine learning was one of those things like nuclear fusion. 00:10:14.420 --> 00:10:17.980 It was always 30 years out no matter when you ask somebody. 00:10:17.980 --> 00:10:21.740 It's like in the 90s, it was like, oh, yeah, we're going to have AI. 00:10:21.740 --> 00:10:23.140 And here, we're working on this in Lisp. 00:10:23.140 --> 00:10:24.460 And it doesn't really work. 00:10:24.460 --> 00:10:27.900 And then 10 years later, it's like we're still doing the same basic thing. 00:10:27.900 --> 00:10:29.160 It's not really making any progress. 00:10:29.160 --> 00:10:35.720 But I feel like kind of like you said, like the last five years, especially the last three, four years, it's just, wait a minute. 00:10:35.720 --> 00:10:36.860 This is actually a thing now. 00:10:36.860 --> 00:10:42.800 This is like really today there's reasonable uses of AI in tons of ways. 00:10:42.800 --> 00:10:54.820 I mean, we have like Star Trek coming to life with Microsoft Skype and PowerPoint, right, where it like in real time translates to other languages as you speak, right? 00:10:54.820 --> 00:10:55.760 And just things like that. 00:10:55.760 --> 00:10:56.440 I mean, amazing. 00:10:56.440 --> 00:11:00.280 Yeah, there's some really powerful uses already deployed in applications. 00:11:00.840 --> 00:11:02.080 But it's kind of funny. 00:11:02.080 --> 00:11:14.980 The field of research for AI and specifically for general AI is almost always like the mountain climbing dilemma where you think, oh, the peak is just over the next pass. 00:11:14.980 --> 00:11:16.340 And then you get over the next pass. 00:11:16.340 --> 00:11:17.860 And it's like, oh, it's the next pass. 00:11:17.860 --> 00:11:18.780 And then the next pass. 00:11:18.780 --> 00:11:19.480 And the next pass. 00:11:19.800 --> 00:11:23.080 Everybody told me about this mountain range on the other side of this one, right? 00:11:23.080 --> 00:11:24.120 Yeah, exactly. 00:11:24.120 --> 00:11:27.780 So we do have some pretty powerful AI nowadays. 00:11:28.020 --> 00:11:33.600 But we keep pushing the limits and pushing the limits and defining this path to AGI as we go. 00:11:33.600 --> 00:11:34.300 Yeah, yeah. 00:11:34.300 --> 00:11:39.240 So one sequence maybe is you talked about we had the neural networks. 00:11:39.240 --> 00:11:40.360 They could kind of do a thing. 00:11:40.360 --> 00:11:44.080 We had the deep learning neural networks that are really good at stuff. 00:11:44.860 --> 00:11:49.040 But you make minor changes and they're kind of entirely lost, right? 00:11:49.040 --> 00:11:53.560 So it's on one hand, we've built these things that you can train and they do a thing. 00:11:53.560 --> 00:11:56.020 But maybe they can't adjust so much. 00:11:56.020 --> 00:12:02.140 So is one of the new areas of like, how do you evolve like a deep learning model or something like this? 00:12:02.140 --> 00:12:08.160 Yeah, this is a big area of deep learning research is the ability to generalize across tasks. 00:12:08.660 --> 00:12:18.100 So you would be able to take your example, AlphaGo, the deep learning model that has been defeating the Go champions across the world, 00:12:18.100 --> 00:12:23.220 and try to generalize that to maybe just a slightly different game board. 00:12:23.220 --> 00:12:31.220 You can use a larger game board, for example, or even generalize that to a different game entirely, like chess or something. 00:12:31.220 --> 00:12:37.320 But here at Vicarious, we're taking a bit of a different approach. 00:12:37.600 --> 00:12:42.800 We aren't using deep learning as much as other AI research shops around. 00:12:42.800 --> 00:12:50.640 We specifically are taking a more structured model-based approach using probabilistic graphical models. 00:12:50.640 --> 00:12:58.940 And we're finding that our models and algorithms are much more attuned to things like generalization and unsupervised learning, 00:12:58.940 --> 00:13:01.460 which are some problems in the deep learning community. 00:13:01.460 --> 00:13:01.980 Yeah, yeah. 00:13:01.980 --> 00:13:05.180 That sounds so interesting to work on. 00:13:05.280 --> 00:13:12.420 So maybe tell us, like, most of us listening know what it's like to write software, create some utility, write a web app, or whatever. 00:13:12.420 --> 00:13:19.800 How does, like, working on AI research projects, how much is that the same or different than writing a web app? 00:13:19.940 --> 00:13:28.000 Well, the workflow is pretty similar to a lot of software engineering companies out there. 00:13:28.000 --> 00:13:31.400 At a high level, it's really just agile practices. 00:13:31.400 --> 00:13:34.800 We iterate and move very quickly. 00:13:34.800 --> 00:13:41.900 But the difference is these are through experiments, experimenting on our models and algorithms, not really through product releases. 00:13:42.220 --> 00:13:42.460 Right. 00:13:42.460 --> 00:13:44.480 You don't just plan it out and say, these are the features we want. 00:13:44.480 --> 00:13:45.360 I'm just going to build them. 00:13:45.360 --> 00:13:48.080 Because you don't necessarily know what the answer is, right? 00:13:48.080 --> 00:13:48.780 Yeah, exactly. 00:13:49.000 --> 00:13:56.220 So we don't have a Kanban board of tasks and picking up and everything that you discuss at Scrum or something. 00:13:56.960 --> 00:13:59.540 It's a lot more loose than that. 00:13:59.540 --> 00:14:06.860 And researchers here, a lot like developers in agile workflows, have a lot of freedom. 00:14:06.860 --> 00:14:14.860 And we need that to come up with these short-term ideas and long-term ideas and flesh them out and experiment out them. 00:14:15.560 --> 00:14:25.720 So when we're building our code bases and our experiments, well, I like to think of developing and software engineering code into kind of three classes. 00:14:25.720 --> 00:14:29.760 There's research, prototype, and product code. 00:14:29.760 --> 00:14:37.020 And we mostly dabble in the research and prototype code, where research code is you're quick and dirty. 00:14:37.020 --> 00:14:38.920 You're moving fast through experiments. 00:14:38.920 --> 00:14:40.840 Hackey code is okay. 00:14:40.840 --> 00:14:44.200 Test coverage is usually discouraged because it slows you down. 00:14:44.820 --> 00:14:52.540 And then prototype is a step up where we want well-designed, well-architected for data flow and interfaces. 00:14:52.540 --> 00:14:59.300 We want unit and integration test coverage, style standards like pet bait, and documentation. 00:14:59.300 --> 00:15:04.700 But still, the mentality is this is code that will someday be scrapped. 00:15:04.700 --> 00:15:06.220 So you're not married to it. 00:15:06.220 --> 00:15:09.460 It doesn't have to be beautiful 99% test coverage. 00:15:09.460 --> 00:15:10.520 Yeah, that makes a lot of sense. 00:15:10.520 --> 00:15:14.660 It seems like you just want to play with the code in the beginning. 00:15:14.660 --> 00:15:17.480 And then once you have an idea, like, all right, now this looks like it's going to work. 00:15:17.480 --> 00:15:20.540 Let's go try to, like, really build this out and see what this does. 00:15:20.540 --> 00:15:21.640 Well, almost. 00:15:21.640 --> 00:15:27.200 The workflow is a little bit different because it really starts with theoretical and mathematical foundations. 00:15:27.760 --> 00:15:43.020 So when we get some sort of research idea, if it's defined as just a problem that you want to try to solve or some new constraints on a different experiment, we first go through, okay, what are the mathematical foundations of this? 00:15:43.020 --> 00:15:47.980 And we derive things like message passing algorithms over factor graphs. 00:15:47.980 --> 00:15:52.880 And then we take that math and abstract it into software. 00:15:52.880 --> 00:15:54.900 And that's how we build our experiments. 00:15:54.900 --> 00:16:03.220 And then we do simple little toy problems of these experiments to really inspect the models and algorithms. 00:16:03.620 --> 00:16:08.640 And then we get into things like building well-designed software and everything. 00:16:08.640 --> 00:16:08.980 Sure. 00:16:08.980 --> 00:16:11.300 And do you guys use Python for a lot of this? 00:16:11.300 --> 00:16:12.260 Yeah, absolutely. 00:16:12.260 --> 00:16:15.360 Python is the main language that we use here. 00:16:15.360 --> 00:16:19.960 And it's really becoming ubiquitous in machine learning and AI. 00:16:19.960 --> 00:16:23.060 It's elegant with a really simple syntax. 00:16:23.060 --> 00:16:24.580 We can iterate quickly. 00:16:25.080 --> 00:16:27.820 And it's relatively easy for people to learn. 00:16:27.820 --> 00:16:36.200 A lot of AI researchers come straight out of academia and just toiling with MATLAB for their PhD for five years. 00:16:36.200 --> 00:16:39.480 So they need to learn how to actually program. 00:16:39.480 --> 00:16:41.480 And Python is great for that. 00:16:41.480 --> 00:16:44.700 And then on top of that, Python is really powerful. 00:16:44.700 --> 00:16:47.020 I love the phrase batteries included. 00:16:47.880 --> 00:16:55.180 There's so much that comes with the standard library and then tons of PyPy packages that you can throw on top of that. 00:16:55.180 --> 00:17:01.540 And Python sometimes gets some flack for being a little slow. 00:17:01.540 --> 00:17:08.400 But it's really easy to control a C++ backend and integrate with some C or C++ code. 00:17:08.400 --> 00:17:08.680 Right. 00:17:08.680 --> 00:17:11.800 So maybe you prototype it out and this part is not quite fast enough. 00:17:11.800 --> 00:17:18.940 So you could say, well, let's write this in Cython or write in C++ and then bring it into our Python app or something like that. 00:17:18.940 --> 00:17:19.680 Yeah, absolutely. 00:17:19.680 --> 00:17:23.220 Of course, we profile and optimize algorithms first. 00:17:23.220 --> 00:17:32.300 So if an algorithm is something like ON squared because we have nested for loops but can be refactored into something sublinear, we do that. 00:17:32.300 --> 00:17:36.280 And then we do more profiling and see, okay, this is really a bottleneck. 00:17:36.280 --> 00:17:37.740 Let's port it to Cython. 00:17:37.740 --> 00:17:38.920 Yeah, sure. 00:17:38.920 --> 00:17:40.600 So that's a really good point. 00:17:40.600 --> 00:17:51.940 Even though maybe you could make it run 10 times faster or something in Cython, if you have an algorithm that has a super high complexity, it doesn't really matter, right? 00:17:51.940 --> 00:17:55.580 If you're going to have to take more and more data, it's really the algorithm that's the problem. 00:17:55.580 --> 00:18:03.560 And a lot of people have pointed out that working in Python, it's easier to conceptualize and evolve the algorithm. 00:18:03.560 --> 00:18:06.080 And then if you have to, to optimize it further, right? 00:18:06.080 --> 00:18:06.760 Oh, yeah. 00:18:06.760 --> 00:18:07.160 Yeah. 00:18:07.700 --> 00:18:16.800 It's pretty straightforward to have an idea and then say, okay, I'm going to implement this in some nested loops, but God, this is awful and ugly. 00:18:16.800 --> 00:18:20.220 And I'm just going to put it to do and come back and vectorize this. 00:18:20.220 --> 00:18:22.540 And then you come back and fix it. 00:18:22.540 --> 00:18:24.820 And it's just magnitudes faster. 00:18:24.820 --> 00:18:25.140 Nice. 00:18:25.260 --> 00:18:36.660 So maybe tell us a little bit of some of the tools that you use, like Python is pretty obvious, but things like TensorFlow, Carreras, NumPy, SciPy, those types of things? 00:18:36.660 --> 00:18:37.300 Yeah, yeah. 00:18:37.300 --> 00:18:39.800 We use TensorFlow for some of our projects. 00:18:39.800 --> 00:18:50.100 It's really becoming very popular in the AI and specifically deep learning community where distributed computing over tensor graphs is really valuable. 00:18:50.280 --> 00:18:50.380 Yeah. 00:18:50.380 --> 00:18:53.600 Maybe for people who are not familiar with TensorFlow, just give them a quick elevator pitch. 00:18:53.600 --> 00:18:54.120 What is it? 00:18:54.120 --> 00:18:54.860 Oh, sure. 00:18:54.860 --> 00:19:01.780 So TensorFlow, it's offered in more than Python at this point, but the main API for it is Python. 00:19:02.380 --> 00:19:15.840 And it abstracts away a lot of the C and C++ computations for parallelizing tensors, which are basically just symbols that are distributed over these computation graphs. 00:19:15.840 --> 00:19:30.120 And they offer some really cool tools for visualizing these computation graphs as you're doing things like training your model distributed across clusters or running inference on these models too. 00:19:30.120 --> 00:19:30.480 Nice. 00:19:30.480 --> 00:19:30.820 Okay. 00:19:30.820 --> 00:19:31.740 Yeah, perfect. 00:19:31.740 --> 00:19:33.720 And what are the other ones that you use? 00:19:33.720 --> 00:19:38.080 Well, just like any software company, we try to leverage external tools. 00:19:38.080 --> 00:19:39.980 We're not trying to reinvent the wheel here. 00:19:39.980 --> 00:19:49.440 So a lot of our vision utility functions, for example, are handled with toolkits like OpenCV, which is C++ with Python bindings. 00:19:49.440 --> 00:19:51.360 So a lot of that is pretty fast. 00:19:51.360 --> 00:19:58.900 And SciPy, like you mentioned, for operating on NumPy arrays like Gaussian filters and convolutions. 00:19:58.900 --> 00:20:04.060 It sounds like there's a lot of math involved and some pretty interesting programming. 00:20:04.060 --> 00:20:09.060 Do you have like cognitive science people doing almost like brain modeling work? 00:20:09.060 --> 00:20:13.920 What other disciplines are trying to like work together to solve this problem there? 00:20:14.160 --> 00:20:14.620 Yeah. 00:20:14.620 --> 00:20:16.380 So there's definitely a lot of math involved. 00:20:16.380 --> 00:20:22.060 And actually, when people come to me for advice and say, hey, I want to get into AI research. 00:20:22.500 --> 00:20:30.040 And one piece of advice I give them is you need to have your first language be math and then whatever programming language come after. 00:20:30.040 --> 00:20:30.440 Sure. 00:20:30.440 --> 00:20:32.520 And so give us a sense of like how much math. 00:20:32.520 --> 00:20:37.460 Like if I was, say, super good at differential calculus, but that's all I knew. 00:20:37.460 --> 00:20:38.160 Is that enough? 00:20:38.160 --> 00:20:40.180 Do I need to know like statistics and probability? 00:20:40.380 --> 00:20:42.760 Do I need to know linear algebra, graph theory? 00:20:42.760 --> 00:20:46.700 Like what, when you say math, is it a PhD or is it a minor? 00:20:46.700 --> 00:20:55.220 Well, we do have a lot of PhDs in things that are just obscenely complicated and impressive. 00:20:56.220 --> 00:21:01.480 And those are across computational neuroscience to theoretical math. 00:21:01.480 --> 00:21:11.700 But I would say graph theory, like you mentioned, specifically modeling and running computations over probabilistic graphical models is very important. 00:21:11.700 --> 00:21:16.600 And the foundations of which are in probability and linear algebra. 00:21:16.600 --> 00:21:17.180 Okay. 00:21:17.180 --> 00:21:23.240 So not super out of touch math, but you definitely need to be able to speak those things. 00:21:23.240 --> 00:21:24.400 Oh, no, yeah. 00:21:24.400 --> 00:21:28.560 Yeah, like a pretty good, not first language necessarily, but second language. 00:21:28.560 --> 00:21:30.420 Sure. 00:21:30.420 --> 00:21:31.040 Okay, sure. 00:21:31.040 --> 00:21:35.500 This portion of Talk Pythonemy is brought to you by Linode. 00:21:35.500 --> 00:21:39.560 Are you looking for bulletproof hosting that is fast, simple, and incredibly affordable? 00:21:39.560 --> 00:21:44.720 Look past that bookstore and check out Linode at talkpython.fm/Linode. 00:21:44.720 --> 00:21:46.740 That's L-I-N-O-D-E. 00:21:46.740 --> 00:21:50.660 Plans start at just $5 a month for a dedicated server with a gig of RAM. 00:21:50.740 --> 00:21:53.260 They have 10 data centers across the globe. 00:21:53.260 --> 00:21:55.880 So no matter where you are, there's a data center near you. 00:21:55.880 --> 00:22:10.240 Whether you want to run your Python web app, host a private Git server, or even a file server, you'll get native SSDs on all the machines, a 40 gigabit network, 24-7 friendly support, even on holidays, and a seven-day money-back guarantee. 00:22:10.240 --> 00:22:13.620 Want a dedicated server for free for the next four months? 00:22:13.620 --> 00:22:19.100 Use the coupon code python17 at talkpython.fm/Linode. 00:22:19.520 --> 00:22:22.560 It seems like a really fun thing to do, right? 00:22:22.560 --> 00:22:29.200 Like every day you come in and people have new ideas and you try them and you're kind of exploring new space, right? 00:22:29.200 --> 00:22:30.240 Yeah, absolutely. 00:22:30.240 --> 00:22:35.160 There's not really a defined path to AGI. 00:22:35.940 --> 00:22:42.600 And we try to build it from like, what do you think an artificial agent is in the world? 00:22:42.600 --> 00:22:44.840 It is sensory perception. 00:22:44.840 --> 00:22:46.460 It's goal-directed behavior. 00:22:46.460 --> 00:22:49.920 It's building a model from sensory motor interactions. 00:22:49.920 --> 00:22:57.360 And from that, we try to derive the fundamental properties and constraints of intelligence. 00:22:57.800 --> 00:23:02.560 And that leads to our research ideas and eventually experiments. 00:23:02.560 --> 00:23:02.900 Yeah. 00:23:02.900 --> 00:23:08.100 So you talk about like open CV and computer vision and sensory stuff. 00:23:08.100 --> 00:23:13.360 How much of what you're working on do you hope exists in the real world? 00:23:13.360 --> 00:23:23.160 And how much are you thinking like if we could make a digital thing that could just live, say, on devices, on the internet, and only interact digitally, right? 00:23:23.160 --> 00:23:27.380 Not actually see the world, but, you know, interact with people, say, with voice or text or things like that. 00:23:27.380 --> 00:23:27.780 Hmm. 00:23:27.780 --> 00:23:28.500 Interesting. 00:23:28.500 --> 00:23:34.560 So at Vicarious, we're specifically building AI towards robotics. 00:23:34.560 --> 00:23:39.240 Any robot, any task, we build the brain for it. 00:23:39.240 --> 00:23:41.400 That's our initiative here. 00:23:41.400 --> 00:23:44.040 So a lot of it is in the physical world. 00:23:44.040 --> 00:23:48.480 And me specifically, I'm focused on our main vision team here. 00:23:48.480 --> 00:23:48.720 Nice. 00:23:48.720 --> 00:23:54.220 And how much of this do you see as like a consumer sort of thing versus, say, like factories? 00:23:54.740 --> 00:24:00.160 Are you focused on doing this for people or for companies or I guess that's the question? 00:24:00.160 --> 00:24:00.440 Yeah. 00:24:00.440 --> 00:24:06.500 So there's not so much I can really share about the specific robots and applications that we're building. 00:24:06.500 --> 00:24:07.380 Don't share your secrets. 00:24:07.380 --> 00:24:08.480 I don't want those. 00:24:08.480 --> 00:24:13.160 But the idea is like we want to build general AI. 00:24:13.160 --> 00:24:15.360 So all of the things. 00:24:15.360 --> 00:24:15.660 Yeah. 00:24:15.660 --> 00:24:18.400 That sounds really cool. 00:24:19.640 --> 00:24:23.280 What are some of the other companies that are doing this type of work? 00:24:23.280 --> 00:24:32.760 Like I know what maybe companies are using it, but like AI research is a pretty small group of people I would expect. 00:24:32.760 --> 00:24:33.260 You're right. 00:24:33.260 --> 00:24:38.160 There's a lot of companies out there using AI, practicing AI. 00:24:38.160 --> 00:24:49.520 But for doing the nitty gritty fundamental research, the shops are namely OpenAI, DeepMind, which is out in the UK. 00:24:49.520 --> 00:24:51.980 And they were acquired by Google some years back. 00:24:51.980 --> 00:24:53.860 And us, Vicarious. 00:24:53.860 --> 00:24:54.220 Okay. 00:24:54.220 --> 00:24:55.360 Now that's a pretty small group. 00:24:55.360 --> 00:24:57.480 There's also Google Brain. 00:24:57.480 --> 00:24:58.700 Yeah. 00:24:58.940 --> 00:25:02.300 Do you know what the relationship between like Google Brain versus DeepMind is? 00:25:02.300 --> 00:25:03.920 They're not even the same continent, right? 00:25:03.920 --> 00:25:04.700 No. 00:25:04.700 --> 00:25:10.360 And I've heard some conflicting information from people on the Brain team and people on the DeepMind team. 00:25:10.360 --> 00:25:14.980 But as far as I can tell, DeepMind is kind of a separate entity. 00:25:14.980 --> 00:25:17.880 They don't interact too much on projects. 00:25:17.880 --> 00:25:22.360 DeepMind is much more building towards general AI. 00:25:22.760 --> 00:25:30.480 And Google Brain does AI research, but a lot of it is directed more towards the applications for Google. 00:25:30.480 --> 00:25:30.840 Sure. 00:25:30.840 --> 00:25:31.880 Makes sense. 00:25:31.880 --> 00:25:41.540 Do you consider companies like Uber or, you know, Google Alphabets, Waymo, sort of self-driving car type things? 00:25:41.540 --> 00:25:43.740 Would you put them into this bunch? 00:25:43.740 --> 00:25:46.980 Or do you think of them more as like using AI? 00:25:46.980 --> 00:25:48.400 I would say using AI. 00:25:48.620 --> 00:26:05.560 So my understanding of a lot of these companies, specifically the self-driving car companies, is they're looking at what sort of deep learning models can I use for my vision problems, my LIDAR sensor problems, and my motion planning problems. 00:26:05.820 --> 00:26:15.620 And they'll take existing models that are out there from toolkits such as TensorFlow or PyTorch or Keras even. 00:26:15.620 --> 00:26:19.420 And then they'll tweak those for their specific problem. 00:26:19.420 --> 00:26:26.340 So that's more of a hyperparameter tuning type work, not so much the real fundamental research. 00:26:26.340 --> 00:26:26.620 Right. 00:26:26.620 --> 00:26:26.940 Okay. 00:26:27.860 --> 00:26:32.620 And a lot of these guys are open sourcing their tools like TensorFlow and whatnot. 00:26:32.620 --> 00:26:36.040 And I think there was even like a pact about this. 00:26:36.040 --> 00:26:43.380 Like Google and Microsoft and some other people, didn't they team up on like sharing their research and their work? 00:26:43.380 --> 00:26:47.060 It's been like a year since I've heard about this. 00:26:47.060 --> 00:26:48.360 So my details are flaky. 00:26:48.360 --> 00:26:48.900 Sure. 00:26:49.060 --> 00:26:54.980 Well, there's been some collaboration over sharing these research environments. 00:26:54.980 --> 00:26:59.160 So like I mentioned, video games are often a testbed that we use a lot. 00:26:59.160 --> 00:27:05.180 So Microsoft has developed Minecraft to be a testbed for AI development. 00:27:05.180 --> 00:27:11.960 And OpenAI has released their gym for different AI agents. 00:27:12.140 --> 00:27:17.120 And then also Universe, which is just a big collection of all these different games. 00:27:17.120 --> 00:27:21.060 And I believe there was some collaboration or teamwork between the two of them. 00:27:21.060 --> 00:27:21.340 Okay. 00:27:21.340 --> 00:27:21.980 Cool. 00:27:21.980 --> 00:27:22.180 Yeah. 00:27:22.180 --> 00:27:24.200 Universe sounds, I heard about that a while ago. 00:27:24.200 --> 00:27:26.260 That sounds really pretty wild. 00:27:26.260 --> 00:27:32.280 And as an AI research company, do you guys find stuff like that useful? 00:27:32.280 --> 00:27:41.020 Like, hey, I could take this AI and drop it into Counter-Strike or I could drop it into Pitfall and all these different places where it can interact. 00:27:41.020 --> 00:27:41.880 Is this helpful? 00:27:41.880 --> 00:27:42.360 Yeah. 00:27:42.360 --> 00:27:42.680 Yeah. 00:27:42.680 --> 00:27:43.460 Definitely helpful. 00:27:43.460 --> 00:27:52.600 We haven't explored Universe so much here at Vicarious, but we also build some internal environments that are very similar. 00:27:52.600 --> 00:28:03.260 Mainly, we use internal tools because then we can define an API that might give us some different information than some of those APIs exposed. 00:28:03.260 --> 00:28:08.820 But it's definitely helpful to have a lot of testbeds and use cases out there. 00:28:08.820 --> 00:28:09.020 Yeah. 00:28:09.020 --> 00:28:09.100 Yeah. 00:28:09.100 --> 00:28:17.180 It seems like if you were, say, a grad student writing a master's thesis, being able to just plug into different video games is like a super big boost. 00:28:17.180 --> 00:28:26.920 But if you're a full-on mini-person AI research company, you need something that you can, like, exactly tweak the environment just so. 00:28:27.040 --> 00:28:32.340 Like, I want to understand how the robot interacts with, like, the squishiness of an object, right? 00:28:32.340 --> 00:28:32.500 Yeah. 00:28:32.500 --> 00:28:35.620 Which you probably can't get in, like, Unreal Game or something, right? 00:28:35.620 --> 00:28:36.200 Yeah. 00:28:36.200 --> 00:28:36.800 Exactly. 00:28:36.800 --> 00:28:37.360 Exactly. 00:28:37.720 --> 00:28:44.400 And a lot of this open sourcing is specifically targeting those grad students in universities. 00:28:44.400 --> 00:28:53.860 I see open sourcing these toolkits and also for the same reason why all these companies are publishing a lot of their papers as a recruiting tool. 00:28:53.860 --> 00:28:54.180 Yeah. 00:28:54.180 --> 00:28:58.360 Because a lot of AI researchers are very academic at heart. 00:28:58.360 --> 00:29:02.600 We want to share our information and help each other. 00:29:02.600 --> 00:29:11.060 So the more that these big companies are sharing and being open about what they're building, the more they can attract top down. 00:29:11.060 --> 00:29:11.340 Yeah. 00:29:11.340 --> 00:29:19.520 So companies like OpenAI and Google might have a better chance of getting somebody who's mostly interested in the research side of things than, say, Apple. 00:29:19.520 --> 00:29:20.140 Yeah. 00:29:20.260 --> 00:29:20.700 Exactly. 00:29:20.700 --> 00:29:23.100 This has been a problem for Apple as of late. 00:29:23.100 --> 00:29:23.360 Yeah. 00:29:23.360 --> 00:29:27.240 I guess there's a couple questions around universities, like, you kind of touched on. 00:29:27.240 --> 00:29:33.720 One is, do you know if there are, like, machine learning AI bachelors these days? 00:29:33.720 --> 00:29:34.400 Hmm. 00:29:34.400 --> 00:29:36.900 Or is there just computer science and you can focus on that? 00:29:36.900 --> 00:29:46.900 You know, I don't know if you can major exactly in AI or machine learning, but there are definitely a lot of undergraduate courses that you can take. 00:29:47.720 --> 00:29:52.780 And from what I've heard, these courses are becoming ridiculously popular. 00:29:52.780 --> 00:29:58.680 I try to mentor a couple of students from, like, Stanford and CMU. 00:29:58.680 --> 00:30:03.620 And I always tell them, like, oh, there's a probabilistic graphical models course. 00:30:03.620 --> 00:30:04.780 You have to take that. 00:30:06.240 --> 00:30:06.680 Yeah. 00:30:06.680 --> 00:30:06.680 Yeah. 00:30:06.680 --> 00:30:07.320 That's cool. 00:30:07.320 --> 00:30:18.560 It seems like there'd be a lot more money in working at Google on TensorFlow or maybe with you guys or whatever, rather than, say, being a professor at a university. 00:30:18.560 --> 00:30:25.760 Is there, like, a drain of, like, as professors sort of get their research going, do they get sucked out of academia? 00:30:26.140 --> 00:30:26.420 Yes. 00:30:26.420 --> 00:30:29.100 For a typical example. 00:30:29.100 --> 00:30:30.860 I know it's a problem for data science. 00:30:30.860 --> 00:30:32.240 I just don't know about machine learning. 00:30:32.240 --> 00:30:33.740 It seems like there's a parallel, right? 00:30:33.740 --> 00:30:33.880 Yeah. 00:30:33.880 --> 00:30:34.960 Oh, there absolutely is. 00:30:34.960 --> 00:30:44.240 Now, the main example of this is Uber going into Carnegie Mellon and grabbing, really, all of their computer vision researchers a few years back. 00:30:44.740 --> 00:30:49.340 But I think you largely see more of a collaboration in sorts. 00:30:49.340 --> 00:31:10.440 So sometimes top professors from universities will have appointments with some companies like OpenAI or Google, and they'll still work with their universities and their research labs on some of their projects, but at the same time work with the industry partner in a way. 00:31:11.020 --> 00:31:15.940 And that also works as an incredible recruiting tool for these companies. 00:31:15.940 --> 00:31:16.280 Yeah. 00:31:16.280 --> 00:31:20.240 And it also seems like it's probably the best of both worlds, right? 00:31:20.240 --> 00:31:21.200 You still get to be a professor. 00:31:21.200 --> 00:31:26.660 You still get to do your research and teach people, but you also get some real-world experiments and experience. 00:31:26.660 --> 00:31:30.880 You're not just, you know, in your lab poking around, right? 00:31:30.880 --> 00:31:32.620 You get to see really what people are trying to do with it. 00:31:32.800 --> 00:31:46.460 And there's a big difference between trying to just fill out grants and get money and resources through academia versus having some sort of private or public company. 00:31:46.460 --> 00:31:53.080 And if you need just infinite AWS clusters, you can get it. 00:31:53.940 --> 00:31:58.020 Yeah, like I need 100 GPUs in a cluster. 00:31:58.020 --> 00:31:58.540 Can we do that? 00:31:58.540 --> 00:31:59.020 Yeah, sure. 00:31:59.020 --> 00:32:00.200 Here, you push this button. 00:32:00.200 --> 00:32:01.520 Yeah, yeah. 00:32:01.520 --> 00:32:02.580 It was kind of funny. 00:32:02.580 --> 00:32:14.720 Someone did a blog post on like huge spike in AWS GPU demand the week leading up to the paper deadline for NIPS, which was like a few weeks ago. 00:32:14.720 --> 00:32:16.800 It is impressive. 00:32:17.400 --> 00:32:18.980 Yeah, we've got to finish our research. 00:32:18.980 --> 00:32:19.760 We've got to get this going. 00:32:19.760 --> 00:32:20.400 Yeah, it's... 00:32:20.400 --> 00:32:20.920 Yeah, yeah. 00:32:20.920 --> 00:32:25.600 Yeah, so you guys probably do a lot of cluster computation stuff like that. 00:32:25.600 --> 00:32:29.720 Do you use GPUs and other specialized chips or anything? 00:32:29.720 --> 00:32:36.800 For some of the models that can take advantage of the parallel compute on GPUs, like deep learning models, yeah, absolutely. 00:32:36.800 --> 00:32:43.740 But a lot of our larger models are still used on AWS, but not necessarily GPUs. 00:32:43.740 --> 00:32:46.220 Just straight EC2 clusters or something like that. 00:32:46.220 --> 00:32:46.800 Yeah, yeah. 00:32:46.900 --> 00:32:47.600 Yeah, all right. 00:32:47.600 --> 00:32:48.120 Very cool. 00:32:48.120 --> 00:32:53.800 So we talked about the math requirements, linear algebra, calculus, probability, things like that. 00:32:53.800 --> 00:33:01.000 But if people, you know, they want to try this out and they want to get started, like how maybe they know Python, but nothing else. 00:33:01.000 --> 00:33:02.340 Like how would they get started? 00:33:02.340 --> 00:33:15.880 I would recommend getting inspired by a model or a problem, maybe a NIPS paper or something from ICML or even a blog post that shows some pretty cool results on like natural language processing. 00:33:16.400 --> 00:33:17.620 And dive in that way. 00:33:17.620 --> 00:33:18.220 Yeah. 00:33:18.220 --> 00:33:18.540 Yeah. 00:33:18.540 --> 00:33:21.420 Do you have some examples of like what a problem might look like? 00:33:21.420 --> 00:33:22.860 Would they have those at Kaggle? 00:33:22.860 --> 00:33:23.860 Yeah, absolutely. 00:33:23.860 --> 00:33:26.240 That's exactly how I got inspired. 00:33:26.240 --> 00:33:32.000 Going to Kaggle and there are a whole slate of machine learning problems that you can dive into. 00:33:32.300 --> 00:33:38.040 And oftentimes those communities will discuss their approaches and share some of their methods. 00:33:38.040 --> 00:33:43.520 So that is really good, almost a code first way to dive in. 00:33:43.520 --> 00:33:53.420 But like I was saying earlier, to build up the math chops to really be able to do AI research, I recommend trying to reproduce paper results. 00:33:53.420 --> 00:33:58.760 So you pick out a research paper and you go through it. 00:33:58.760 --> 00:34:06.880 You try to understand the models and the algorithms, the assumptions, the experiments, and then you try to implement that from scratch. 00:34:06.880 --> 00:34:08.640 It's not easy. 00:34:08.640 --> 00:34:10.300 You're going to fail a lot. 00:34:10.980 --> 00:34:12.540 But it's the best way to learn. 00:34:12.540 --> 00:34:18.340 And fortunately, if you're doing this in Python, a lot of the Python community is very helpful. 00:34:18.340 --> 00:34:18.620 Yeah. 00:34:18.620 --> 00:34:19.700 So what are some of the tools? 00:34:19.700 --> 00:34:23.020 Maybe the ones we talked about, TensorFlow, Carreras, things like that? 00:34:23.140 --> 00:34:32.540 TensorFlow and Carreras, they have some great prepackaged models that you can take out of the box and even some that point specifically to the papers that they're from. 00:34:32.540 --> 00:34:35.080 So you can get up and running pretty quickly. 00:34:35.080 --> 00:34:35.500 Yeah. 00:34:35.500 --> 00:34:44.640 Maybe a really simple way to get started might be to take somebody who's written a paper, done some project, but it's only in MATLAB, and say, let me try to reproduce this in Python. 00:34:44.640 --> 00:34:46.940 Like, just even started from their code, right? 00:34:46.940 --> 00:34:47.400 Yeah. 00:34:47.400 --> 00:34:47.700 Yeah. 00:34:47.700 --> 00:34:48.500 That's a good idea. 00:34:48.500 --> 00:34:57.620 You kind of, like, take it, like, let me not actually try to solve the problem, but just take it technically through the steps of, like, creating it in Python and seeing how that goes. 00:34:57.620 --> 00:34:58.980 And then you've kind of got that experience. 00:34:58.980 --> 00:35:02.280 And then you can go and try to, like, actually solve it independently. 00:35:02.280 --> 00:35:02.860 Yeah. 00:35:02.860 --> 00:35:11.440 Ultimately, it comes down to just trying to build a link, a mental model of how the math is connected to the software implementation. 00:35:11.580 --> 00:35:20.420 And if that's building the code and then trying to figure out how the algorithms and equations are defined there, that works best for you. 00:35:20.420 --> 00:35:21.080 Go for it. 00:35:21.080 --> 00:35:23.580 If it's more of a bottom-up approach, do that. 00:35:23.580 --> 00:35:23.840 Yeah. 00:35:23.840 --> 00:35:24.380 Yeah. 00:35:24.380 --> 00:35:24.660 Cool. 00:35:25.760 --> 00:35:37.660 So you talked about your dev workflow being very sort of loose, and you start from just let's play around and not worry too much about patterns and factoring our code and doing testing stuff. 00:35:37.660 --> 00:35:42.480 What are some of the pain points that you might end up with from, like, that workflow? 00:35:42.480 --> 00:35:49.020 Because we move fast and it's a lot of research and prototype code, we can build up some technical debt. 00:35:49.020 --> 00:36:03.000 And this largely comes because people, researchers here, want to build the next thing, not put in another few days to refactor their experiment into beautiful abstractions with 99% test coverage. 00:36:03.000 --> 00:36:04.560 And they're probably, they're researchers, right? 00:36:04.560 --> 00:36:11.560 There's more mathematicians and scientists, not pro developers who really care about, like, this design pattern or something, right? 00:36:11.560 --> 00:36:13.060 We have a mix, fortunately. 00:36:13.060 --> 00:36:21.980 So there are some that are very academic researchers who just do some kind of ugly things in the code. 00:36:21.980 --> 00:36:32.380 And fortunately, we have our code review, our pull request review process that can help teach them and be a tool for building better software engineers. 00:36:32.600 --> 00:36:42.640 But then we also have a lot of people here that are very strong engineers and have built up their software chops coming from companies like Google or Facebook. 00:36:42.640 --> 00:36:43.420 Yeah, of course. 00:36:43.420 --> 00:36:45.920 And that's pretty serious development right there. 00:36:45.920 --> 00:36:46.500 Yeah. 00:36:46.560 --> 00:36:56.460 So do you use, like, special, like, any particular tools for, like, refactoring code or for profiling or things like that? 00:36:56.460 --> 00:36:56.700 Yeah. 00:36:56.700 --> 00:37:08.540 So profiling code is really useful around here because, I mean, the biggest mistake a program can do is trying to optimize our code without actually knowing where the pain points are. 00:37:08.540 --> 00:37:08.800 Yeah. 00:37:08.940 --> 00:37:13.820 I find that even harder in this type of computational stuff where it's sometimes not intuitive. 00:37:13.820 --> 00:37:14.340 Yeah. 00:37:14.340 --> 00:37:14.700 Yeah. 00:37:14.700 --> 00:37:19.420 And Python comes prepackaged with some useful profilers. 00:37:19.940 --> 00:37:31.260 I prefer to actually use Kernprof line profiler because some of the Python built-in profilers will really just show you, like, this function is your bottleneck. 00:37:31.260 --> 00:37:34.700 But going line by line, I've found to be much more helpful. 00:37:34.700 --> 00:37:35.320 Yeah, sure. 00:37:35.320 --> 00:37:38.200 So, yeah, here's a 100-line function. 00:37:38.200 --> 00:37:38.620 It's slow. 00:37:38.620 --> 00:37:39.020 Like, great. 00:37:39.020 --> 00:37:39.740 That helps me a lot. 00:37:39.740 --> 00:37:40.760 Yeah. 00:37:40.760 --> 00:37:41.640 Yeah, exactly. 00:37:41.640 --> 00:37:53.360 And maybe, I guess also, if you had your code more factored, right, into little methods and things like that, breaking across even packages and things, maybe it's easier for it to say, well, this is slow. 00:37:53.360 --> 00:37:55.680 Like, okay, well, I clearly see the function that is slow. 00:37:55.680 --> 00:38:01.120 But if you're just playing around and experimenting, like, you maybe don't have it so fine-grained, right? 00:38:01.120 --> 00:38:03.160 So it's even worse than normal, maybe. 00:38:03.160 --> 00:38:15.180 Well, we try to keep our code well-designed for both conceptual level and practical level, where conceptual is like functions and objects are abstracted to follow the math. 00:38:15.180 --> 00:38:20.060 Practical is more of running end-to-end experiments efficiently. 00:38:20.060 --> 00:38:28.900 So conceptual is important because we want well-designed code so we can extend it naturally with future research. 00:38:29.100 --> 00:38:33.580 So that might be where things like computational bottlenecks aren't really obvious. 00:38:33.580 --> 00:38:40.280 But when you try to run these experiments end-to-end, that's when you see, like, oh, well, we need to go back and profile and optimize. 00:38:40.280 --> 00:38:40.720 Yeah. 00:38:40.720 --> 00:38:41.700 Yeah, that makes sense. 00:38:41.700 --> 00:38:47.620 Yeah, so that kind of touches on, like, this tension between research and software priorities. 00:38:47.620 --> 00:38:49.040 Yeah. 00:38:49.040 --> 00:38:50.760 How do you keep those in balance? 00:38:50.760 --> 00:38:52.020 Like, what are the trade-offs there? 00:38:52.260 --> 00:39:04.600 Well, it's interesting because the research rewards, experiment results, and maybe sharing some cool figures and visualizations with the team are more near-term. 00:39:04.600 --> 00:39:12.500 While the software rewards, like benefits of a clean, well-documented code base, those are farther off. 00:39:12.500 --> 00:39:18.360 So it makes it difficult to prioritize good software like refactoring and cleaner abstractions. 00:39:18.360 --> 00:39:25.480 I found it very helpful to try to communicate, overly communicate, actually, the value of high-quality software. 00:39:25.480 --> 00:39:34.760 Like, maintainable code base makes onboarding new engineers much smoother, and they can get up and running and contributing much faster. 00:39:34.760 --> 00:39:39.800 And then also, I try to reward, in a way, examples of good code. 00:39:40.020 --> 00:39:51.520 An example recently was there was a good use of Dunder slots for fast attribute lookup in a PR I was reviewing the other day. 00:39:51.520 --> 00:39:53.760 Yeah, so maybe tell people about Dunder slots. 00:39:53.760 --> 00:39:56.960 Like, it's used in the right time, in the right place. 00:39:56.960 --> 00:40:00.840 It can, like, dramatically change memory and performance, and it's interesting. 00:40:00.840 --> 00:40:03.820 But we haven't talked about it much in the show, so what's Dunder slots? 00:40:03.820 --> 00:40:04.260 Oh, sure. 00:40:04.260 --> 00:40:15.460 So my understanding of it is, basically, when you have a class in Python, and you have attributes for that class, it automatically will create this Dunder dict object. 00:40:15.460 --> 00:40:26.000 And if you don't need that representation for every attribute of your class, it could be a lot of overhead to do lookups in that Dunder dict. 00:40:26.660 --> 00:40:36.880 So sometimes you can use Dunder slots, which avoids the creation of Dunder dict, and you can define your attribute lookups there. 00:40:36.880 --> 00:40:37.620 That sounds right. 00:40:37.620 --> 00:40:48.320 So there's, you can put in a class just anywhere, and just say Dunder slots equals, and you give an array, a list of basically the field names, the variable names. 00:40:48.660 --> 00:40:51.840 And then those are the only variables it can have, right? 00:40:51.840 --> 00:40:56.660 So in normal Python classes, you could dynamically add another field to it, or things like this. 00:40:56.660 --> 00:41:00.580 And it uses that Dunder dict to manage that. 00:41:00.580 --> 00:41:06.780 But every instance of the class has a separate dictionary with separate names of the keys and things like that. 00:41:06.780 --> 00:41:13.080 And so the slots basically means there's none of that, none of that creation, allocation, assignment, all that kind of stuff. 00:41:13.080 --> 00:41:14.180 Yeah, so it's really amazing. 00:41:14.180 --> 00:41:16.940 All right, so somebody did this, you're like, oh, wow, look at this thing. 00:41:16.940 --> 00:41:21.900 Yeah, so it was just like a quick shout out on our Slack channel, just saying, hey, this is awesome. 00:41:21.900 --> 00:41:24.700 Prop to somebody, yeah, that's really cool. 00:41:24.700 --> 00:41:25.460 Yeah, yeah. 00:41:25.460 --> 00:41:29.660 This portion of Talk Python To Me is brought to you by us. 00:41:29.660 --> 00:41:35.580 As many of you know, I have a growing set of courses to help you go from Python beginner to novice to Python expert. 00:41:35.580 --> 00:41:37.600 And there are many more courses in the works. 00:41:37.600 --> 00:41:41.860 So please consider Talk Python training for you and your team's training needs. 00:41:42.080 --> 00:41:48.280 If you're just getting started, I've built a course to teach you Python the way professional developers learn by building applications. 00:41:48.280 --> 00:41:53.360 Check out my Python jumpstart by building 10 apps at talkpython.fm/course. 00:41:53.360 --> 00:41:56.480 Are you looking to start adding services to your app? 00:41:56.480 --> 00:41:59.660 Try my brand new consuming HTTP services in Python. 00:42:00.040 --> 00:42:05.100 You'll learn to work with RESTful HTTP services as well as SOAP, JSON, and XML data formats. 00:42:05.100 --> 00:42:06.980 Do you want to launch an online business? 00:42:06.980 --> 00:42:11.140 Well, Matt Makai and I built an entrepreneur's playbook with Python for Entrepreneurs. 00:42:11.140 --> 00:42:16.380 This 16-hour course will teach you everything you need to launch your web-based business with Python. 00:42:16.380 --> 00:42:19.920 And finally, there's a couple of new course announcements coming really soon. 00:42:20.120 --> 00:42:25.380 So if you don't already have an account, be sure to create one at training.talkpython.fm to get notified. 00:42:25.380 --> 00:42:29.160 And for all of you who have bought my courses, thank you so much. 00:42:29.160 --> 00:42:31.120 It really, really helps support the show. 00:42:31.120 --> 00:42:37.020 So I looked at the Vicarious website to sort of check out what you guys are up to. 00:42:37.020 --> 00:42:38.140 And I looked at the team page. 00:42:38.140 --> 00:42:39.720 And there are a lot of people that work there. 00:42:39.720 --> 00:42:41.460 Way more than I expected, actually. 00:42:41.460 --> 00:42:43.140 How many people do you work with? 00:42:43.620 --> 00:42:46.100 So we're at about 50 now. 00:42:46.100 --> 00:42:51.180 And I started eight months ago, and I was number 30 or so. 00:42:51.180 --> 00:42:52.780 We're growing a lot. 00:42:52.780 --> 00:42:57.920 And we're looking to keep adding more engineers and researchers. 00:42:57.920 --> 00:43:04.780 And we're expanding also to a big new office building and everything out here in San Francisco. 00:43:04.780 --> 00:43:07.380 And there's a lot of us here. 00:43:07.380 --> 00:43:14.000 And the really interesting thing is that there's a lot of us from very diverse technical fields. 00:43:14.000 --> 00:43:25.500 So we have some people that are theoretical neuroscientists, some people that specialize in robotic grasping and manipulation and everything in between. 00:43:25.500 --> 00:43:27.600 Sounds like a pretty cool space to be in. 00:43:27.600 --> 00:43:28.120 Yeah, yeah. 00:43:28.120 --> 00:43:28.540 It's fun. 00:43:28.540 --> 00:43:28.840 Nice. 00:43:28.840 --> 00:43:41.140 So you guys are working on this general artificial intelligence, which is maybe the most controversial kind, but also has the greatest possibility for helping society out, right? 00:43:41.140 --> 00:43:44.420 Like, it's one thing to make a car that can drive itself. 00:43:44.420 --> 00:43:49.640 It's another to make like a robot that can do all these different jobs, right? 00:43:49.640 --> 00:43:51.140 And learn and so on. 00:43:51.980 --> 00:43:59.660 So, you know, Elon Musk came out with his statement that we're living in a multiverse and he's scared of AI and things like that. 00:43:59.660 --> 00:44:02.800 Like, how do you guys, do you guys talk about this kind of stuff? 00:44:02.800 --> 00:44:05.180 Do you put any credence in it? 00:44:05.180 --> 00:44:06.960 I'm not so sure about the multiverse. 00:44:06.960 --> 00:44:08.940 I'm pretty skeptical of this. 00:44:08.940 --> 00:44:15.260 But the thing becoming too smart, is this something that you guys actually are seriously concerned about? 00:44:15.260 --> 00:44:17.520 It's something that we seriously consider. 00:44:17.520 --> 00:44:19.920 I wouldn't necessarily say concerned. 00:44:21.320 --> 00:44:22.660 It's funny. 00:44:22.660 --> 00:44:28.600 Whenever I meet random people and say what I do, I end up having this conversation a lot. 00:44:28.600 --> 00:44:30.620 I bet you're like, oh my gosh, here we go again. 00:44:30.620 --> 00:44:31.020 Yeah. 00:44:31.020 --> 00:44:32.360 No, we're not living in a multiverse. 00:44:32.360 --> 00:44:35.520 This is just like, damn off the... 00:44:35.520 --> 00:44:45.220 Well, AI will do amazing things for society, like transforming healthcare from reactive medicine to something that's predictive. 00:44:45.220 --> 00:44:49.280 And essentially eliminating motor vehicle accidents. 00:44:50.660 --> 00:44:56.620 I dream of the day I can ride my bike around San Francisco and not worry about someone slamming into me. 00:44:56.620 --> 00:44:56.900 Yeah. 00:44:56.900 --> 00:44:57.740 Yeah, absolutely. 00:44:57.740 --> 00:45:06.540 But on the spectrum of fear and optimism, where the former are guys like Nick Bostrom, who wrote that super intelligence book. 00:45:06.540 --> 00:45:07.180 Yeah. 00:45:07.180 --> 00:45:13.660 Elon Musk and the latter, where the optimists are those of like the futurists, Ray Kurzweil. 00:45:13.660 --> 00:45:20.920 I would say most AI researchers, and myself included, fall somewhere in the middle, leaning towards optimism. 00:45:22.180 --> 00:45:32.080 So I would say, as with previous technological revolutions, there's definitely concern over the big societal changes, and rightfully so. 00:45:32.080 --> 00:45:35.660 Jobs will be lost to automation and industries transformed. 00:45:36.140 --> 00:45:45.480 But it's much easier to look at the existing jobs as examples, painting the picture of, oh, wow, AI is going to replace all of these, we're going to lose all of these jobs. 00:45:45.940 --> 00:45:53.020 Than it is to imagine a future where there are new industries that are yet to be created and all the jobs to find. 00:45:53.020 --> 00:46:00.500 Every change in human history has not wiped it out and people just went and sat in the corner, right? 00:46:00.580 --> 00:46:07.380 Like the industrial revolution, the move away from farming, the move to knowledge workers, like all of these things have been done. 00:46:07.380 --> 00:46:08.640 Exactly, exactly. 00:46:08.640 --> 00:46:10.020 We will go on. 00:46:10.020 --> 00:46:20.720 But I would say the real concern is the speed with which this transformation will take place is significantly faster than the agricultural and industrial revolutions. 00:46:20.720 --> 00:46:24.940 Fortunately, companies are taking this into real consideration. 00:46:24.940 --> 00:46:27.320 Elon Musk's OpenAI is one. 00:46:27.320 --> 00:46:31.000 Here at Vicarious, we have a group dedicated to AI ethics. 00:46:31.000 --> 00:46:31.560 Oh, wow. 00:46:31.560 --> 00:46:32.740 Okay, that's pretty interesting. 00:46:32.740 --> 00:46:36.140 Yeah, I definitely think the change will go quicker. 00:46:36.140 --> 00:46:43.340 You know, like the change for the industrial revolution, things like that, they banned people's working careers. 00:46:43.340 --> 00:46:52.880 So it's not like next year the thing that you did was completely gone and it was totally fine the year before, right? 00:46:52.880 --> 00:46:59.660 And so I think there's going to, I think you're right, there's probably going to be some kind of step where there's like a serious shakeup. 00:47:00.220 --> 00:47:03.640 One example of this that comes to mind is the U.S. 00:47:03.640 --> 00:47:12.180 The number one job title, job role for men is driving of some sort, like trucks and things. 00:47:12.180 --> 00:47:20.940 You know, self-driving cars, self-driving trucks and semis and deliveries could put a serious dent in, you know, guys' jobs. 00:47:21.300 --> 00:47:35.340 And if that happened too quickly, I can see like some serious social unrest because if you have a large young group of men who are not employed and have no positive outlook, that could really turn badly, right? 00:47:35.480 --> 00:47:36.100 Yeah, it could. 00:47:36.100 --> 00:47:59.600 And you see some early examples of this with companies like Uber and Lyft who aren't necessarily replacing drivers, but they're transforming an industry in a way where taxi drivers are the source of this unrest because now they've opened up the workforce to this kind of car sharing or ride sharing. 00:47:59.600 --> 00:48:04.620 Yeah, I would say it's similar to what the Internet did for knowledge work, right? 00:48:04.620 --> 00:48:10.300 Like it used to be you would go to your office and you would compete with the people in your city for those jobs. 00:48:10.300 --> 00:48:14.480 And then all of a sudden the Internet came along and open sourcing and globalization and everything. 00:48:14.480 --> 00:48:17.860 You were all of a sudden competing with everybody, not just the people in your town. 00:48:17.860 --> 00:48:20.640 And it's a little like that. 00:48:20.640 --> 00:48:24.680 Like how many people are going to actually go and become taxi drivers versus like, hey, I have a car. 00:48:24.680 --> 00:48:25.600 I have a smartphone. 00:48:25.600 --> 00:48:27.280 I got two hours and need some money. 00:48:27.280 --> 00:48:28.320 Let's just go do it, right? 00:48:28.320 --> 00:48:33.040 It really opened up the space of potential people in the marketplace. 00:48:33.040 --> 00:48:33.400 Yeah. 00:48:33.400 --> 00:48:41.740 I agree that there's we'll probably get through it pretty smoothly and there's there's very likely going to be something amazing on the other side. 00:48:41.740 --> 00:48:48.480 Let's think maybe one more question on this on this theme for you because I know you I know you have these questions a lot. 00:48:48.480 --> 00:48:56.120 But what do you think the biggest change that AI is going to bring that people will like obviously notice? 00:48:56.120 --> 00:48:56.580 Right. 00:48:56.580 --> 00:48:59.760 Not something super subtle that makes a change that they don't necessarily see. 00:48:59.760 --> 00:49:01.080 But like what are people going to go? 00:49:01.080 --> 00:49:01.420 Wow. 00:49:01.420 --> 00:49:04.460 Like here's what AI has brought us in 10 years or 20 years. 00:49:04.720 --> 00:49:16.660 Well, it's kind of like the mountain climbing dilemma that we were talking about earlier where new AI technologies roll out, but it's very incremental in a way. 00:49:16.660 --> 00:49:24.320 And you don't necessarily notice that Facebook's algorithm for identifying whose faces are whose has improved a lot. 00:49:24.320 --> 00:49:25.340 But it did. 00:49:25.340 --> 00:49:29.120 And that was the outcome of some really impressive AI research. 00:49:29.560 --> 00:49:40.320 But something that I think I hope people will look at and just be completely floored by is how AI is going to transform health care and medicine. 00:49:40.320 --> 00:49:44.180 Yeah, that's one of the things I definitely think as well is diagnosis basically. 00:49:44.180 --> 00:49:44.840 Yeah. 00:49:44.840 --> 00:49:45.100 Yeah. 00:49:45.100 --> 00:49:48.840 Like I had mentioned earlier, health care has always been very reactive. 00:49:48.840 --> 00:49:54.960 Someone is sick and has some sort of symptoms and we guess at what is going on. 00:49:55.460 --> 00:50:01.680 And that is all suspect to what kind of quality of health care you can see. 00:50:01.680 --> 00:50:06.700 And fortunately, we live in a country where we have a lot of good health care doctors. 00:50:06.700 --> 00:50:09.500 But in third world countries, this absolutely is not the case. 00:50:09.500 --> 00:50:15.040 AI I see as more less diagnostic, more preventative in a way. 00:50:15.040 --> 00:50:20.140 So you know that you're getting sick months before you actually show any symptoms. 00:50:20.140 --> 00:50:22.600 And the treatment is minor instead of catastrophic. 00:50:22.600 --> 00:50:22.980 Yeah. 00:50:22.980 --> 00:50:23.360 Yeah. 00:50:23.360 --> 00:50:24.000 Interesting. 00:50:24.560 --> 00:50:35.000 So one of the rumors, you know, there's not an iPhone 8 yet, but one of the rumors around the new iPhone coming out is that Apple is working on a dedicated AI chip that goes in the iPhone. 00:50:35.000 --> 00:50:37.020 What do you think of these types of developments? 00:50:37.020 --> 00:50:54.020 This is really interesting because we've been hearing for years a lot about this neuromorphic chip development, which was basically companies like Intel and ARM or whatever doing some research towards, okay, what is the next platform for computing? 00:50:54.020 --> 00:50:58.380 And recently we've seen or heard about Apple's new chip. 00:50:58.380 --> 00:51:03.460 And Google also has their TPU, which is a Tensor Processing Unit. 00:51:03.460 --> 00:51:12.320 And these are devices dedicated to running specifically their algorithms on devices, I guess, on iPhone and iPad. 00:51:12.320 --> 00:51:23.100 And I don't know too much about the hardware of these devices, but they would be optimized for computing inference on deep learning models and such. 00:51:23.240 --> 00:51:23.380 Yeah. 00:51:24.060 --> 00:51:28.880 I think it really opens up the possibility for something interesting, but I have no idea what it's going to be. 00:51:28.880 --> 00:51:30.640 We'll find out in a few years, I guess. 00:51:30.640 --> 00:51:31.560 Yeah. 00:51:31.560 --> 00:51:37.080 The exciting thing is a lot of the work in AI is no one knows what it's going to be. 00:51:37.080 --> 00:51:37.460 Yeah. 00:51:37.460 --> 00:51:38.800 That keeps it fun. 00:51:38.800 --> 00:51:39.760 All right, Alex. 00:51:39.760 --> 00:51:41.600 Maybe we should leave it there for the AI topic. 00:51:41.600 --> 00:51:46.160 I really appreciate you sharing what you guys are up to and your thoughts on AI research. 00:51:46.160 --> 00:51:46.580 Yeah. 00:51:46.660 --> 00:51:47.920 I'm glad you guys are interested. 00:51:47.920 --> 00:51:48.580 Yeah, it's cool. 00:51:48.580 --> 00:51:49.300 All right. 00:51:49.300 --> 00:51:51.340 So now it's time for the final two questions. 00:51:51.340 --> 00:51:57.800 So when you write some of your deep learning Pythonic code, what editor do you open up? 00:51:57.800 --> 00:52:00.160 Oh, I definitely go for Sublime. 00:52:00.160 --> 00:52:07.060 There are some add-ons like GitGutter I like a lot and PyFlakes Linter. 00:52:07.060 --> 00:52:09.940 And then for C++, I'll use CLion. 00:52:09.940 --> 00:52:10.240 Yeah. 00:52:10.240 --> 00:52:10.800 Okay, cool. 00:52:10.800 --> 00:52:12.800 Yeah, definitely the plugins for Sublime are cool. 00:52:12.800 --> 00:52:16.980 And I played a little bit with CLion, but I just don't have enough C++ code to write. 00:52:16.980 --> 00:52:19.660 So it looks neat, but I haven't done anything with it. 00:52:19.660 --> 00:52:22.480 And notable PyPI package. 00:52:22.480 --> 00:52:23.720 Oh, there's so many. 00:52:23.720 --> 00:52:29.680 The one I mentioned earlier, KernProf Line Profiler, has saved my neck many times. 00:52:29.680 --> 00:52:34.440 And then also, I like working from the command line a lot. 00:52:34.440 --> 00:52:40.220 A lot of people in research will use Jupyter Notebooks for the interactive visualizations. 00:52:40.220 --> 00:52:43.820 But working from the command line, I like PTPython. 00:52:43.820 --> 00:52:44.340 Oh, yeah. 00:52:44.340 --> 00:52:45.240 I love PTPython. 00:52:45.240 --> 00:52:46.100 That's great. 00:52:46.100 --> 00:52:47.240 Yeah, I use that as well. 00:52:47.240 --> 00:52:49.580 Yeah, the tab complete is so clutch. 00:52:49.580 --> 00:52:55.480 But then also, arg parse, I love a lot because I can just throw an experiment in a script and 00:52:55.480 --> 00:52:58.260 then have some parameters that I define at the command line. 00:52:58.260 --> 00:52:58.480 Yeah. 00:52:58.480 --> 00:52:59.100 All right. 00:52:59.100 --> 00:52:59.340 Awesome. 00:52:59.340 --> 00:53:00.640 Those are three good recommendations. 00:53:00.640 --> 00:53:02.360 So final call to action? 00:53:02.360 --> 00:53:06.940 Well, if you're trying to get into AI research, like we were discussing earlier, you can definitely 00:53:06.940 --> 00:53:12.980 check out some of these toolkits that have prepackaged models like Keras and TensorFlow and PyTorch. 00:53:12.980 --> 00:53:18.640 But you should try to implement, for example, a convolutional neural net in numpy code. 00:53:18.640 --> 00:53:20.420 That's how you really learn what's going on. 00:53:20.420 --> 00:53:20.880 All right. 00:53:20.880 --> 00:53:23.400 So let's start from the ground up and figure it out. 00:53:23.400 --> 00:53:24.400 Then use the tools, huh? 00:53:24.400 --> 00:53:24.860 Yeah. 00:53:24.860 --> 00:53:25.160 Yeah. 00:53:25.160 --> 00:53:25.580 All right. 00:53:25.580 --> 00:53:26.200 Excellent. 00:53:26.200 --> 00:53:29.140 So thanks so much for being on the show and sharing your story with everyone. 00:53:29.140 --> 00:53:29.900 Appreciate it. 00:53:29.900 --> 00:53:30.160 Yeah. 00:53:30.180 --> 00:53:30.920 Thank you for having me. 00:53:30.920 --> 00:53:31.200 You bet. 00:53:31.200 --> 00:53:31.440 Bye. 00:53:31.440 --> 00:53:35.760 This has been another episode of Talk Python To Me. 00:53:35.760 --> 00:53:41.240 Today's guest has been Alex Lavin, and this episode has been brought to you by Linode and 00:53:41.240 --> 00:53:42.180 Talk Python Training. 00:53:42.180 --> 00:53:46.860 Linode is bulletproof hosting for whatever you're building with Python. 00:53:46.860 --> 00:53:51.220 Get your four months free at talkpython.fm/Linode. 00:53:51.220 --> 00:53:53.720 Just use the code Python17. 00:53:53.720 --> 00:53:56.360 Are you or a colleague trying to learn Python? 00:53:56.840 --> 00:54:01.040 Have you tried books and videos that just left you bored by covering topics point by point? 00:54:01.040 --> 00:54:07.040 Well, check out my online course, Python Jumpstart, by building 10 apps at talkpython.fm/course 00:54:07.040 --> 00:54:09.620 to experience a more engaging way to learn Python. 00:54:09.620 --> 00:54:14.440 And if you're looking for something a little more advanced, try my Write Pythonic Code course 00:54:14.440 --> 00:54:16.980 at talkpython.fm/pythonic. 00:54:16.980 --> 00:54:19.700 Be sure to subscribe to the show. 00:54:19.700 --> 00:54:21.920 Open your favorite podcatcher and search for Python. 00:54:21.920 --> 00:54:23.160 We should be right at the top. 00:54:23.160 --> 00:54:26.480 You can also find the iTunes feed at /itunes. 00:54:26.480 --> 00:54:32.440 Google Play feed at /play and direct RSS feed at /rss on talkpython.fm. 00:54:32.440 --> 00:54:37.540 Our theme music is Developers, Developers, Developers by Corey Smith, who goes by Smix. 00:54:37.540 --> 00:54:42.240 Corey just recently started selling his tracks on iTunes, so I recommend you check it out at 00:54:42.240 --> 00:54:44.240 talkpython.fm/music. 00:54:44.240 --> 00:54:48.680 You can browse his tracks he has for sale on iTunes and listen to the full length version 00:54:48.680 --> 00:54:49.600 of the theme song. 00:54:49.600 --> 00:54:51.680 This is your host, Michael Kennedy. 00:54:51.680 --> 00:54:52.960 Thanks so much for listening. 00:54:52.960 --> 00:54:54.160 I really appreciate it. 00:54:54.160 --> 00:54:56.280 Smix, let's get out of here. 00:54:56.280 --> 00:55:26.260 Stay tuned.