#295: GIS + Python Transcript

00:00 Geography is the study of places and relationships between people in their environment. Often we think of maps but maps are static GIS gets interesting when you realize that we're studying and visualizing data flowing through these locations and communities. In this episode, you'll meet Silas Tom's. He's an author of several Python GIS books, and the co host of The Memphis hour podcast. Are you ready to dive into GIS with Python? This is talk Python to me, Episode 295, recorded October 27 2020.

00:42 Welcome to talk Python to me, a weekly podcast on Python, the language, the libraries, the ecosystem, and the personalities. This is your host, Michael Kennedy. Follow me on Twitter, where I'm at m Kennedy, and keep up with the show and listen to past episodes at talk python.fm and follow the show on Twitter via at talk Python. This episode is brought to you by linode and Talk Python Training. Please check out the offers during their segments. It really helps support the show

01:07 and talk Python, we run a bunch of web apps and web API's. These power the training courses, as well as the mobile apps on iOS and Android. If I had to build these from scratch again, today, there's no doubt which framework I would use. It's fast API. To me fast API is the embodiment of modern Python and modern API's. You have beautiful usage of type annotations, you have model binding and validation with pedantic and you have first class async and await support. If you're building or rebuilding a web app, you owe it to yourself to check out our newest course, modern API's with fast API over at Talk Python Training. This is the first course in a series we're building on fast API and for just $39. It'll take you from interested to production with fast API. Learn more and get started today. Just visit talk python.fm slash fast API, or click the link in your podcast player show notes. Silas Welcome to talk Python to me. Wow. Thank you so much for having me, Michael. Yeah, it's great to have you since is a GIS episode. Where are you calling in from? Where are you located? I'm in San Francisco, where I've lived for the last 13 years now since I moved down for grad school. Oh, right on. I grew up in Northern California, kind of in the middle of nowhere, but I always really enjoyed this city. And I've come to really love it. Yeah, it's a new place. Fewer to jump on I five, the north south major highway. People don't know and go north for a very long time. The next big city you hit you get to Portland, where I am so right on. You know the Ashton's a big city.

02:38 You know Ashlyn is nice to visit. Yeah, but if you could throw a rock across it, it's not a big city. There you go. Yeah, you can only have so much Shakespeare in your life. Yeah, yeah, that's right. That's right. We're gonna talk about GIS and Python, and all those sorts of things, but in a bunch of cool job experiences that you've had, as well as some of the libraries and applications and whatnot. But let's start with the beginning. How did you get into programming in Python, it was kind of a circuitous route. I mean, my parents of all things were organic farmers up in the middle of nowhere, when I was born in northern Humboldt County, and my dad had always kind of had an interest in technology. He wasn't averse, or anything. So when I was about eight, he got a little computer that was programmable, that had a basic program on it. And he used that to like, keep track of sales that are fruit stand around. And he taught me how to, yeah, it was cute. And probably 800 lines or something, which seemed like amazing to me, because he would, he taught me how to do a little bit. And I would do little silly things when I was seven years old, or whatever, you know, what's interesting about code is like, it's not always about how big or how complex it is. It's sometimes it's about what it empowers, or the problem that it solves, like, I can think of a couple of libraries are like, gosh, this is so nice. This exists. And it's like 100 lines of code. Yeah, but I just love that it exists, you know, completely, like we'll get into it. But like pie shape. It's an implementation reading shapefiles for geospatial data to format. And it's relatively simple, compact, and it's beautiful. I love it. Yeah, for sure. Okay, so you had this experience with this basic app that was right, in the family business? Yeah. Well, there was that. And then in high school around 97, I took an elective in computer programming, also basic, which I didn't like love the other crowd in the room. Really. I was always thinking of myself as the more active I played sports, but I didn't like it. I just couldn't fully identify with it. And being from like a rural kind of macho County. Like it just didn't occur to me that that was something I could do. So if you could see the future, like, yeah, yeah. Now all the cool stuff you can build and like, sort of tech is changing perception. Who knows, maybe, maybe if I could just read the newspaper because at that time, it was like the first calm wave and I just wasn't really in the mindset that that was something I would do with my career. But I went to college, I went to Santa Cruz, and then I was kind of homesick. I finished up back at Humboldt State University and got really into geography.

05:00 V, which is a really interesting quasi science, I guess you'd say that has a lot of really interesting theories about why the world is how it's divided up and why it's divided up the way it is how people sort of present themselves across space, how cultures interact with the physical sciences, and somebody who like spent my whole childhood like reading through encyclopedias, it was just the best combination of learning and interest that I could ever find. So without even realizing it, it, I had never heard of geography before I really got into college. It was amazing. It was perfect for me. And it had this component that was, I think, a bit too lightly touched on called GIS geospatial information systems that I got exposed to. And I thought it was kind of cool. But I wasn't again, expecting to make computers a part of my professional career. At that time, I was really focusing on environmental policy and water policy, especially mo county has a lot more rain than the rest of California. And they've built these giant dam systems to bring it down to the Central Valley. And it seemed like it was there was a huge fish die offs in 2001 2002, that were directly attributed to the lack of water. And so I got kind of interested in like, how did they make these decisions to build these dams? Why does this water policy exist? How did they benefit and value agriculture, like sort of profit or cotton that wasn't really making a profit in the Central Valley versus salmon, which could be very valuable where I was from, so I got really deep into those weeds and ended up going into grad school in San Francisco State for geography with a focus on resource management, that was 2007. Nice. And did you end up with like, a lot of data to process and I got, I guess, I'm gonna have to do these computers. Again, I just remember, like late nights, trying to figure out like, just really struggling with the software and doing some interesting. Also, another interest of mine was wine. And I remember trying to do like a temperature map of Humboldt County because a lot of the organic farming families that I've grown up with have started growing wine at the time. And I wanted to see like, how does it compare temperature wise to Napa Valley, or to other well known places, and I just hated that manual data entry that I was doing. And I thought there's got to be a better way. So those feelings and a couple internships that I got when I got to grad school really led me to value programming. And yeah, and I felt so stupid with it at first, because even though I'd had that experience as a child, it had been a decade since I'd really touched keyboard that way. And I felt like there's got to be something to this. So I'll just keep hitting my head against the wall. Even though I feel kind of dumb. I know that I can use this and it really worked out. Yeah, that's super cool. I think it's really interesting your story about how you kind of, like slowly just dragged you and you're like, fine, fine, you've got, I kind of went down the same path as you in some regards. And that I didn't really see myself as a developer, I kept looking to do other things chemistry and math, and so on. And yeah, and eventually, I just like, you know, what, actually, I do like this better. But it's easy to forget those early days of like fighting with syntax. Just Yeah, I remember being so excited about getting a C++ program to compile.

08:18 It's going to have to run and work correctly. That's the hard part, not the compiling. But these early days. Yeah, you step over these challenges. And then they're kind of in the past, but everyone goes through them. Well, I remember when I was really trying to learn in 2008, I took a class that was sort of an elective offered through the computer science department. And I got to be really good friends with the guy who was just a master's in engineering student himself. And he really saw that I cared even though I didn't know anything, basically. And so he spent some time and taught me a bit of Django at that time was was just coming out. And I remember getting so excited when I was able to have Django, use arc pi, talk to a shapefile get a location out and put it on a browser, just like a little tiny thing. Yeah, I showed my grandmother, I'm like, Grandma, I did it.

09:04 She had no idea what I was talking about. I bet she was really happy to see it must have been psyched. She She didn't discourage me, she was so excited. But I just remember that moment of just like I did it and the wizardry of that I still don't think I'm over that really, dad. Now. It's I never I haven't gotten over it either. It's amazing. And just that feeling of I built something that could be useful to people and it's out there and they can play with it and, and then to see them come and use it. It's just amazing.

09:30 I started just that same process of like, whatever I could do at work is at this point, I had gotten a job as a GIS technician, I was making maps of the park district lands that they were considering adding, and I found it a great to be paid, especially during the recession there. But I was not like loving the work itself. So I found all kinds of ways to use Python to automate to figure out how to make like a database of all the other maps we had made to just anything that I could think of that was interesting to me. Never gonna end

10:00 Have a map a second time. Yeah, exactly. Stupid map. So and then somewhere in there, I got a call from a recruiter, I had some friends of friends who were in the program who knew that I did some Python. And they recommended me for a job at an environmental consulting firm called ICF. And the recruiter, I remember him saying, like, do you really want to work for the government? You know, you're like, 27, right?

10:24 Yeah, I suppose you're right, I was so happy to just have a steady job at that point that I wasn't actively looking. But there was some poll to just something different and exciting. And I'd be really benefited because it brought me into the opportunity to work on big environmental projects like the high speed rail project, and figure out how to use Python to automate this because we were right, we were getting engineering outlines in AutoCAD, and we had to convert that into GIS formats, and then run that against all of the data we had on species range, or any kind of protected resource, even like farmland or wetlands or anything like that. And we had to do it week after week after week. And I was kind of I just remember that first call when I first got hired where I was like, I think I can automate this. Yeah, with that upward tilt in my voice, but, but I could, and I did, and that's awesome. I kind of became a team, they probably saw you, like you said, as a wizard. They're like silos can make it do it automatically. Do you understand what that means? on its own? Right, like not other people. Yeah. And some of the libraries we're going to talk about in GIS are like, hey, I've got this lake, and I've got this other thing. Do they intersect? Or does this line pass through this area? And really interesting stuff that builds on common libraries that people will know like sequel? Well, exactly. You know, the one big one that really got me at that time, because I was running into like licensing issues with the Esri ArcGIS software that we were running couldn't really do some of the higher level analyses I needed to do. So I heard about this thing called post GIS, which was built on top of what is it PostgreSQL. And then a library called psycho PG two that allowed me to write spatial sequel and directly talk to a database and do those analyses in code like I had never been able to do before. And it kind of blew my mind. This is around 2010 11 or something. I just, that's when I realized like beyond the world of the software that they'd given me this as a sophomore, there was this incredible open source software community that was providing tools that are exactly the ones I needed, and I can tweak them and do so much more and so much faster. Yeah, so interesting. You don't have to ask for licensing or worry about that, or Yeah, or whatever.

12:39 This portion of talk by me is sponsored by linode. Simplifying infrastructure and cut your cloud bills in half with linode. Linux virtual machines, develop, deploy and scale your modern applications faster and easier. Whether you're developing a personal project or managing large workloads, you deserve simple, affordable and accessible cloud computing solutions. As listeners have taught Python to me, you'll get a $100 free credit, you can find all the details at talk python.fm slash linode. linode has data centers around the world with the same simple and consistent pricing. regardless of location, just choose the data center that's nearest to your users, you also receive 20 473 65 human support with no tears or handoffs, regardless of your plan size. You can choose shared and dedicated compute instances. Or you can use your $100 in credit on s3, compatible object storage, managed Kubernetes, clusters, and more. If it runs on Linux, it runs on linode. Visit talk python.fm slash linode. Or click the link in your show notes and click that create free account button to get started.

13:44 I before we get off this sort of intro topic, though, real quick, what are you up to these days? You got a pretty interesting job working with A intersection A lot of these concepts? Well, yeah, so you know, that sort of like do gooder mentality has never really gone away. And I've worked on some big enterprise systems for a whole city, let's say, which included creating a GIS for a Superbowl for Levi's Stadium. When it was here, I got to go to the Super Bowl. We did every concert and event for two years before that. So I saw Beyonce, you know, I saw Taylor Swift or all these people, right, just because I built a mapping system that used Python to talk to a 911 database to locate calls that were coming from the stadium. And also we figured out how to put an iPhone app in the pocket of the police so that we can see where they were in that we could keep 70,000 drunk people from beating themselves up.

14:36 So there's that. But then like after that, I felt like I really wanted to just kind of go even more into front end development and learn more JavaScript. So I ended up joining a team at Neo which is they've like shut down their us operations, but it's a Chinese electric car company that's working on automatic cars and self driving cars. And that sparked a lot and that team that I worked with

15:00 did leave and go to Zillow, which I'll get to in a bit. But that team that I worked with I loved and that team, one of the main guys, Josh Butler founded this company, kinetic guy, which is, I think the biggest realization from NEA from all the work we did all these smart people we worked with is that self driving cars is so hard. And so using computer vision, using machine learning, without like, a really directed mission is going to be an incredibly hard thing to master for the next decade or so even if you spend $10 billion on it. Yeah, I honestly can't believe we have self driving cars. It's Yeah, I always thought of the AI realm being like, Oh, yeah, we're gonna do this Turing test. And you're gonna have the app chat with somebody over text. Yeah. And then if it seemed real enough, your AI is like real. And somehow that always felt so shallow and fake, and then it was somehow kind of stagnated there. And then all of a sudden, cars are driving themselves around at a level of complexity so far beyond that, they don't even compare right. And somehow we couldn't quite get one. And we got.

16:01 I mean, yeah. And Tesla just rolled out their beta, right, last week or so. Yeah, exactly. It's advancing all the time, I still felt like it was a huge problem. And level four, level five self driving cars, which we were working towards was going to be a decade off. So this team that split off, we're focused on computer vision in small spaces in warehouses to see like, how safe are people being? How much risk are people causing with their driving habits, or with their work habits? And how can we give business owners the ability to understand how safe the practices are at their work, how much potential risks they have for like a worker's comp or an injury. And the dunamis loves it, because like, injuries just ripple out throughout a community. If somebody is hurt, then they're out of work for a while, then their family is hurting, their community is hurting. And we really want to, like have this use of computer vision that identifies intelligently risk and safety. And this was a cool mission, and also great technology. Yeah, so if I had a factory and maybe after lunch, people will come back and they weren't putting on their hard hats. Yeah, or something like that. You would say like, missing some hard hats. Yeah, yeah. Yeah. hard hats, vests, masks, social distancing was one we built right away, obviously, when COVID hit Oh, yeah, of course. Yeah. And then forklift, speeding, forklift, near misses is really cool, because we're able to plot it on a map, you know, like, brings it back to the map to all those insane, like YouTube videos of like, people crashing, stupid stuff on like, you know, work just after hours, like you would know about that kind of stuff. Yes, exactly. You know, we're working towards like a real time notification. I think that that's in our future. But more than anything, it's the all the time risk environment that we want to let people know about, like, How well are you doing? And how safe are your people? Sure. That sounds like a great mission. Let's talk broadly about GIS. Mm hmm. For the moment, just some of the areas which been used. I mean, we talked a little bit about this with some of the environmental things you talked about. I mean, I guess one of the things that most starkly stands out for me, maybe two is one is directions. Like another thing that I'm just blown away by. Yeah. And also probably like, a little bit weakened by just as a human is GPS, the ability to ask Google Maps, or something else how to get me there. And just I'm thinking of the feeling I have when I, for some reason, forgot my phone, or the battery dies. I'm just like, Oh, I'm done. Yeah, what kind of? I have no GPS, right? Whereas like, I grew up without even the cell phone. So it's just such a weird thing. But absolutely GPS. Right? Well, 20 years later, it's just become part of our lives, like, I remember in like 2001, let's say trying to drive through San Francisco and being so freaked out driving probably the longest way I could, because it was the only route I knew to get safely from one side to the other, and just couldn't imagine you know how easy it is today. I went and visited my uncle who had gone to the Naval Academy, he was living in Annapolis in like 2005. And I had some friends in DC. And I said, I'd like to go visit them. How do I get down there? And he said, Oh, well, there's GPS in my car. So you can just take my car, and it will just tell you how to get there. And I couldn't believe it was the easiest trip I'd ever had door to door in an area I'd never visited in my life. It blew my mind. Yeah, it's so amazing. I remember doing some work in Shanghai and had one of the early iPhones and I was using Google Maps I had gotten into the habit of I'd sort of pick a random orientation go I'm gonna go that way for about an hour of walking and exploring. I get tired, I'm gonna just hit take me back to my hotel. And I had gotten really far one day, and the phone was down to like, 10% I'm like, Oh my god, I forgot the name of my hotel and I can't speak Chinese. I can't speak Mandarin. I'm going to be in really big trouble. So I had to just turn it off and like, please turn back on and get me close. You know, like, that was the most extreme concern of lacking GPS. But yeah, certainly those things are amazing. Another area that I think is really probably interesting, and maybe actually has

20:00 A little more personal touch for you is agriculture, right? Like understanding your crops and growing things. And just, you know, you think of the idyllic farmer with just basic tools, right. But there's a lot of IoT and a lot of smarts you can apply to those things. Now, the marriage of GPS and GIS and farm tech is just all over, especially, let's say, the Midwest where they have, you know, one of the first self driving applications was tractors, right? Because it's so boring to just be an attractor for a straight line for a long, long, long time, and then turn around. And so they figured out how to make the thing turn around using a satellite. And it's pretty impressive. I always thought to that even like down to the small organic farms that I grew up with, you know, where you're incredibly resource specific, you need to know exactly what went where, and keep track of it, you know, for regulations, but also just to know what to grow next, and to keep track of your nutrient levels and your soil that GPS, GIS and mapping would be a really great use. I'm not sure how far it's gotten into there, because a lot of them are kind of old school hippies, who don't, you know, they still use pen and paper as much as possible, but it's maybe if you tell them it's open source, it's not.

21:07 them. I love that. Yeah. So yeah, it's become incredibly important. And you know, like just watering, monitoring water levels, for instance, like, what do they call it? nd VI, which is like the what, how you can see how well your crops are watered or thriving based on like, the infrared that you can't see from the satellite imagery we can get now it's just incredible revelation, you know, and seeing I can imagine, yeah, but then, you know, I kind of pulled away from that when I came down to the city and got more into like environmental services, environmental industry. And the two biggest project I worked on was the high speed rail, as I mentioned, and then also the pg&e has power lines everywhere as that's caused some issues, because there's branches that have rubbed up on it, and fires have started. They have programs that monitor how much trimming they do around all of their lines throughout all of Northern California. And they keep track of it. And then they also have to kind of estimate within all that work, how many, you know, endangered species, they stepped on accidentally, that kind of thing. And because it's something that is expected work, they have to mitigate. So we would use GIS to calculate basically how much pg&e needed to pay act. Yeah, yeah, that was really cool. The species animal equivalent of a carbon tax or carbon offset or something like that, right? Yeah. Pretty much. Yeah. It's just kind of how it goes. thing is if you if they don't trim those lines, I mean, they're and then those fires are horrible this year. And, you know, it's not like you could say, well, you shouldn't shouldn't trim them right. It's gonna be worse, right? It's like a necessary evil, basically, yeah, you have to do it. I don't want to see I got disillusion. But I realized in some ways that the industry is it's sort of just like insurance it's it's something unexpected business payment that you have to do to cover your the effects of your business, especially for those giant utilities or other large construction projects, which is fine, it maybe dampen my enthusiasm a bit. But really, at that point, I was really getting excited by the technology. Turns out I really liked program.

23:05 And I decided to move on to a company that was focused on city plant like city GIS, like enterprise GIS being able to integrate, you know, the police, with the sewers with everything. Because when we got there to that city, which was Santa Clara, basically build the Smart City Smart City. Yeah, you know, they had like 11 different address databases that they were using, each department had their own and they were all different. So like, imagine the police not knowing when you're at where your addresses? Well, it's, it's more common than you would think, unfortunately, right. And Smart Cities really appealed to me. So I spent a couple years doing that. And that was was amazing work. But I just again, really like the programming so much. I started moving away, probably from pure GIS into a lot of like front end stuff. But that kind of gave me an opportunity to get to know web mapping, and JavaScript and all the amazing JavaScript tools that have come out over the last 10 years to enable mapping on the web. Yeah. So you worked at Zillow, for example, which is pretty awesome. Yeah. Zillow is definitely one of those cutting edge companies. I remember being blown away when Zillow, my first experience, Ilana, I could see what my neighbors paid for their house. Yeah, in the value predicted value of it now and just like all that sort of neighborhood II stuff really interesting. Yeah. Well, when Zillow came out in like, 2005, let's say ezri really dominated the GIS space. And but you start to see these little cracks appear were like mashups as they call them in the late aughts. That's right. That was the web two. Oh, yeah, shove mash. That was all the hype. And in reality, you know, Zillow is just still a mashup. It's a map with a data overlay of parcel information and some machine learning predictive algorithms. I'm assuming that go into the values the the the zestimates that you see. Yeah, that's right. So it was really cool and what we were focusing on or at least what I was

25:00 brought into focus on as the geospatial lead there was mapping every single neighborhood in America and Canada to understand like how people defined the space around them. Because again, for me, in some ways, the geography was more important than the code itself, per se. It was, yeah, yeah. And it got really cool. But that project was cancelled to focus on other big projects. So I got into data pipelines. And I did actually a lot of spark work, some pi Spark, but also using Scala, I did a little bit more Scala. So I cheated on Python there. That's for a different podcast.

25:36 But yeah, like, it was amazing work. And it, it is still, though, in some ways, like web 2.0. And I kinda wanted to really be a part of the future, I really be a part of whatever this wave is, is this to 3.0 4.0 10? point, I don't, I don't know anymore for now. But I think that that's having worked at Neo right before that, and seeing what was going on there with the self driving and the computer vision, which is fascinating stuff. And just my experience with computer vision actually comes from GIS because classification of data of satellite data is a big part of GIS. There's this division between like vector and raster data and the raster data coming out of satellites, you got to look at the pictures out of the satellites and make some sense of it, or is that a river is that a lake is that a car is incredible, just to be able to make these colors, these cells next to each other and make sense of it. It's incredible, it really is, it still blows my mind in some way. So it was a natural move to go into, you know, computer vision. And what I really realized recently is that it's not that different. We have this video data that we're overlaying all kinds of information about what you're seeing, and you're interacting with it. It's a different kind of map is what it is. Yeah. Yeah. You know, I wanted to just dive in that for a second. Because I think it's so interesting how much these skills that we develop, and these things you feel are so specific to some particular area, right, like GPS, or something like that, my experience from going around and doing, I used to in person training classes, I remember, I was one week working at Edwards Air Force Base. And then I was working at like a hedge fund in New York the next week. And I'm just like, these cannot be more different groups. And what I realized was I was teaching them basically the same thing, and they both needed it exactly. They both needed a lot, right. And it's like, there's this sort of core of what you need to be super good at all these ideas. And I suspect in the GIS realm, it's kind of similar, like the computer vision, the ML, the data, database, you know, geospatial queries, and all those kind of things. Like, everyone just needs that. And then it's like, that little 20% difference that goes on top, it seems like it's the whole difference, but it's just a little bit, right. Totally. Yeah, I guess we both chose wisely and that Python became our focus, because it's, it incorporates all of those things. And very nicely, by the way, I think. And I really do think that yeah, it's it's not that different. And it is so cool to be able to just get this basic of programming, the basics of programming, and, and whatever you're passionate about. And you can just take that across fields and feel really successful and useful. Yeah, yeah, it's super cool. I totally, I've seen it happen over and over. And I just want to highlight it cuz I know there's a lot of people out there listening, I'm knowing that would be really useful, right? Like they, they probably feel like they're struggling to just make it in one area. But really, that's actually that's the whole that's everything. Almost all of it right? It's cool. Oh, somebody said one time that was he like, it's amazing being a developer, but it's even cooler in some ways being like, let's say a chemistry person who knows how to program, you know, or for me, like environmental water policy. Yeah, the set of program, all of a sudden, you're so valuable in that office, they can't believe they live without you. And it's not that different from other types of programming. Really, it's just data that you're manipulating a lot of ETL. Right. That's, that's kind of what we do. And that's awesome. It's so fun. I agree. All right. So let's actually, next spend a little time talking about some of the libraries and whatnot that might be worth thinking about. They're out there, the types of problems we can solve with them. So you spoke about shapely already. Maybe give us a quick rundown on shapely again. Yeah, there's a pro team, I suppose. But headed by this one programmer. I forget his name right now, because he's Shawn. But anyway, shapely and austereo are pure Python tools for manipulating geospatial data. shapely is more about once you've read in the data into your code somehow, then you can perform distance analysis or you can say, you know, is this car in this neighborhood? How far has this car traveled? How close was this car to this particular house? All those questions that you might need to ask, or you might, you know, calculate demographics across space. Anything that you need to do with polygons, points and lines, which is the majority of vector data, you're able to do with shapely and it's very useful in that sense. It has a really nice

30:00 interop between JSON as well. Yes, super nice, right? Yeah, you can store your data in JSON like it just naturally reads and writes that. Yeah, exactly. There's a format called well known text as well that it reads. So it's it's very smart in that sense. And raw stereo or roster IO, I always get that confused is the raster raster raster version of that, I always say roster needs arrestor.

30:21 But the other one I really like that you have here is, is GL alchemy, two, which is an add on to SQL alchemy, which is amazing just in itself, like SQL omake. So cool to be able to write code for any particular or, you know, General database, and then be able to have the code work is incredible. I really like ORM, I love RM two. And I think that they're, they just make data so much more accessible, and remove some of the error potential error that you might have in working with it, right? Like if I'm going to insert something, don't forget to add this other part of it, right, this piece of data, or you might, you know, you load it up, you might forget to convert some value from a string to an Andrew, who knows, but right like that you build these classes and adjust, you do a few simple things. And you've got a really legit app working with the database. Yeah. And so that's SQL alchemy, obviously, right, and Django has its own version. But if you're not doing Django, basically, you're probably using SQL alchemy, which has been going strong. So geo alchemy, what's the story of that it is specific to PostgreSQL with the post GIS spatial engine add on. But it allows you to, as you're saying, just write a database model that includes geospatial columns. So the columns that store these points, lines and polygons that we're describing. And you're able to do with that module, all of these different buffers or intersections, or any other kind of geospatial analysis that you need to do directly with the data that you're storing in your database. Nice. So if anyone's familiar with SQL alchemy, it's exactly the same like you create a declarative base, you create an engine, you do the create tables, like all that stuff that you do, you create a session and you add stuff to it. So like all those are the same. But you can have interesting stuff. Like you could define a class, which is a lake, for example. And it's got a primary key, which has an ID, it's got a name, which is a string, but then it could have a geometry, which is a polygon, which is one of these things coming out of geo alchemy, right. Yes. Well, GIS Yeah. in general. Yeah. Yeah. Yeah. Yeah, exactly. So you just create, like, it's just a different column type out of that library. Yeah. And yeah, yeah, you're off to the races. There's, there's a weird idea within GIS that like, is spatial special data, are we specialists? Or are we tech people, like, there's always kind of a weird nebulous feeling there. And what it does, in some ways is it makes spatial data not special, it makes it really easy to deal with, it's logical and stored exactly like any other data in a column associated features are right there in that same row. And you can do relationships with it, like just like ID based relationships. But you can also do spatial relationships, you can say, given this neighborhood, how many homes above this age, or this price, let's say, exists in that area, you know, that kind of query. And, and you can just treat it like just any other data, but there's just this couple of special tweaks that allow it to be really flexible. And it's just more database queries, just like pseudo session query have like Lake filter, that is like, excuse me, the geometry contains this point, or intersects this line? And presumably, those are done with, I guess, some form of an index. So that's fast. Yes. I don't really know, right? There's spatial indexing, that improves it quite a bit. In fact, with the right engine underneath your database, these queries go fast. For millions of points. Postgres is pretty, pretty amazing for being a free day. Awesome. Yeah. Yeah. Yeah, it is super cool. You know, we were talking a little bit about it earlier, like Tinder, you know, all these modern apps are geospatial, because they've just integrated you know, where you are, where the object that you're interested in, or like Uber, like, you know, nobody understood what I was saying, when I said GIS, but when I finally was like, Oh, it's like Uber, you know, you see where you are, and you see the car coming to you, which is, you see the roads where it can go? Yeah, maybe even optimize a path if it's like a shared Uber, whatever. Yeah. All of those are juicy share anymore, but they used to,

34:21 at least work together when they had to. Yeah, so yeah, it's built into our modern lives in ways like you were talking about mapping, you know, but it's mapping plus information Overlay and sorting through that information, to find the things that are interesting to you across a varied and unique space that you may not be familiar with a new city. Yeah, yeah. Well, when you were first talking about studying your master's degree, and geography, just thinking back of when I was in middle school, or some is like, well, what you're gonna have to do is name all the states in the US you have to name all the countries in Europe like okay, well, not super interesting, but I guess I'll like I'll commit to them.

35:00 And then we'll just we're going to do this right and then probably forget most of them. And, you know, that's kind of was my, like, young minds impression of that. Yeah. But you know, if you think of like space overlaid with blowing data and all the questions and answers that you you have all the sudden it becomes this really interesting programming problem, and it really important for people's lives. Well, let me have a quick diversion here. So back in the 30s, especially, there was a lot of geographers hired to like make flood maps or other kind of maps, they were trying to figure out, like, Where are disparities across space? And how can we help people and it was a top level science, geography had departments all across like Harvard, Yale, all of those schools. But then, to be honest, there was like a supposedly, like, homosexual scandal at one of these schools where like, you know, a geography professor was living with one of his grad students and president who had skepticism around the science itself kind of decided just to cut it out. And because like, he gave a different reason for it, but that was the rumor behind it. Oh, oh, my god. Yeah. So it was Harvard or Yale, they cut it out, and then all the other top Ivy's followed. But in most other countries, geography is a top level study, because you're really understanding like, Why are these people suffering from pollution more than these people next to them? What are the prevailing winds? What are the reasons that they live there? What are the historical you know, why? Why were black communities affected by freeways in the 70s more than other communities? You know, do it was partially their political power, partially where they were living because the elevation was flatter, and it was more prone to floods, there's all these geographic questions that are kind of amazing to study. And you can get hyper specific with it, but didn't appeal apparently, to these top level schools until around 2005 when Harvard started its spatial library. And that was like a GIS focus library, and they brought back geography or at least some component of it that was tangible to them. Oh, yeah.

36:53 Nobody does. It is sort of a unique component to this, like science that's everywhere. And nowhere. No, no pun intended. Really? Yeah, fully. So we talked about geo alchemy, and there's a geo alchemy there. It's called geo alchemy to what happened to one or no suffix. Do you know, go straight to two? Yeah. Whoever started one abandon it and somebody else took it over. Yeah, but they didn't want to edge it out or saying I will call it Yeah, if we're in Django. We have geo Django. Hmm, that sounds better. geo geo. Sounds fun. But maybe it's been a live within the Django models and stuff like that. Yeah. When obviously, if you're using Django, you probably would consider that it's actually right built in. You just have to import one more thing in your admin.py. I believe that's what's called or haven't used your Django in a few years. But it's like, you know, right from the get go. When Adrian and somebody were designing Django. They were doing the every block project that they ended up selling to CBS, which was like it kind of like a citizen app on 2007, where they were saying, like, here's what's happening on every block in America. And so geography was built in early into Django. Oh, interesting. I had no idea that that history Yeah, me Django comes out of newsrooms, right. Yeah. From the Lauren. Yeah, world or something like that. So it makes total sense that it would have that some of those I totally, yeah. As soon as he left. Adrian was I forget his last name right now. Anyway, he started every block. And I think he did well, selling that it was like a huge part, as you said, and it's very much related to reporting. So I was like, yeah, I'm more into flask these days, which is just kind of simple. And you can just add what you need when you need it. No disrespect to Django. Django is fantastic. Yeah. So if you're in flask, geo alchemy to obviously, right. Yeah. If you're in Django probably just use the features that are built straight into it. Right. Tons of tutorials about it all around. Yeah. As well supported. Yeah. So you have some that are on ArcGIS, which, as a commercial package, which they actually do a ton with Python as well over as ray which, so if you're doing Python, you might, it might still be a great place to be. But it's also you know, it's me pay Horde. So you might alternatively consider some of the piecing together things from some of the open source tools, right. But yeah, so tell us about your books. Sure. Yeah. I got the opportunity. Having volunteered to review some books in like 2012 and open early open source Python GIS book. It was great. And I was on pack publishers list. And they kept emailing me like, Can you write a book about network security in Python? And I said, No, I can't I don't know anything about cutting edge cryptography. Yeah, exactly. So no, it sounds like you've got the same email. So yep.

39:29 So I finally wrote back to them. I was like, Look, I am a GIS professional. And I can write about that. If you have any interest, let me know otherwise, stop emailing me. And they said, Yeah, actually, we do. So there, they recruited me to write a book on arc pi, which was at that point about five years old, but it was just getting more and more prominence. And I was obviously an early proponent, and I wrote that first book ArcGIS arc pi in ArcGIS, geospatial analysis with Python, which we did a second version of me and my coworker, Dara, and that one focused more

40:00 On the newer ArcGIS Pro, which uses the Python library for that as Python three, the problem with our PI is it's Python two based. So it's, you know, it'll be supported for a while in terms of essary. But like the greater Python world is rapidly leaving it behind there. Yeah, there are some interesting industries that kind of gets stuck like this. So embrace Python, but they embrace it in a way where they like, package it into their tooling. And they just, I'm thinking of like the movie and entertainment workflow, industry, there's like all these tools like Maya and whatnot that have Python built in, and a lot of them embedded the Python two runtime, and they just until they, as a group decide to change like, how it is, you know, yeah, exactly. You know, as he keeps trying to retire, it's ArcMap product in favor of this new ArcGIS Pro, but it's not going to happen, because they have a huge sort of lazy, let's say, No offense, workforce that does doesn't want to learn a whole new software. And I totally get it. So they now maybe there have stuff that works. And they don't consider like building things or maintaining their job. They just need that to work for their job, right. And that's like, it's good for them. But it's a tough place to be. They wrote the software called Avenue, which was their original programming language, or one of them in the early aughts. And people are still using some of those scripts, I'm sure in places so you know, things stick if they work, just like you said, and it makes sense. Yeah. So you took another take on this. Yeah. And it was fun. I actually recruited some help, because reading books is a lot on Twitter and found a couple guys that I follow on Twitter. And we wrote this book, mastering geospatial analysis with Python three, which is like a 16 chapter review of available modules does a pretty good job of going from like very specific geospatial stuff all the way up to big data. And there's a couple chapters in there on interactive web mapping that I wrote around flask and geo alchemy, as well as to Django. I thought it was great. We even touched on it, you know, it, map the they became Omni sigh. It's sort of a I don't know, it's like a GPU database tool that does some pretty amazing visualizations. And they have a Python module pi map D, they may have changed the name. Yes. Things like that, like all these incredibly cool modules that just kind of blew me away. We just touched on it really, because there's so much to touch on. But I think it's a great book. I really liked writing it. And so why Yeah, well, it sounds like it really does a good coverage of all these things if you're into Python plus geospatial, huh? Yeah, well, yeah, it is. It's not like it's selling like hotcakes or anything. You know, these before your parents on the podcast? Yeah. There you go. Yeah. I'll be looking for those checks rolling in.

42:42 That's right. Yeah. It's only 30%. Like the App Store don't want to be fine.

42:47 Just kidding. But speaking of podcasts, you spent close to a year doing a podcast on GIS topics, right? I did. And that was a joy. I mean, it's fun being back here and remembering how much fun this is. My co host, Todd Barr, who is the spatial punk, he's sort of ESeries aside in the thorn of ezri is sort of his like thing. And you know, he's just criticizing, like the lack of support for big data, and just really innovative, or any kind of innovation, the way he sees it. I think that there's some great tools coming out of s ri, like arc pi and arc servers, very cool. But he felt like within the round, they had too much dominance and not enough like push towards really cool new and fast pipeline supports, especially for big data. So sure, we had a lot of discussions with all kinds of different users, though, around like different focuses cartography, to big data to archaeology, we had a journalist up in my hometown who does like it's a Django based website that gets like 3 million hits a month. And they use a lot of geo Django to like locate where all the different see arrests or accidents happen. They just pull in records from the courthouse and automatically add location to it. So it's one of those things like you were saying about just adding in just a little bit of God that makes your field whatever it is way better. Yeah. I recently did an episode on Python and AI ml in journalism. And there was a lot of geospatial stuff there as well about like automatically understanding, you know, like earthquakes in real time, or crime in real time, and those types of things as well. So yeah, I think it spans a lot. Have you heard of a headline down here, headline now they've gotten a national In fact, I think they were purchased by next door, but it is a journal. It's like a local neighborhood journalism site focused on all kinds of cities. But it started here actually, in the neighborhood. I live in San Francisco in the lower Haight, and it was all about trying to automate a lot of the like, exposition within a story if like a restaurant opened or something they could tell you about all the other restaurants that opened within a certain radius, that kind of thing. So machine learning, and ghostwriting. A lot of their stories was

45:00 big focus. I thought it was amazing. Oh, yeah, How interesting. That makes perfect sense for next door. Of course, they're trying to like hyperlocal stuff. Maybe just give us a quick couple of your favorite episodes for your podcast, people link to the show. And people can check it out if they're interested. The cartography. One was Sarah Bell was really fun, because we talked to her about just kind of, you know, cartography and her GIS career, but also like, what it's like being a woman in the field and, and how she's loved it and hated it at times. That one really resonated. I thought the journalism one was fascinating, because, you know, we're talking about all kinds of different crazy stories that they've covered. For instance, they did a you know, where I'm from Humboldt County, there's a lot of marijuana there. And so they did like a comparison with satellite imagery saying, like, do you think this satellite has marijuana grown in it, and pissed off all the locals, but it made for an amazing like application through their, their website?

45:53 This kind of crazy mix of GIS and crime and reporting, which is just such a fascinating way to approach the world, because that's really how things work. You know, things are located somewhere. Yeah. And a lot of those things are affected by the location that you're in, right? quality of schools, the attitude or predispositions the police might have and perceiving the people in that neighborhood may vary a lot, right? Things like that. Oh, yeah. You can tell these stories with with data, right? Yeah. Well, I was. So the CEO of hoodline. When she brought me on as a consultant, I was encouraging her like, you have to know, every gang territory in LA, you know, like, that's the kind of thing that would really make for an amazing story, and especially for automating it. So we never went that far with it. But I think there's still room to do that. I guess they're probably what, right?

46:43 Yes, the Matthew star was really fun. And Todd was a great co host. We're still talking about doing some more maybe next year, but it was a very big year for us both. So you had you had season one? And then maybe you could do season two. There you go. Yeah. Well, with this kind of encouragement, I'm loving this. So yeah, maybe I'll just ride this wave, Colorado.

47:03 It's always fun. I just like to have a sidebar with fellow podcaster. Like, it's really interesting that the people that you get to meet, right, if there's so many folks that I've spoken to that generally, at least before I got into the podcasts five years ago, would have not really bothered to give me the time of day, or at least they wouldn't have been excited about it right. And now, almost everyone I reached out to is excited to be part of the show to share their story to talk and you just get to meet so many interesting people. It's just such an honor to sort of be part of that, right? It's cool. Oh, my God, it really is. I'm loving this. And you must have so much fun. You're doing this full time now or full time Yeah, to show so we had great, that's great. It's just such a cool dive into who we as developers are, which is much more much less monolithic than it sounds like initially. And there's just so many different ways, and things you can do with code. Yeah, and you touched them so much on it when you talk about that person who not necessarily a CS grad, but some other expertise, but now has the magic programming power that most people don't, there's just so many interesting stories and use cases and libraries and applications coming out of that side of the world that it's it's bottomless, it's cool, just got a grasp of the for loop is and you're well on your way.

48:18 Exactly pip install, you're gonna be good. Alright, let's see, one more thing that we threw out to talk about was web jazz for flask, or Django API's tells me about that real quick. Well, you can just do some amazing API's with flask, or Django in general. But with geo, especially with if you're building like a phone and iPhone app or an Android app, you know, you can just send your location and have all kinds of context sent back to the application based on these API's. So they're quick to put up, they're easy to maintain. And you can integrate spatial right into it, it really isn't that much work. There's a little bit deeper understanding of like the type of spatial function you may need for a specific moment. But none of that is going to take a master's degree. In fact, I think just like reading some online tutorials or some stack, there's a geo Stack Overflow even I think, Oh, yeah, the GIS Stack Overflow, something like that. It's very, very useful Stack Exchange. Yeah, probably. There you go. That's it. Yeah, yep. Or you can buy my books.

49:16 Yeah, now I the amount of data that you have to exchange is actually pretty small for these these applications, like where am I or what path have I taken? Yeah, and so on, but it's easy to build API's. Right. And, I would suspect, like, on the back end, have it something like geo seek walk geometry, too, just indexes, make sure you have indexes. The coolest thing I ended up doing at Zillow was building what we called the surveyor it was just a working title. But it was a just literally a web map built in flask post GIS to be able to actually see a lot of the data that we were building and creating and comparing because internally, we didn't have like a mapping tool that could really speak to all of the geospatial data that we had. It was such a surprise to me, and I ended up winning the hack week. There.

50:00 Earlier this year before I left, and that was just like, so fun because I presented it at the end of hack week, and they instantly were like, yes. Oh, why don't we have this? Sorry? That's awesome. Yeah, that's super cool. Congratulations. Yeah, one more thing on the library side, I guess is probably worth throwing out. So we've talked a lot about server side things. And you talked about your cool experiences with JavaScript and the front end, if I'm gonna do mapping data, especially interactive maps, there's got to be some fancy library that goes on the front end that makes this happen. So what would you recommend to check out? Sure, yeah, there's less fancier ones, I would say that leaflet is considered which is a JavaScript library, it's considered a medium level knowledge that you could use and then map gl mapbox. gl, I'm sorry, is a little bit higher level, but it has really fine controls over what you can do. And you can have all kinds of events, there are some pure Python front end tools, like dash and do others that do have mapping integrated, as well as graphs. And that can be really useful. You don't have to handle much of the HTML if you don't want to, but I find it if you're going to get into interactive web mapping, really useful to know both Python and JavaScript to make them work. Yeah. Yeah, a lot of the interactive stuff happens on it has to happen on the client side, right? That means some kind of JavaScript.

51:22 Get used to curly braces, right? Yeah, exactly. Don't forget the semicolons. Actually, in JavaScript, you can't forget them, which is kind of good. I guess. It took me a long time to get into JavaScript, because I was so used to the clean nature of Python like and yeah, no joke that attracted me to Python, it was so important to be able to read the code. Yeah. And I feel like JavaScript used to be closer to that world used to be pretty simple. Now, it's like, you know, all sorts of dependencies require this. And then like, you know, you pick up these things, and it's just like, Oh, my gosh, a very, or you pre compile or transpile, your TypeScript over to this. And then you pack it up, and you minify it like, Whoa, whoa, whoa, what happened to just including something in the page.

52:01 I mean, when I do demos, or proof of concept, it's always pretty vanilla with like, one mapping library or something. There's some other cool ones, too, that Uber is putting out a ton of really cool stuff like deck GL. And there's another one I forget. But if you want to get into super fancy data visualization, I recommend looking into Ubers set of JavaScript tools. It's pretty amazing stuff. But yeah, I prefer a much more vanilla implementation myself, just for my brain to manage it. And to understand

52:32 typical AI, let me close out our conversation with one sort of more general question here. So people out there listening, a lot of them have Python skills, obviously, or building up their Python skills. And maybe they're interested in this, but they haven't done anything with geography, cartography, GIS, how much do you need to know to be useful? and powerful? Like, do you need to go get a degree? How much work is it to get into this? If you want to really understand the theories behind geography? Yeah, I think grad school can be really used for just an undergrad degree, because it really helps you understand, like, just why it's cold in the Arctic and warm at the equator, like just at that basic level, it is amazing. And the relationship between the two, and the relationship between that and human culture. So if you want to really understand humans, yes, go into geography. But if you want to make a geospatial data pipeline, I think what you really have to know is just like, what is the data that you're getting in? And what is the data you want to get out? And what are your resources in between? You don't have to know much more than that. There's all kinds of tutorials, there's books, there's, there's a weekend of exploration ahead of you. It's not that hard, but it is really powerful. Awesome. Well, it does sound like a really fun area to get into. And a lot of times these visual aspects are pretty cool for people who are not sure that they're necessarily programmers. But then if you build something that you can see and touch and share, like all of a sudden, it makes a big difference I find completely agree. It's one of the coolest things you do and I still get that thrill I got, you know, from showing my grandmother, the little lat long that I was able to pull from a shapefile.

54:05 Super cool. Alright, now before I let you out of here, you got to answer the two questions. Always put it in the show. If you're going to write some Python code, what editor Do you use lately? It is pi charm. I really like it before that it was sublime, because it's just simple. And you don't have to think too much about it. But pi charm is fantastic. Yeah, it's my favorite as well. I love it. And notable pi package. I mean, we definitely threw a bunch out there. I think pi shape is the like lowest level one because unfortunately still this really ancient data format is the major data format in GIS. So pi shp, that tool will allow you to read and write it just in pure Python. Awesome. Cool. All right, so final action. people interested in GIS, what do you say to them and Python? It's an amazing field. I put those together. Yeah, this is the future or maybe the future is here. But either way there's so much that can be done in this space. It's we're just touching on it like people are just hearing about it, let alone realizing that there's so much

55:00 Much more to do with it. So yes, come join me. Yeah, it definitely feels like maybe the future is here at, like part of the future like I remember so so many of these things, the GPS stuff, the Zillow stuff, all those different just literally blew my mind when I saw them. And I'm reminded of the William Gibson quote, like the future is here. It's just not evenly distributed. Exactly. Yeah. Yeah, that is so true. The future is generally in your hands these days, but not everybody has the same phone. That's true. That's true. All right. Well, Silas, thank you for being on the show was great to chat with you. It was so fun, Michael, thank you so much. Yeah. Thanks, Jenna. See,

55:33 this has been another episode of talk Python. To me. Our guest on this episode was Silas Tom's has been brought to you by us over at Talk Python Training and linode. Simplify your infrastructure and cut your cob bills in half with linode. Linux virtual machines develop, deploy and scale your modern applications faster and easier. Visit talk python.fm slash linode and click the Create free account button to get started. Want to level up your Python. If you're just getting started, try my Python jumpstart by building 10 apps course or if you're looking for something more advanced, check out our new async course the digs into all the different types of async programming you can do in Python. And of course, if you're interested in more than one of these, be sure to check out our everything bundle. It's like a subscription that never expires. Be sure to subscribe to the show, open your favorite pod catcher and search for Python. We should be right at the top. You can also find the iTunes feed at slash iTunes. The Google Play feed is slash play in the direct RSS feed net slash RSS on talk python.fm. This is your host Michael Kennedy. Thanks so much for listening. I really appreciate it. Get out there and write some Python code