#176: The Python Community by the Numbers Transcript
00:00 Michael Kennedy: The Python landscape is changing pretty dramatically. Python's rapid growth over the past five years means it doesn't look the same as the early days. On this episode, we'll take a deep look inside the state of the Python ecosystem with Ewa Jodlowska and Dmitry Filippov. They led the recent PSF and JetBrains Python survey. And they're here to dig into the results. This is Talk Python to Me, Episode 176. Recorded August 27th, 2018. Welcome to Talk Python to Me, a weekly podcast on Python, the language, the libraries, the ecosystem, and the personalities. This is your host, Michael Kennedy. Follow me on Twitter where I'm @MKennedy. Keep up with the show and listen to past episodes at talkpython.fm and follow the show on Twitter via @talkpython. This episode is brought to you by Linode and brilliant.org, check out what they're offering during their segments. It really helps support the show. Ewa, Dmitry, welcome to Talk Python.
01:09 Panelists: Thanks for having us, Michael. It's a pleasure to be on your show.
01:12 Michael Kennedy: It's great that you're both here. I'm really glad that you came to talk about this project. It's really important, I think, to take the pulse of the community every now and again. And this is a perfect project for that. So, excited to share, excited to share that the results you guys got from this whole survey thing that you've done. Before we get to that though, let's start with your story. How'd you get interested in Python? Ewa, start with you, I guess.
01:35 Panelists: Sure, so I actually stared working on Python by itself. And it was a very great experience and eventually, I left the employer that I was working through as a meeting planner and joined the PSF. And that's how I got involved in the PSF and the Python community.
01:54 Michael Kennedy: That's great. Dmitry, how about you?
01:56 Panelists: So, I started my development career at Astronautics Corporation of America, working on Boeing 777 and 787 projects. So if you are on those airplanes, be sure they're safe. I've put a lot of efforts into testing them. Like actually working on such projects, you quickly learn that testing is a extremely important part of software development. But, my love story with Python actually began when I started working for Oracle, about seven years ago. I was responsible for interoperability between Oracle-Linux and Oracle VM. So, back then I would make sure that Oracle Linux runs smoothly as a guest virtual machine on Oracle VM. So that required some Linux kernel hacking and writing a lot of automated tests to check that those Oracle Linux versions work properly. So Python came in handy for writing tests and automation. That's how I started with Python. Later, I discovered an open position at JetBrains.
02:52 Michael Kennedy: You thought you'd go make some Python tools for Python people.
02:55 Panelists: Exactly.
02:56 Michael Kennedy: That's cool. It must have been fun to work on the 777, especially the 787. That's a really fantastic airplane.
03:03 Panelists: Yes, it's a dream plane, dreamliner.
03:06 Michael Kennedy: It totally is. I've gotten to fly it a couple time and I've always really enjoyed it. I guess, just to recap what you guys do day to day. Dmitry, you're in charge of marketing and overseeing PyCharm at JetBrains. Would you say that's your general role?
03:21 Panelists: Yeah, basically I'm responsible for many tasks around Python project, PyCharm project, I mean. I do some product management activities like making sure that PyCharm is released on time, with everything ready by the release date. I also responsible for everything marketing. So, as an example, I coordinate content creation, our appearance at major Python events. I run PyCharm blog, manage partnership campaigns, among many other things.
03:48 Michael Kennedy: Yeah, that sounds like a fun job. Ewa, how about you?
03:50 Panelists: So, I'm the director of operations at the Python Software Foundation. So that pretty much means that I oversee everything that the PSF does. That includes PyCon, which is the conference that we run yearly here in the US. And then our grants program as well as any trademarks that we manage for the Python programming language. And several other things like managing our staff and day to day things like that.
04:15 Michael Kennedy: Yeah, that's quite a bit. And PyCon itself is a big project, right?
04:19 Panelists: It absolutely is. I would say at this point it takes up two full time employees, at least eight months out of the year.
04:26 Michael Kennedy: That's incredible. But it's a huge event. It's really fun. So speaking of PyCon, the reason I reached out to both of you is you did a really interesting presentation at PyCon for a project that had been maybe six months in the making but previous to that, about this survey. So, PSF and JetBrains put some efforts together to get a bunch of the Python community come fill out a nice questionnaire and you guys at JetBrains did a really great job on analysis of it. So, we'll get to talk about all the results there and stuff. But let's start with how this project came together.
04:59 Panelists: So, the story of the Python developer survey goes back in 2016 when I personally was fascinated by an ideal fronting a survey on Python developers. By the time I already had some experience with surveys amongst various Python users. But I wanted run something on a completely different scale. So, I gathered a working group, comprised of people from the resource team, where people have a lot of experience with math and stats. People from the survey team who helped me to organize previous surveys. People from the marketing department to help us to promote the survey. Designers, to create a nice looking report. And we did the survey, which was quite a success. We gathered something around, I believe, four thousand responses. Through our own channels, through advertisement on Google, Stack Overflow and others.
05:47 Michael Kennedy: This is the 2016 survey, right?
05:49 Panelists: Yes, this is 2016 survey. Sorry, so it's telling the story prior to our latest 2017 survey. So, back then, we've got some nice results and they were really good and of high quality. I did a talk at PyGotham, the 2016 in New York and the survey was well received. The only problem with that survey was that we had around 50% of PyCharm users among respondents. Which was way higher than our estimation of PyCharm share among Python editors and IDEs.
06:21 Michael Kennedy: In some sense, you were surveying your customers rather than the broad.
06:23 Panelists: Exactly, exactly. So back then I started thinking about how we can make the survey better. How to make it independent and unbiased. So later in 2016 I bumped into Ewa, I believe at PyCon APAC in South Korea, if I'm not mistaken. Back then I told her about that Python survey that we did and- together with Ewa we came up with an idea that Python Software Foundation can be the key to make this survey much better. So in 2017, we ran the survey in a different format. Instead of promoting survey through JetBrains channels like we did in 2016, we promoted this only though the official Python channels, official Python Software Foundation channels. Exactly, so we reached via python.org. We put banners up on the download pages which received the highest amount of feedback, as well as our Twitter, on the PSF blog. We reached out to several communities that we work with since the PSF gives out grants pretty much globally. And we're in touch with a lot of different communities, so we're going to continue to do a lot of outreach that way as well for the next version.
07:39 Michael Kennedy: Yeah, that's really cool. I think, I think having this sort of neutral, vendor neutral version, you know really puts a lot of authority behind this survey, right? Like it's one thing to say well look, PyCharm is really popular among PyCharms users. Or maybe, you know, Python thre3e version Python 2 usage varies on that because maybe more professionals use PyCharm. I'm just making up this stuff, right? But having it really neutral seems like a super valuable addition. And we'll talk about where some of the participants came from, but it sounds to me like you really pulled that off quite well.
08:15 Panelists: Yeah, we think it worked. It worked very well, looking at the countries that we received responses from. We noticed that we're missing a lot of feedback from people in African continent. Probably Indian continent, but we're going to improve on that for the next version.
08:32 Michael Kennedy: Yeah, and now you know, right?
08:33 Panelists: Exactly. There's always has to be a starting point, so you live and you learn.
08:39 Michael Kennedy: Exactly, so what were some of the goals of the survey?
08:41 Panelists: We initially, we wanted to slowly, We did the survey for the purpose of making out an accurate landscape on the Python community. We wanted to identify the latest trends and gather insight on how the Python development work looks today. We basically want to learn how Python is being used by diverse developers, what compliments its use, and what makes it one of the most loved language. And from the PSF standpoint, it was important to us because you know, the PSF is there to support the community. So the more that we know of the direction that the community is going in, the more we'll be prepared to help them in the future. So, it's a very useful tool, not just for the community, in my opinion, but also for the non-profit behind the language.
09:28 Michael Kennedy: Yeah, certainly knowing where things are going and where the support needs to be and you learn some interesting lessons, like you were hinting at there, about, say, you know, maybe, you know, the PSF does some outreach and some support for various communities. And maybe some of these areas where it's somewhat underrepresented, you could, maybe do more, right?
09:48 Panelists: Exactly.
09:49 Michael Kennedy: Yeah, that's great. So, Dmitry, you said in your 2016 edition, you had about 4,500 developers and now, this survey had 9,500 developers from almost 150 different countries. That's pretty awesome.
10:03 Panelists: Yeah, it is. And I hope in this year, actually, when we're going to run this survey next time. I hope we'll get even more respondents. So it will be bigger and better in quality as well.
10:17 Michael Kennedy: Yeah, absolutely, this, I'll put this question out to both of you, how can people, so last time it was in October 2017 and I think you're planning similar time frame for this one of 2018, so that'd be in about a month and a half or so, the survey will come out again. Is there a way for people to say, join a mailing list or get announcements so they don't have to like, bump into a banner on python.org or something to know about it.
10:41 Panelists: So we don't have a mailing list, but we'll definitely announce it via the PSF Twitter account which is @ThePSF and we'll be announcing it via our Slack channels and our blog, which is pyfound.blogspot.com.
10:57 Michael Kennedy: How long was the survey open for? This one, that you already did?
11:01 Panelists: It was open for three weeks.
11:02 Michael Kennedy: Three weeks, so people have a decent amount of time and I'll definitely share it on my Twitter account as well and probably talk about it on the Python Bytes podcast. And try to get the word out. That'd be great to have as many people, you know, participating cause I know there's more than 9,000 Python developers out there.
11:18 Panelists: Definitely. Absolutely.
11:21 Michael Kennedy: Ewa, do you know the number actually? Do you have an official PSF number? Do you have an official PSF number?
11:26 Panelists: No, there is no PSF official data on that. But we're definitely hoping to get there with a survey like this.
11:33 Michael Kennedy: Yeah, that's great.
11:34 Panelists: I think it's really difficult to estimate the overall number of Python developers and there can't be an official number for that. Because in the end, like with the 2017 survey, we run into the problem cause, Who is a Python developer? It's very difficult question because people use Python to different extent. Some of them use Python fulltime as their primary language, some of them use it as an an additional language and it's hard to say how many can be considered to be as like, to be called Python developers. Exactly, exactly. Even for those people that are not, quote unquote, developers, in their day job. They might use Python as a tool in their day job. It's definitely difficult to categorize.
12:22 Michael Kennedy: Yeah, that's a...
12:22 Panelists: Exactly, for example, many people who wear, like a, data analyst hat, sitting in large banks, doing a lot of data analysis, can we call them Python developers or data analysts if they use Python?
12:38 Michael Kennedy: Yeah, it's got to do 20% of their day in Python? Or 15, is that enough, right?
12:41 Panelists: Right, yeah. Yeah, exactly.
12:44 Michael Kennedy: Yeah, that's a really good point. Another thing, another idea I want to just throw out there as people think about this. This is not my idea, this idea came from Scott Hanselman, whose really well known in the developer space, has his own podcast. But he has this really great idea, talking about sort of reaching all of the developers and he says, look there's this entire world of what he labeled dark matter developers. We know they're out there, but we can't see them, we can't detect them, they don't come to conferences. They're not on social media, they just go to their job, and they work in their code. And they basically are not visible to the community, in a sense. If there's tons of those people, and that concept, I think, really makes it hard to know the entire extent, but you guys are doing a pretty good job of trying to take that pulse anyway.
13:29 Panelists: And we hope to improve it year on year.
13:31 Michael Kennedy: Yeah, for sure. So, one of the things you did with this, and I hinted at the beginning is you gave this as a presentation at PyCon in PyCon 2018. So it was Python By The Numbers. So if people want to watch your presentation, with some slides and graphs and stuff, they can check that out there. And then of course, the results of the survey are up and I'll put the link in the show notes. And who was responsible for actually writing up the analysis of that? Was that JetBrains, the PSF, someone else?
13:59 Panelists: We did it all on the JetBrains side. So our partnership Python Software Foundation was that Python Software Foundation was responsible for promoting the survey through their official channels and JetBrains did all the rest. So we set up the survey, set up the logic. We then analyzed the results and created the report.
14:22 Michael Kennedy: Yeah, congratulations because that was a really really nice write up. I love each section had like a here's the key takeaway from this section and would sort of, you know, not just show the stats but put it into more colloquial terms, I guess. Like, here's a bunch of stats and oh, by the way, what that means is there's, you know, four out of five Python developers use Python as their main language. Or, you know, something to that effect would be like a thing that would be in there. That's, that was really cool.
14:48 Panelists: Yeah, that was a cool feature of this report.
14:50 Michael Kennedy: Yeah, I really found it super, super approachable. So, let's go back to, first, before we get into the results, I want to make sure we make it really clear on who participated, where they're coming from, what groups are being measured. Because, you know, one of the whole reason I wanted to have you all on the show is there's so much hype around Python right now, and there's so many surveys that are just putting Python at or near the top where it has a really steep growth curve, or something like that. I'm thinking Stack Overflow surveys, the IEEE survey just came out, it had Python right at the top, things like that. But those are all, people say well, yeah, that's what that survey said, but it's this slice of people or it's that slice of people. So the people that took this survey are primarily coming from the Python community itself, right? So, let's talk about where, maybe, Ewa, you could cover this, where did the participants actually come from? Like what sources did you drive them to the survey?
15:46 Panelists: Sure, so the sources that we used to get people to the survey were python.org website, mainly through the downloads page and the home page. PSF Twitter account, the Python blog, LinkedIn, as well as, I believe your podcast even brought some people in.
16:04 Michael Kennedy: Yeah, I saw you list Talk Python podcast, and I'm like wait, is that mine? That's cool. Yeah, that's great. So 60% came from python.org and then maybe social media pulled in...
16:13 Panelists: Right.
16:16 Michael Kennedy: Pulled in for you guys maybe pulled in another 30%. So the vast majority was sort of from the PSF, right?
16:22 Panelists: Correct, yeah for the next version, we definitely want to do more one on one outreach to communities that might not necessarily have the need to go to python.org because mainly that's, those are people that are downloading or they might not follow us on Twitter or whatnot. So we're going to do a lot more one on one outreach as well.
16:40 Michael Kennedy: Yeah, that's really tricky though, right? Because if you do one on one outreach to certain communities they might have really different views or like really different uses, right? Different style.
16:49 Panelists: It'll definitely give us the global outlook, right?
16:52 Michael Kennedy: Yeah, absolutely. Alright, so, let's I guess, maybe let's start talking through some of the results. We could just start at some really high level stuff, like age, maybe give us a quick summary of the ages of people who took this. Was it like mostly young students or what kind of professional developers? What did you see?
17:09 Panelists: We definitely saw a younger audience. Half were between the ages of 18 and 29. And a fourth were between the ages of 30 and 39.
17:19 Michael Kennedy: Yeah, that's good. There were still a fair number of people who were older. I know a lot of folks feel like if they're, you know, 40, it's too late to learn to program. But there was plenty of folks in those age ranges, right?
17:30 Panelists: When you put all the older categories together, they definitely added up to a significant amount.
17:36 Michael Kennedy: Yeah, that's great to hear. Because I feel like a lot of people are like, well, I missed my chance to be a developer. And I just don't feel like that's really true. Another interesting...
17:44 Panelists: It could also be the way that we were marketing the first one, that we need to improve on for the second one. Is to reach that older audience as well.
17:49 Michael Kennedy: Right, absolutely, the whole dark matter thing. This portion of Talk Python to Me is brought to you by Linode. Are you looking for bulletproof hosting that's fast, simple, and incredibly affordable? Look past that bookstore and check Linode at talkpython.fm/linode. That's linode, plans start at just five dollars a month for a dedicated server with a gig of RAM. They have 10 data centers across the globe. So no matter where you are, there's a data center near you. Whether you want to run your Python web app, host a private git server, or file server. You'll get native SSDs on all the machines, a newly upgraded 200 gigabit network, 24/7 friendly support, even on holidays. And a seven day money back guarantee. Do you need a little help with your infrastructure? They even offer professional services to help you get started with architecture, migrations, and more. Get a dedicated server for free for the next four months. Just visit talkpython.fm/linode. Another interesting fact that you guys had highlighted in your talk was that from younger developers who participated, more disproportionately came from outside the US, I guess, is what I'm trying to say.
19:01 Panelists: We had a lot of people that came from India and China between that age range.
19:06 Michael Kennedy: Yeah, I just think it's interesting that people are maybe people see more opportunity or you know, a better opportunity by jumping into software development there, whereas in more stable, traditional markets like the US or Europe maybe, there's less people going into software development. I don't know, it's an interesting takeaway though.
19:24 Panelists: Yeah, it could also depend on, you know, what schools are using. It's an unknown at this point.
19:30 Michael Kennedy: Yeah, for sure, for sure. How about countries? So, speaking of coming from outside the US, like, US was probably still the most popular, right?
19:38 Panelists: Yeah, the US brought in 18%.
19:40 Michael Kennedy: What about other ones? Where else did people come from?
19:42 Panelists: On the second place, we have India with 13%. China's 7%, 6% for United Kingdom, 5% Germany, then goes Brazil, France, Russia, Poland, Canada.
19:55 Michael Kennedy: Yeah, it starts to tail off pretty quick probably. As farther you go out.
19:58 Panelists: Yeah, exactly, exactly. So, well, Brazil has 4% and like the next, France is 3%, and all the rest is two and one percent.
20:08 Michael Kennedy: Yeah, interesting. I wonder how much of this is a language thing? Alright, to say with 13 almost as many people came from India as they did from the US, whereas English is not the, you know, first language of folks in India. So, that's almost a stronger endorsement to have 13% coming from there, right?
20:25 Panelists: True, true.
20:26 Michael Kennedy: Pretty interesting. How about experience? Like years of experience, you said sort of, ask people how long they've been in programming? What's the time that a average Python developers been a Python developer?
20:38 Panelists: So the Pythons survey respondents reported a wide range of experiences. Like we had 22% of people with less than one year of experience in the IT industry. We had the same percentage for people with more than 11 years of experience in software development. And there was a small discretion in the middle.
20:59 Michael Kennedy: Yeah, so, some new people, which is great. You've got to have new blood or it's not going to be a vibrant community but also, you know, some experienced mentors around, right?
21:08 Panelists: Yeah, exactly. I think this shows how strong the Python language is growing to be. Just like you said, Michael, you need, you need newcomers and you need people that have been doing this for many years in order to be, in order to be a successful community.
21:22 Michael Kennedy: Yeah, absolutely. Another thing that was interesting, you measured this in a couple of ways to try to get the sense of how large of a team, or what types of folks work there. So you have are do you work by yourself or do you work in a team was one interesting thing you asked and then the other one was how large of a team do you work in? So, let's see, you guys want to speak to that?
21:45 Panelists: Sure, so, in terms of working versus working in teams versus working independently, we found out that three out of five developers work on independent projects. So, 56% work on their own projects independently. While a 4% work as external consultants. And then in terms of team sizes, about three quarters of the developers who work in teams work in very small teams. So 74% of their respondents in the survey said they work in team sizes between 2 and 7 people. And then the next highest group was 60% in 8 to 12, and that's already getting kind of large.
22:28 Michael Kennedy: Yeah, yeah, that's most of them right there. I think it's really interesting to measure and I think it also measures something interesting about Python as a technology. So when I think of like, big corporate teams, you know, I've done lots of training and I've traveled lots of companies and worked with their various teams and, you think of like these, like .NET folks, and Java folks, on these like large teams that are building like, enterprise software and stuff like that. And those I think, probably tend to be a little bit more than what you're measuring here. And Python, I think one of its main advantages is it doesn't require a team of a hundred people to do the same thing. And I think you're kind of measuring that here. Or there's evidence of that here.
23:11 Panelists: Yeah, it could definitely allude to evidence of that being the reason, absolutely.
23:16 Michael Kennedy: There's a really interesting story that came out from Mike Discroll's Python Interviews book and I'm sorry, I don't remember who he interviewed because there's like, 10 people we talked about that actually talked about this history. But there was, back in the day, there was Google Video, and there was YouTube. And YouTube was a little scrappy start up using Python, and Google had a team of like, 100 people using C++, I think they had said. And little scrappy YouTube kept adding features and features faster and faster than this large C++ team at Google. So Google eventually just bought them. Just bought YouTube. It's the way to win, but you know it's kind of, I feel like this part of this survey kind of captures that contrast there and I think it's pretty interesting. So, let's talk about jobs and what people are doing. Like you ask about whether people were full time employed, whether they were students, where they're free lance or self-employed, which is an interesting distinction. I think it's a proper one, but interesting. What did you find there?
24:14 Panelists: So we found that the large majority of the respondents, 52% are fully employed by a company or an organization. And then, to what we were talking about earlier, 28% of the respondents are students.
24:28 Michael Kennedy: Yeah, that's pretty high. I think that's a good thing though. I mean, students now become professional developers couple years later, right?
24:34 Panelists: Yeah, absolutely. It keeps that funnel being filled. From newcomer to long time contributor.`
24:40 Michael Kennedy: Yeah, especially where...
24:40 Panelists: Well, it's also good to see that Python is finally being introduced in schools. Because that is something that we're continuously discussing, even at that level.
24:48 Michael Kennedy: Yeah, maybe we could talk about it, just for a second. I'm sorry I only have like, really US visibility to this because I don't pay that much attention to schools all over the world. But I know in the US, like Python has been becoming, or is the most taught first year computer science language at universities, which is great. And I think that trend's generally global. But in the US, we also have these advanced placement, AP classes, where you can take a Computer Science official course as a high school person. And that official program is like, nationwide. And I think it's still in Java, which makes it really, really the basically just- nobody seems to want to change that to catch up with the times cause you know, it's the way we've done things, right? Do you ever address that or try to deal, like get around that or try to convince people maybe Java's not the best option for students to start with?
25:44 Panelists: So, this is a continuing discussion, I guess, debate, argument, whatever we want to call it. So at PyCon, we even have a dedicated workshop called the Education Summit where educators get together and they discuss how to approach these situations and how to best convince people that Python is the right way to go. Whether it's you know, the tools that they have set up for children to learn Python or the type of support that's out there, but mainly I think one of the scenarios that we're running into is that a lot of places just lack the instructors, maybe?
26:21 Michael Kennedy: Yeah, whose going to reteach all the teachers?
26:24 Panelists: Exactly.
26:24 Michael Kennedy: That have spent all the time to learn Java, right? They're not really developers. They're probably like math teachers or chemistry teachers or something, right?
26:31 Panelists: Exactly.
26:33 Michael Kennedy: Interesting. Yeah, I'm sure that's a constant challenge that you guys deal with at the PSF. Alright, how about salary range? Like, one of the things that I think, probably hasn't hurt Python's growth is you'll see a lot of different surveys from like, you know, Indeed, or the job board type places where Python developers get paid above average, at least in the software space. What'd you find?
26:55 Panelists: As we found that some are definitely above average. I'm not sure what we want to call average at this point.
27:03 Michael Kennedy: It varies so much by region, right?
27:05 Panelists: Right, exactly.
27:05 Michael Kennedy: Even in the mid-west, right? You live in the mid-west in a small town versus if you live in San Francisco. Like one of the rich people, same salary in San Francisco, you might not be able to afford a house, it's crazy.
27:15 Panelists: Exactly, and that is even a bigger range when you compare it to countries like India, right? So I think that that even is evident in some of the other surveys that we see other communities put out. But yeah, it's hard to say which groups above average and below without knowing that.
27:36 Michael Kennedy: Sure, so one of the things that you covered that I thought was really interesting was global job demand. And if you just take just the straight numbers, demand for Python developers in the US is, I can't remember, I'm trying to visualize the graph from last time I saw it. I think it was roughly like, twice as much as any other country. But that's partly cause there's so much tech. What I found was really interesting was how much growth there was for Python jobs in different countries.
28:05 Panelists: Yeah, exactly, so I used the job board that's on python.org to gather some of those stats. So I'm not sure how well those results, you know, actually blanket everything that's happening but it's definitely amazing to see what's happening in Brazil right now. How it's, you know, doubled in percentage in some of the areas, especially in countries that we might not consider as tech hubs. But this is telling that maybe in the future, we will.
28:34 Michael Kennedy: Yeah, that's really interesting, it definitely could be. Yeah, so you had like, year over year growth from 2016 to 2017, like four times the demand in Brazil. Twice in China, and almost not quite twice. But quite a bit in India as well. And those were all really good signs for people that know how to write Python.
28:51 Panelists: Yeah, absolutely. Probably these chart explains the numbers that we gathered with the survey on the distribution in more countries. So we also secured India has grown a lot. China, Brazil.
29:04 Michael Kennedy: And those were all the ones where the participants were now coming from, right?
29:08 Panelists: Yeah, exactly.
29:09 Michael Kennedy: Yeah, yeah, great. So, Dmitry, one of the things that you talked about on the survey that I thought was pretty interesting was, and we kind of touched on this earlier is are you a Python developer or a developer that uses Python? So one way you try to get at that was is Python your main development language? So what'd you find out there?
29:30 Panelists: Yeah, so, with the survey we identified that almost four out of five Python developers use Python as their main language. And comparing to the previous survey, it's 4% increase.
29:45 Michael Kennedy: Right, so we have more Python developers, and the ones among them, more of them are choosing to make it their main language.
29:51 Panelists: True, true. So, yeah, and I think that this can be explained by rapidly growing popularity of Python among data scientists. Cause the segment of users is growing much faster than others.
30:02 Michael Kennedy: Yeah, I guess if I was going to take like, what is the one, if there was one biggest takeaway from your survey that eats you dead. It would have to be that wow, data science is making an impact.
30:12 Panelists: Absolutely. True.
30:13 Michael Kennedy: Right, there was a lot of, if you look at the growth curves of Python popularity over time, there seem to be a pretty strong increase in the rate of growth 2012, 2013, and then it's gone on from there, and I feel like that's when, you know, and Jupyter is kicking in. The machine learning stuff in Python is kicking in. There's just a lot of people that are sort of moving away from R and those other languages and you know, Matlab and whatnot, into Python. It definitely shows up here, doesn't it?
30:42 Panelists: Mhm-hm.
30:42 Michael Kennedy: Yeah, so you asked what are the types of, like what type of developer are you or what types of development do you do? And you have 50% data analysis and 48 web developers. That's right there, isn't it?
30:56 Panelists: Yeah, actually, in the survey, we asked two similar questions. What do you use Python for, where respondents could select multiple answers and what do you use Python for the most? Right. With only one possible answer. So, you just described the results from the first question, where multiple answers were allowed.
31:14 Michael Kennedy: What about the other one?
31:15 Panelists: But yeah, about the other one, when we asked about the primary type of development, 26% of users identified as web developers. Outpacing data analysis reported by only 18% by a wide margin. However, when we combine, we had two options for data science, actually. Data analysis and machine learning. And if we combine those two types of development, we come up to 27% of people doing primarily data science. So actually, there is 26% of web developers and 27% of people doing data science.
31:53 Michael Kennedy: So it's like one to one. And you asked this question on the survey, you asked them, I can't remember exactly what the question was. But it was like, what do you expect people primarily do or what ratio among data scientist and web developers? And then one out of ten got that ratio correct.
32:12 Panelists: Yeah, that was one of the most intriguing questions in this survey. So we asked people to estimate the ratio between web developers and data scientists in the Python world. Respondents could answer based on just on their own experience, belief, and general gut feeling because we explicitly asked about primary types of development in the previous question, we could compare the community's perception with real stats that we've got from this survey. So based on the data we've got from the question about main type of development, ratios one to one. And just so, as you said, only one in ten people guessed this correctly.
32:49 Michael Kennedy: Yeah and the most common answer for the community was there are five times as many web developers as there are data scientists.
32:57 Panelists: Right. Exactly.
32:58 Michael Kennedy: That's way, it's not like a little off. You want to, what I find most interesting about this, like this is already super interesting, but what I find even more interesting is half of the people who took this survey are data scientists. And still, that was the most common answer. So, in some sense, I think it's interesting that even probably among data scientists, they feel like they're more underrepresented than they actually are.
33:24 Panelists: That's a great point.
33:26 Michael Kennedy: Yeah, ain't that crazy? Yeah, so I think you know, maybe this will sort of shine a little light on that, right? That people are like wait, those people do what I do? I thought I was the weird guy in the corner that they came and asked for graphs for whatever, right?
33:37 Panelists: Well hopefully it will bring many communities together.
33:40 Michael Kennedy: Yeah, you know, it makes me think of the key note from PyCon 2017, I think it was when Jake Vanderplas talked, you know he did a great key note there, here in Portland actually about Python, the Python community being a mosaic of all these different ways in which people use Python and their goals and what their whole training and what they want to do and the types of things they're building. And just appreciating the differences and I think this is just one more reason we should appreciate that, that keynote.
34:09 Panelists: Yeah, I agree, hundred percent. Even when we look at the 2018 key note by Q, and she's talking about kids learning how to use Python to program Raspberry Pi so they could water their flowers outside. It's a wide, wide range and a big mosaic, so I'm glad you brought that up.
34:30 Michael Kennedy: This portion of Talk Python to Me is brought to you by brilliant.org. Many of you have come to software development and data science through pasts that did not include a full on computer science or mathematic degree. Yet, in our technical field, you may find you need to learn exactly these topics. You could go back to university. But then again, this is the 21st century, and we do have the internet, why not take some engaging online courses to quickly get just the skills that you need? That's where brilliant.org comes in. They believe that effective learning is active. So master the concepts you need by solving fun, challenging problems yourself. Get started today, just visit talkpython.fm/brilliant and sign up for free. And don't wait either. If you decide to upgrade to a paid account for guided courses and more practice exercises, the first 200 people that sign up from Talkpython will get an extra 20% off an annual premium subscription. That's talkpython.fm/brilliant. So another hot button item in the community has got to be Python 2 versus Python 3, right?
35:34 Panelists: Exactly. So, with this survey, we've learned that Python 3 is now, is strong leader with 75%. And Python 2 is used as the main interpreter by only 25% of people.
35:46 Michael Kennedy: 75% Python three?
35:48 Panelists: Yeah, and the share of Python three's growing rapidly. In 2016 survey, we had just only 16% of people using Python 2 and 40% for Python 3. So it's two times increase for Python three.
36:03 Michael Kennedy: That's really great, I think it's super good news for the community. I feel like the story used to be I can't switch to Python three because library x, y, and z doesn't support Python three and I need it, right?
36:16 Panelists: Mhm-hm. Yes.
36:17 Michael Kennedy: And now I feel like the story is there's these 3 libraries, I really would like to use, but they only support Python 3. So I better move to catch up. You know what I mean, like Django, Numpy, all the new features there are only Python 3. And you look at some of cool, parallel web apis and stuff that are only available in Python 3. It's definitely a changing landscape.
36:40 Panelists: Yeah, true. Another interesting finding from this same question that the distribution of Python 3 users is different comparing web developers and data scientists. So among web developers, only 70% of people use Python 3. At the same time, around 80% of data science people use Python 3. So more data scientists use Python 3 than web developers
37:06 Michael Kennedy: Why do you think that is?
37:07 Panelists: I think that this can be explained that web developers still have to maintain some legacy code while transitioning to Python 3. And, many data analysts and machine learning specialists, they have joined the community just recently and went straight forward to Python 3.
37:25 Michael Kennedy: Yeah, I think you're probably right. This is a really great stat, because I was talking to somebody on Twitter talking about a colleague saying, hey, my colleague says we should be using Python 2 because almost everybody uses Python 2, there's this graph or graph that they found somewhere. And you have, it's not just not does everybody, it's not false that everybody uses Python 2, it's like really false, right? 75% of the people are using Python 3 and I think this number is going to rapidly increase as we get to January 1st, 2020 when Python 2 is no longer supported.
37:59 Panelists: True.
38:00 Michael Kennedy: So, I have here, I'll throw this out to all the consultants out there. So, I have a theory that October, November, and especially January, right around that time frame, so October, November, December of 2019 and January 2020, there are going to be a lot of executives and stuff going, wait a minute, are you telling me that our main platform for our website or our company actually no longer supported if there's bugs, we can't get them fixed? It's like, yeah, because you guys wouldn't let us rewrite it in Python 3 so here we are. And there's going it's going to be kind of like the Y2K. There's going to be all these consultants are like hey, I'm a specialist in converting from Python 2 to Python 3. And there's going to be a lot of work around then.
38:42 Panelists: There's a great comparison. Y2K Python.
38:46 Michael Kennedy: Exactly.
38:47 Panelists: Great job opportunity.
38:46 Michael Kennedy: Exactly. So, you guys, people out there listening, you have about a year to get really good and market yourself as a Python 2 to 3 expert. Yeah, it's pretty cool, that this is out there. Alright, so I guess we could talk a little bit about what languages and frameworks are popular? What did you find there?
39:05 Panelists: Yeah, the most popular languages used together with Python is Javascript, so 50% of Python developers use Javascript.
39:13 Michael Kennedy: You know that ties almost exactly to the 49% of web developers.
39:18 Panelists: True. And, another thing we also found that html and css used by 49%, so it's pretty close number. Which kind of confirms the same thing. Like, the same takeaway thing. Right.
39:30 Michael Kennedy: Uh-huh.
39:31 Panelists: We also had 39% of people using SQL, which kind of also related to web development. Bash and shell, 39%. Interestingly, 34% use C and C++.
39:45 Michael Kennedy: Hmm.
39:46 Panelists: 23% Java, 14% PHP, that's interesting. 11% C#.
39:53 Michael Kennedy: Yeah, that's pretty interesting. Certainly there's the web languages tie in there and the database stuff. What about like, popular libraries. Like Django versus Flask, those types of things?
40:05 Panelists: Yeah, we identified that Django is the most popular framework cause it's used by 41% of Python developers.
40:11 Michael Kennedy: It's not a big surprise, is it?
40:13 Panelists: It's not.
40:14 Michael Kennedy: So, one thing that's interesting about your analysis here is it's kind of interactive where you can see all the results or from a web developer view or a data science view. So you have Django at 41%, you have Numpy, Pandas at 39. So they're almost tied in popularity. But if you say, show that to me for web developers, it's 70%.
40:31 Panelists: Big difference.
40:32 Michael Kennedy: 70% of web developers Are using Django Just a little bit of Numpy. And data science, it's like you flip the graph, right? Quite a bit there.
40:39 Panelists: Which is kind of expected, yeah.
40:41 Michael Kennedy: Yeah, you also have an analysis of like, additional parallel technologies like Jupyter and Docker and Anaconda, so on in, it's similar stuff, right? Like, a web developer's more Docker, data science very much Jupyter.
40:54 Panelists: Yeah, when asked about those additional technologies. Jupyter notebook ranked highest at 31%. Which makes sense, given the large number of developers involved in data science.
41:04 Michael Kennedy: Yeah, I thought another interesting thing that you, you guys, spoke about was cloud platforms.
41:10 Panelists: Oh yeah, when asked...
41:10 Michael Kennedy: Which one do you use? When I think of AWS certainly they're the 800 pound gorilla in the room. The big thing is AWS, right?
41:21 Panelists: True.
41:21 Michael Kennedy: And you hear stats that Microsoft Azure is in second place, among all the other ones. But restrict that view to the Python community and that's not true, right?
41:30 Panelists: Mhm-hm, exactly. So Microsoft Azure ranked at 16%.
41:36 Michael Kennedy: What about some of the other ones? What was number two?
41:39 Panelists: Yeah, number two was Google App Engine. We had, Heroku with 26%, Digital Ocean, 23%, and fifth place is for Microsoft Azure.
41:51 Michael Kennedy: Yeah, yeah, that's pretty interesting. You can tell for sure it, AWS plays an important role because when they have an outage, it seems like the internet itself is broken. Why is everything down? Oh, AWS Virginia's down, I get it.
42:07 Panelists: This question about a popular cloud platforms, we didn't list a lot of other popular options. However people answered in the other option. Mhm-hm. So, they mentioned Linode, Python Anywhere, Open Shift and Open Stack. So these platforms are also quite popular among Python developers and we are going to include those options in the next iteration of our survey.
42:30 Michael Kennedy: Oh yeah, that'll be really interesting.
42:32 Panelists: Well I also feel like some of these have gotten more momentum in the most recent times. So I'm very interested to see what the next version of the survey's going to bring in comparison to this one.
42:42 Michael Kennedy: Yeah, you'll start to be able to develop these trends, which will actually be super interesting.
42:46 Panelists: I think that would be the most interesting part.
42:49 Michael Kennedy: Yeah, for sure.
42:49 Panelists: Yeah, exactly, it's going to be the main feature of- of the next survey.
42:53 Michael Kennedy: Yeah, and so, Dmitry, since you brought up Python Anywhere something that really surprised me a little while ago, well, quite a while ago, actually. I'd gone to python.org so many times and there's a little text box with like, code right there. And like, like a prompt looking button. But I didn't realize was that if you click that prompt, that will open an interactive shell on Python Anywhere, where you can just start typing Python.
43:20 Panelists: Yes, it will.
43:22 Michael Kennedy: That's pretty cool. I wonder how many people actually know about that.
43:25 Panelists: We probably don't tweet about it as much as we should. That's a good point.
43:28 Michael Kennedy: Yeah, it's cool. Like, I'm looking at like bash shell. Or actually, a Python REPL. Because it's right here on my...
43:34 Panelists: Exactly, 3.6.
43:37 Michael Kennedy: Very cool.
43:38 Panelists: How cool is that?
43:39 Michael Kennedy: Yeah, it's super cool. Alright, so let's see, maybe we could talk a little bit about tools and features. One of the questions you ask where here's a whole bunch of features that editors and other tools like linters and source control. How much do you use them or not use them? Do you want to touch on that, Dmitry?
43:53 Panelists: Yeah, sure. So, the top spots were occupied by code completion, code refactoring, writing unique tests, using virtual environments for writing projects. On the other hand, no skill database is Python profilers and code coverage tools for among the more rarely used tools and features.
44:12 Michael Kennedy: Yeah. So, I think it's too bad that like, the profiler and code coverage weren't use that much. But, you know, they're such a valuable thing. I think it's overlooked.
44:21 Panelists: Yeah. And by the way, the results that we've got in 2017, is very similar to the data we've got in 2016, so. The picture of the users of those tools doesn't change much. Seems to be.
44:35 Michael Kennedy: Okay, interesting. I should probably ask about this one. Dmitry, you'd probably be really more, more than happy to answer this one. What about editors?
44:44 Panelists: Well for editors and IDEs, we asked two similar questions. So one of them was what editor or IDE have you considered for using your Python development, with multiple answers allowed and a single answer option. A question, what is the main editor you use for your current Python development? So, PyCharm is the most popular tool.
45:07 Michael Kennedy: You had 32% of the people saying PyCharm.
45:09 Panelists: Yeah, 32% for PyCharm, for both editions, PyCharm Community and PyCharm Professional.
45:13 Michael Kennedy: I think that number would be massively skewed if like, you and JetBrains were the ones promoting this project. But like we talked about the sources, right? It wasn't really. And then what are some of the other editors people use?
45:28 Panelists: Some of the other popular editors are Sublime Text, Vim, Idle, Atom, and VS Code.
45:35 Michael Kennedy: Yeah, I suspect the Idle speaks a lot to the students. I feel like this survey was doing leading up to October 2017. I feel like VS Code will make a much higher appearance.
45:46 Panelists: I agree.
45:47 Michael Kennedy: Right
45:48 Panelists: We're really looking forward for an updated results this year.
45:52 Michael Kennedy: Yeah, I think the VS Code team has like, really put a lot of effort into Python. I ask this question every time at the end of the podcast. What editor people use, and I hear VS Code showing up more. A lot more than it used to.
46:04 Panelists: I see a lot of great positivity around it on Twitter as well.
46:08 Michael Kennedy: Yeah, for sure. Another thing that's interesting, I think, is to discuss how people get Python, right? Like, Ewa, I'm not sure what you can do, but if you could just get Microsoft to ship Python three with Windows, we'd all be in a good place. But that doesn't happen and so there's different ways in which, we might get Python on your system. So you asked that question and some of the options were like you go to python.org and you click the download button. Or use something like homebrew or aptitude on the Linux, or Anaconda.
46:39 Panelists: Exactly.
46:41 Michael Kennedy: Surprisingly, many people build it from source.
46:44 Panelists: 7%, yeah.
46:45 Michael Kennedy: Dog gone.
46:45 Panelists: The true hackers.
46:48 Michael Kennedy: That's a lot, 7%, right? I mean, I know people do it, but that's- that seems like a lot. I guess we had 39% go and download it from python.org. 31% get it through some kind of package manager in their OS. And 15% on Anaconda.
47:04 Panelists: Correct.
47:05 Michael Kennedy: Those are pretty interesting things to know, I think.
47:07 Panelists: Well, it makes me happy that, you know, one of main infrastructure items that the PSF helps maintain for the community is python.org. So I'm very happy to see that it is being used worldwide.
47:19 Michael Kennedy: Yeah, absolutely. I guess maybe the last question directly we can touch on is operating systems. Because, to me, if I had to be somehow stripped of my knowledge and just sort of walked around conferences or something. I would feel like well, the majority of the Python world runs on Mac, right?
47:39 Panelists: I would think so.
47:40 Michael Kennedy: Cause you see like everyone's carrying around a Mac. They got all their stickers on it. They're all cool. That's not exactly where, how it breaks down, is it?
47:47 Panelists: Yeah, exactly. So it feels like a very limited view. Well, exactly. That's why we think the survey is more of a global outlook on how things are versus the way that we see them, so.
48:02 Michael Kennedy: Yeah, so your survey found that 49% of the people use Windows. 19% Linux, 15% Mac, and then other people, other stuff. I think it's pretty interesting and it really, you know, I try to drive the point home that Windows is super important in the Python space. And, I know it gets a lot of love. But also, I feel like, periodically, packages people build, they're not tested on Windows, or they won't install on Windows. Or if they do, they're hard, unlike Mac and Linux, it might just be pip install. But on Windows, you've got to install this other weird thing to make it work, you know. So, I feel like Windows kind of gets lost. And people thinking about where people are using Python. But in fact, it's more than two times as popular as anything else. Yeah, worth thinking about. Alright, well I think that's probably it for the topics of the survey directly I wanted to cover. Well done, you guys. That was really nice.
48:51 Panelists: Yeah, thank you. Thank you.
48:53 Michael Kennedy: People want to give you feedback. How do they do it?
48:56 Panelists: They should email us, we have a mailing list and we also have a Github repo. So the mailing list is surveys@python.org. And that will go to Dmitry and I. And the Github link, Michael, can you include that in the links below? Awesome.
49:12 Michael Kennedy: Yeah, I'll definitely include it. So the link to the Github repo, which has an issue tracker, where people can submit issues, that's cool. And then also...
49:20 Panelists: Also feedback. Not just issues, but if they have any ideas for improvements or you know, let's say, they know of a community that wants to get involved for next time. They could definitely post it there.
49:31 Michael Kennedy: Yeah, interesting. What about the raw data? Like, Dmitry's team took this and came up with, what I said was a really beautiful result. But you know, it is an interpretation of the data somewhat, right?
49:42 Panelists: Yes. Exactly.
49:44 Michael Kennedy: Can people do their own interpreting?
49:45 Panelists: Yeah, absolutely. So, along with a report, we also share the raw data in csv format.
49:51 Michael Kennedy: Okay, cool. And that's, you can find that off the link off the survey results.
49:55 Panelists: Yeah, you go by the link to the report. Down like, in the bottom of the page, you can find a link to the raw data. Exactly.
50:02 Michael Kennedy: Yeah, I won't link to it directly cause you have a couple of things like, please consider this as you look at this data. A lot of people read that first, but yeah, nice.
50:09 Panelists: Yeah, unfortunately, last year, we didn't provide Python scripts, which people can use to, like, bootstrap their analysis with this raw data. Yeah, that was a suggestion, mhm-hm. Yeah, that was a suggestion after our talk. So now we know to do-
50:27 Michael Kennedy: So maybe a Jupyter notebook that does the analysis or something, right?
50:29 Panelists: Exactly. Exactly.
50:31 Michael Kennedy: Yeah, that'd be really sweet to have. Alright, so, you're doing this again?
50:35 Panelists: We are. Absolutely! Very soon! Not sure when you're going to be airing the episode but we're planning on launching in October.
50:42 Michael Kennedy: Okay, so yeah, this episode will be out really soon. So people should have some time to get ready and prepare for its arrival, but sometime in October. Do you actually have a date, when you're letting out? Or is it just general time frame?
50:53 Panelists: We're going to start the survey in the beginning of October. We don't have an exact date, but stayed tuned for the beginning of October.
51:00 Michael Kennedy: Alright, cool. And I'll try to spread the word again to all of the listeners when that happens.
51:04 Panelists: Much appreciated.
51:05 Michael Kennedy: Yeah, well, I think, think we probably should leave it there. We've covered a lot of stuff and I really think this is a great service you all are doing for the community and I'm like you, I think the trends are going to be as relevant as the actual numbers.
51:19 Panelists: Yeah, I agree, and I'm very happy that it's gaining some momentum and interest. And thank you so much for inviting us to talk about it.
51:26 Michael Kennedy: Yeah, absolutely. So, final call to action, maybe people could spread the word like, when you hear that survey's out, tell everyone you know. Encourage them to drop it
51:35 Panelists: Exactly.
51:36 Michael Kennedy: in your internal companies Slack channel, things like that, right?
51:39 Panelists: Yeah, exactly. We encourage everyone to take part in the survey and promote it to everyone. Post it on social media, exactly. Take it it your user group.
51:48 Michael Kennedy: Yeah, that's a good idea. Take it to the user group. Very cool, alright, Ewa, Dmitry, it's been great to have you on the show. Thanks for doing this project and sharing it with everyone.
51:55 Panelists: Thank you, and thank you for having us. Yeah, thanks Michael.
51:58 Michael Kennedy: You bet, bye.
51:59 Panelists: Bye. Bye.
52:02 Michael Kennedy: This has been another episode of Talk Python to Me. Our guests on this episode have been Ewa Jodlowska and Dmitry Filippov. And it has been brought to you by Linode and brilliant.org. Linode is bullet proof hosting for whatever you're building with Python. Get four months free at talkpython.fm/linode. That's linode. Brilliant.org wants to help you level up your math and science through fun, guided problem solving. Get started for free at talkpython.fm/brilliant. Want to level up your Python? If you're just getting started, try my Python Jumpstart by Building 10 Apps. Or our brand new 100 Days of Code in Python. And if you're interested in more than one course, be sure to check out the everything bundle. It's like a subscription that never expires. Be sure to subscribe to the show. Open your favorite podcatcher and search for python. We should be right at the top. You can also find the iTunes feed at /itunes, Google play feed at /play, and direct rss feed at /rss on talkpython.fm. This is your host, Michael Kennedy. Thanks so much for listening. I really appreciate it. Now, get out there and write some Python code.