WEBVTT 00:00:00.001 --> 00:00:03.820 The largest machine ever built is the Large Hadron Collider at CERN. 00:00:03.820 --> 00:00:10.100 Its primary goal was the discovery of the Higgs boson, the fundamental particle which gives all objects mass. 00:00:10.100 --> 00:00:18.140 The LHC team actually achieved this audacious goal in 2012, winning them the Nobel Prize in physics in the process. 00:00:18.140 --> 00:00:25.220 Today on Talk Python To Me, Kyle Cranmer is here to share how Python was at the core of this amazing achievement. 00:00:25.760 --> 00:00:31.640 This is episode number 29, recorded Thursday, September 24th, 2015. 00:00:51.520 --> 00:01:05.240 Welcome to Talk Python To Me, a weekly podcast on Python, the language, the libraries, the ecosystem, and the personalities. 00:01:05.240 --> 00:01:09.360 This is your host, Michael Kennedy. Follow me on Twitter where I'm @mkennedy. 00:01:09.360 --> 00:01:15.820 Keep up with the show and listen to past episodes at talkpython.fm and follow the show on Twitter via at Talk Python. 00:01:16.520 --> 00:01:19.780 This episode is brought to you by Hired and CodeShip. 00:01:19.780 --> 00:01:25.120 Thank them for supporting the show on Twitter via at Hired underscore HQ and at CodeShip. 00:01:25.120 --> 00:01:31.060 I don't have much news to share this week, but I am both honored and thrilled to bring you this episode, 00:01:31.060 --> 00:01:32.800 and I can't wait for you to listen to it. 00:01:32.800 --> 00:01:34.940 So, let's get right to the interview. 00:01:34.940 --> 00:01:37.160 Let me introduce Kyle. 00:01:37.160 --> 00:01:44.340 Kyle Cranmer is an American physicist and professor at New York University at the Center for Cosmology and Particle Physicists 00:01:44.340 --> 00:01:48.340 and affiliated faculty member at NYU's Center for Data Science. 00:01:48.340 --> 00:01:54.480 He is an experimental particle physicist working primarily on the Large Hadron Collider based in Geneva, Switzerland. 00:01:54.480 --> 00:02:00.540 Cranmer popularized a collaborative statistical modeling approach and developed statistical modeling, 00:02:00.540 --> 00:02:06.060 which was used extensively for the discovery of the Higgs boson at the LHC in July 2012. 00:02:06.060 --> 00:02:08.300 Kyle, welcome to the show. 00:02:08.300 --> 00:02:10.180 Thank you. Thank you. It's a pleasure to be here. 00:02:10.780 --> 00:02:17.340 Yeah, we have some amazing science and programming to talk about today, so I'm really excited to dig into all these topics with you. 00:02:17.340 --> 00:02:20.020 Yeah, no, I'm excited to see where it goes. 00:02:20.020 --> 00:02:21.600 Yeah, for sure. 00:02:21.600 --> 00:02:26.640 So, let's, you know, we're going to talk about the Large Hadron Collider, 00:02:26.640 --> 00:02:31.540 about using Python for scientific research and all those sorts of things, 00:02:31.580 --> 00:02:33.620 as well as some other cool projects that you've got going on. 00:02:33.620 --> 00:02:41.500 But people like to know how folks like you and your position kind of got started and they like to hear the background. 00:02:41.500 --> 00:02:47.580 So, maybe we could start with, you know, what got you interested in physics and what got you interested in programming and how do you get to where you are? 00:02:48.020 --> 00:02:53.200 I've been interested in, you know, in physics, you know, since I was a kid, not really knowing that that's what it was called. 00:02:53.200 --> 00:03:00.860 But later on, you know, I think, I guess it was in high school is when I really realized that it was physics that I wanted to do. 00:03:01.460 --> 00:03:09.420 I grew up in Arkansas and, you know, Arkansas is not exactly known for, like, leading the tide of physicists and computer scientists in the world. 00:03:09.420 --> 00:03:15.300 But they had started a special math and science high school that it was public school, but you actually lived there. 00:03:15.300 --> 00:03:21.600 And when I was there, I was just surrounded by all sorts of people, kind of the nerds and geeks of Arkansas. 00:03:21.600 --> 00:03:23.260 And it was really a special time. 00:03:23.260 --> 00:03:29.980 So, during that time, I, you know, got even more into physics, but it's also, that's when I was first exposed to serious programming. 00:03:30.880 --> 00:03:41.940 So, actually, even Python, like in 95, actually 94, I guess, I had a friend that was into early web things and he was playing with Zope and the Zope, you know, object database. 00:03:41.940 --> 00:03:49.560 And so, I started working with him and did some early web projects and that was kind of my first exposure to Python. 00:03:49.560 --> 00:03:51.320 So, that was a long time ago. 00:03:51.320 --> 00:03:58.060 And then that's, it's funny how much those experiences kind of keep being revisited today. 00:03:58.060 --> 00:04:00.620 Yeah, I'm sure you keep coming back to that. 00:04:00.780 --> 00:04:06.780 You know, basically, programming these days seems like a required skill to be a physicist. 00:04:06.780 --> 00:04:14.060 Yeah, no, well, it depends on what you do, but definitely for what we do, programming is a required skill. 00:04:14.060 --> 00:04:22.400 And unfortunately, it can, you know, for people that don't have those strengths, it really takes away from their ability to try to do the physics that they want to do. 00:04:23.060 --> 00:04:31.020 So, you know, so for incoming graduate students, you usually see a pretty big divide between people that are, have some programming skills and don't. 00:04:31.020 --> 00:04:36.800 And usually, the people that don't will catch up a little bit later, but, you know, you lose time and that's unfortunate. 00:04:36.800 --> 00:04:37.500 Right. 00:04:37.560 --> 00:04:43.920 I'm sure it's like a huge scramble, like, oh my gosh, I got to learn all this programming stuff too because, you know, we have projects or whatever, right? 00:04:44.400 --> 00:04:44.940 Right, right. 00:04:44.940 --> 00:04:56.180 It also colors a lot of the flavor about how we approach computing because somehow you have this enormous computing problem that you need to deal with and you would like to do it as nicely as possible. 00:04:56.180 --> 00:05:02.720 But it also can't be too fancy or the bulk of the physicists might not be able to understand what's going on. 00:05:02.720 --> 00:05:10.860 You have older physicists from the Fortran days and you have younger physicists that maybe never took any, you know, programming courses, like any serious programming courses. 00:05:10.860 --> 00:05:16.800 So things have to be somehow kept simple but still work for the difficult problems. 00:05:16.800 --> 00:05:18.800 And so it's a difficult balance to strike. 00:05:18.800 --> 00:05:19.860 Yeah, I'm sure. 00:05:20.660 --> 00:05:24.820 So let's talk a little bit about what you guys are doing at the Large Hadron Collider. 00:05:24.820 --> 00:05:30.000 And first of all, you know, congratulations on the Higgs boson discovery. 00:05:30.000 --> 00:05:30.580 That's amazing. 00:05:30.580 --> 00:05:31.720 Oh, thank you. 00:05:31.720 --> 00:05:35.580 No, it was, yeah, many, many years of work. 00:05:35.580 --> 00:05:39.480 And when it finally came, it was a huge treat. 00:05:39.480 --> 00:05:39.940 I don't know. 00:05:39.940 --> 00:05:44.500 It's funny to have such a big thing like that happen fairly early in your career. 00:05:44.500 --> 00:05:47.180 It's like, now what? 00:05:47.180 --> 00:05:47.960 Yeah. 00:05:47.960 --> 00:05:53.800 So, but yeah, so at the LHC, you know, we have this huge collider that's in Switzerland. 00:05:53.800 --> 00:05:57.220 It's about 17 miles around and underground. 00:05:57.220 --> 00:06:03.580 There are all these super connecting magnets that help protons go, you know, bend them in a circle at essentially the speed of light. 00:06:03.580 --> 00:06:06.320 And they're colliding together all the time. 00:06:06.420 --> 00:06:11.480 And they smack into each other and they make, you know, a lot of new particles that come flying out. 00:06:11.480 --> 00:06:13.660 And they hit our detector. 00:06:13.660 --> 00:06:17.280 And our detector, you can think of sort of like a digital camera. 00:06:17.280 --> 00:06:19.100 You know, it's like basically a bunch of pixels. 00:06:19.100 --> 00:06:22.860 And the particles smack into it and you get an image. 00:06:22.860 --> 00:06:24.500 But it's a 3D image. 00:06:24.500 --> 00:06:25.680 So it's a 3D detector. 00:06:25.980 --> 00:06:29.380 And the detector is like the size of a 12-story building. 00:06:29.380 --> 00:06:30.580 So, yeah. 00:06:30.580 --> 00:06:43.520 I think that, you know, when you just hear about particle colliders and especially LHC, you have maybe this idea of like a tube where things are shooting around. 00:06:43.520 --> 00:06:45.080 And, you know, how big does the tube have to be? 00:06:45.080 --> 00:06:45.860 Not that big. 00:06:46.120 --> 00:06:53.260 But the actual experiments, the collectors, I was blown away when I learned about how big they are. 00:06:53.260 --> 00:06:54.800 Like you said, 12 stories. 00:06:54.800 --> 00:06:55.640 These things are huge. 00:06:55.640 --> 00:06:56.280 Yeah. 00:06:56.280 --> 00:06:56.860 No, they are. 00:06:56.860 --> 00:07:04.960 And the range of scales is pretty crazy because we have to be able to track where these particles go very precisely. 00:07:04.960 --> 00:07:09.140 So, like, close to where they interact, you know, we're measuring things at the, like, micron level. 00:07:09.680 --> 00:07:14.480 And then they fly out over, like, the size of a building. 00:07:14.480 --> 00:07:18.260 And we're still measuring where they're going at this very precise level. 00:07:18.260 --> 00:07:20.920 But, you know, it's just this gargantuan thing. 00:07:20.920 --> 00:07:24.280 So you have to align it properly and all sorts of challenges there. 00:07:24.280 --> 00:07:30.820 And there are about 100 million, you know, well, a few hundred million electronic readouts coming out of this beast. 00:07:30.820 --> 00:07:35.580 So, you know, it's like a 100-megapixel camera or something like that. 00:07:35.580 --> 00:07:38.760 And we're taking 40 million photos every second. 00:07:38.760 --> 00:07:41.300 That's a stunning amount of data. 00:07:41.300 --> 00:07:42.800 That is a stunning amount of data. 00:07:42.800 --> 00:07:56.140 And so if you, you know, we have to slap special electronics, like, straight onto the detector to be able to start preprocessing it and compressing it and sort of, you know, coming up with some way to deal with the data volume. 00:07:56.140 --> 00:08:04.340 Because, you know, it's something, it's, you know, there are just totally staggering numbers about the data flow that's coming straight out of the detector. 00:08:04.340 --> 00:08:06.640 So how do you capture and store that? 00:08:06.700 --> 00:08:11.900 Do you store that, like, on hardware right on, like, Atlas in the machines? 00:08:11.900 --> 00:08:15.400 Or do you, like, get that into, like, a cluster of servers? 00:08:15.400 --> 00:08:16.440 Or what happens? 00:08:16.440 --> 00:08:17.440 Right, right. 00:08:17.540 --> 00:08:27.320 So we have a kind of a hierarchical online real-time system for tossing away, you know, the majority of the data. 00:08:27.320 --> 00:08:35.000 So we have to actually, we write algorithms that have to look at the data and real-time decide, does this look interesting or not? 00:08:35.000 --> 00:08:41.500 And so we go from the sort of 40 million a second through this, like, three levels of filtering down. 00:08:41.500 --> 00:08:47.300 And then we get to the point that we save something like a few hundred of these collisions every second. 00:08:48.000 --> 00:08:55.260 And that turns into, you know, several petabytes a year of data that we actually analyze later. 00:08:55.260 --> 00:08:56.660 That's amazing. 00:08:56.660 --> 00:08:59.800 It's got to be a little stressful to work on that initial filtering algorithm. 00:08:59.800 --> 00:09:03.240 Because what if you threw away the Higgs boson before you discovered it, right? 00:09:03.240 --> 00:09:04.140 That's right. 00:09:04.140 --> 00:09:04.320 Yeah. 00:09:04.320 --> 00:09:08.620 No, people, we always worry that we're, you know, kind of throwing the baby out with the bathwater. 00:09:09.580 --> 00:09:13.620 And sorry about the living in New York here. 00:09:13.620 --> 00:09:14.740 Yeah, no worries. 00:09:14.740 --> 00:09:18.540 So the, yeah, we call that thing the trigger. 00:09:18.540 --> 00:09:21.700 And, you know, that's something that I worked on a bit. 00:09:21.700 --> 00:09:29.000 It's true that, like, if we don't find anything else in this next run of the LHC, you know, a lot of people will think exactly that. 00:09:29.000 --> 00:09:33.960 That maybe, you know, the way the trigger was configured, we were throwing away the interesting stuff. 00:09:33.960 --> 00:09:36.000 But luckily, we're not stuck to that. 00:09:36.000 --> 00:09:38.360 You know, we can go and we can change it and things like that. 00:09:38.360 --> 00:09:40.200 But that is the worry. 00:09:40.200 --> 00:09:41.120 Sure. 00:09:41.120 --> 00:09:41.500 Yeah. 00:09:41.500 --> 00:09:45.540 I mean, you still have the time spent and the energy and all that, right? 00:09:45.540 --> 00:09:46.140 Yeah. 00:09:46.140 --> 00:09:46.660 Yeah, sure. 00:09:46.660 --> 00:09:48.560 You can rerun it, of course. 00:09:48.560 --> 00:09:54.180 But, you know, you got to, I suspect time is a valuable thing on that machine. 00:09:54.180 --> 00:09:55.640 Yeah. 00:09:55.640 --> 00:09:56.280 No, for sure. 00:09:56.280 --> 00:09:57.540 It's expensive to run. 00:09:57.540 --> 00:09:59.300 So, absolutely. 00:09:59.300 --> 00:10:00.740 Yeah. 00:10:00.740 --> 00:10:07.660 So, you know, I have a lot of listeners who are scientists and physicists and data science and so on. 00:10:07.660 --> 00:10:12.180 But a lot of them who are probably not. 00:10:12.180 --> 00:10:23.200 And so, I wanted to make a movie recommendation and a book recommendation just for people to, you know, if they want to kind of set the stage and learn the background, you know, as part of this whole thing we're talking about. 00:10:23.200 --> 00:10:26.780 I wanted to recommend the Particle Fever documentary. 00:10:26.780 --> 00:10:27.680 Have you seen this? 00:10:28.380 --> 00:10:28.880 Yeah, yeah. 00:10:28.880 --> 00:10:30.780 I actually have a credit in that movie. 00:10:30.780 --> 00:10:35.180 I worked with them quite a bit. 00:10:35.180 --> 00:10:45.020 And at one point there was a scene that was shot in my office but ended up having to cut it because it didn't really fit well with the, you know, it was a good choice. 00:10:45.120 --> 00:10:46.300 But it was painful. 00:10:46.300 --> 00:10:47.780 But they were nice. 00:10:47.780 --> 00:11:00.740 I worked with them a fair amount and got to go to the, like, you know, the opening in Sheffield at a documentary film festival and hang out with the producers and the whole crew. 00:11:00.740 --> 00:11:03.660 But it's a great film. 00:11:04.020 --> 00:11:08.340 I think it definitely, it's good for a non-physics audience also. 00:11:08.340 --> 00:11:09.500 It's not a technical film. 00:11:09.500 --> 00:11:18.260 It just basically captures what it's like to be inside one of these experiments and the sort of stress and the, you know, the drama associated to it. 00:11:18.260 --> 00:11:21.520 I think it's really one of the best science documentaries ever. 00:11:21.520 --> 00:11:22.900 I absolutely agree with you. 00:11:22.900 --> 00:11:30.540 I think it really captures the excitement, the imagination, the drama in a way that, you know, anybody could appreciate. 00:11:30.540 --> 00:11:32.440 And so, I definitely recommend people watch that. 00:11:32.600 --> 00:11:36.560 It's available for streaming on Netflix and iTunes and other places. 00:11:36.560 --> 00:11:44.980 And then the other thing is the book called Present at the Creation, Discovering the Higgs Boson by Amir Axel. 00:11:44.980 --> 00:11:45.960 I messed up his name. 00:11:45.960 --> 00:11:47.400 But that's also really good. 00:11:47.400 --> 00:11:51.580 So, people who are out there and want to learn more about what we're talking about, I think I recommend those. 00:11:51.580 --> 00:11:52.920 Okay, great. 00:11:52.920 --> 00:11:54.640 Yeah, I actually haven't read that second book. 00:11:54.640 --> 00:11:58.000 Yeah, I really enjoyed that book as well. 00:11:58.000 --> 00:12:00.420 It predates the Higgs Boson. 00:12:00.420 --> 00:12:02.160 So, it's like a lot of anticipation. 00:12:02.480 --> 00:12:02.800 So, that's cool. 00:12:02.800 --> 00:12:03.020 I see. 00:12:03.020 --> 00:12:03.940 Okay, great. 00:12:03.940 --> 00:12:11.320 Maybe we could talk a little bit about, like, the really big picture of software at the LHC. 00:12:11.320 --> 00:12:14.480 Because there's not just one team and there's not just one experiment. 00:12:14.480 --> 00:12:15.620 There's how many collectors? 00:12:15.620 --> 00:12:16.700 Are there seven collectors? 00:12:16.700 --> 00:12:17.920 Right. 00:12:17.920 --> 00:12:23.140 So, well, there are two really big kind of multipurpose particle detectors, Atlas and CMS. 00:12:23.140 --> 00:12:24.260 And I'm on Atlas. 00:12:24.640 --> 00:12:31.440 And those two experiments have, you know, in the neighborhood, a little bit more than 3,000 physicists working on them. 00:12:32.060 --> 00:12:35.340 So, you know, so it's a, there are big groups of people. 00:12:35.340 --> 00:12:45.620 And then there are two other experiments that are, you know, slightly smaller in scale, but they do, you know, and slightly more specialized in terms of the physics that they do. 00:12:45.940 --> 00:12:51.020 And then there are several other smaller dedicated experiments that are quite a bit smaller. 00:12:51.580 --> 00:12:55.820 And so, I don't, you know, it depends on how you count a little bit. 00:12:55.820 --> 00:13:05.520 But usually, you know, there's sort of the two big multipurpose detectors and two other more specialized ones that are the dominant, like, LHC experiments. 00:13:05.520 --> 00:13:06.560 Okay, cool. 00:13:06.560 --> 00:13:15.260 And so, maybe from, like, the higher level or larger scale, like, the thing that actually runs the machine down into the experiments, down into more, like, the data processing details. 00:13:15.400 --> 00:13:18.680 Could you give us a picture, like, what the software looks like there, what you guys are doing? 00:13:18.680 --> 00:13:19.820 Sure, yeah. 00:13:19.820 --> 00:13:23.600 So, I mean, it's mainly, you know, we have a whole bunch of collisions. 00:13:23.600 --> 00:13:31.180 And each collision, you know, if you think of what this metaphor of it being like an image, you know, it's like a pipeline for doing a bunch of image processing, you know. 00:13:31.180 --> 00:13:39.360 And you're looking for, you're trying to find the collisions that, you know, maybe have evidence of some new particle. 00:13:39.360 --> 00:13:50.400 So, you have lots of teams of people that are looking for different things, and each of those teams will develop a little pipeline to process the data to try to, you know, to search for what they want. 00:13:50.400 --> 00:14:00.120 Also, to put it into perspective a little bit, we had a quadrillion, you know, a couple quadrillion collisions total at the LHC. 00:14:00.120 --> 00:14:08.180 And when we discovered the Higgs, it was, you know, of the order of 100 or 1,000 of those collisions that were the interesting ones. 00:14:08.460 --> 00:14:11.180 So, it's a huge needle in a haystack problem. 00:14:11.180 --> 00:14:17.420 But it's also not really like a data mining kind of just generally looking for something weird in the data. 00:14:17.420 --> 00:14:30.080 We have theories that tell us, you know, what to look for, which is good because there's such small little deviations in the data that it would be basically impossible to find if you didn't have a good guide. 00:14:30.320 --> 00:14:41.280 And then this processing chain, because there's so much data and performance is such an issue, most of it, well, several years ago, the decision was to write most of the software in C++. 00:14:41.280 --> 00:14:44.880 You know, C++ has also evolved a ton during the time. 00:14:44.880 --> 00:14:49.900 Are you using like, are people using like C++ 11 and those types of things? 00:14:50.140 --> 00:14:50.640 Right. 00:14:50.640 --> 00:14:59.880 So, the different experiments kind of, you know, move to these new, you know, new standards and new computing technologies kind of at different paces. 00:14:59.880 --> 00:15:01.540 There's a lot of worry. 00:15:01.540 --> 00:15:07.800 You know, it's generally a pretty conservative attitude, you know, but we are making those kinds of transitions. 00:15:08.580 --> 00:15:14.040 But, you know, you just, it has to go through a lot of vetting before we make a big jump like that. 00:15:14.040 --> 00:15:33.520 We also usually have a very homogeneous computing environment in terms of like operating systems and things like that because, you know, we run into issues where, you know, you don't want to have to be worrying about like floating point arithmetic in your kernel or something, you know, when you're, because it's, so, so we just tried it. 00:15:33.520 --> 00:15:42.420 Yeah, so it's a little bit funny, you know, that CERN was responsible for sort of developing the web browser, right, you know, and HTML and things like that. 00:15:42.420 --> 00:15:45.700 And so, they had this huge win of, you know, where the web was born. 00:15:45.700 --> 00:15:51.420 And then that was followed by this idea of like, okay, we had the web, now we're going to have grid computing. 00:15:51.420 --> 00:15:53.760 And there was a lot of money poured into it. 00:15:53.760 --> 00:15:59.160 And the promise of the grid basically turned into what has happened with the cloud. 00:15:59.560 --> 00:16:09.400 And, but in, you know, and then within IMG physics, we do have the grid, but it's kind of like a huge global, you know, batch system in some sense. 00:16:09.400 --> 00:16:20.020 So, it tends to be, you know, more uniform and things like that than what, you know, at first people were working really hard to be able to work over very heterogeneous computing environments. 00:16:20.020 --> 00:16:24.320 But, you know, that all evolved over, you know, more than a decade. 00:16:24.320 --> 00:16:25.960 Yeah, I'm sure. 00:16:25.960 --> 00:16:29.740 I used to do a lot of work in sort of scientific computing and visualization. 00:16:29.740 --> 00:16:35.920 And it's super hard to do reproducibility and checking stuff, you know. 00:16:35.920 --> 00:16:36.620 Right. 00:16:36.620 --> 00:16:41.940 If you've got a sufficiently complicated series of mathematical steps, you can apply to something. 00:16:42.180 --> 00:16:46.680 You know, like, if it's so complicated, how do you know when you're right or not? 00:16:46.680 --> 00:16:46.880 Right. 00:16:46.880 --> 00:16:52.420 You know, how do you know when you're discovering something new versus, oh, it's like I expected or whatever, right? 00:16:52.420 --> 00:16:53.020 Right. 00:16:53.020 --> 00:17:01.880 Well, we're working a lot right now on trying to address the sort of reproducibility, you know, issues kind of specific and the challenges associated to our field. 00:17:02.460 --> 00:17:05.960 And there are a lot of challenges because there's so much data and software is very complicated. 00:17:05.960 --> 00:17:06.940 Yeah. 00:17:06.940 --> 00:17:13.640 So the core algorithms tend to all be C++, but they're, you know, they're organized into lots of, you know, lots of different tools. 00:17:13.640 --> 00:17:19.480 And, you know, you have a way of kind of composing this pipeline between different processing algorithms. 00:17:19.480 --> 00:17:29.960 And in the end, the configuration of that thing is such a beast that that's the first place where you see Python happening is that we have a way of kind of, you know, doing introspection on all of the tools. 00:17:30.040 --> 00:17:33.820 And then we just represent their configuration in terms of Python objects. 00:17:33.820 --> 00:17:40.400 And then there's a whole separate layer of, you know, of computing, which I mean, of programming, which is just essentially the configuration. 00:17:40.400 --> 00:17:55.600 And that includes both this trigger, that online system that's tossing out the data, as well as the, you know, the people that are analyzing the data, how they, you know, configure all these tools to be able to process the kajillions of events into something that's more manageable. 00:17:55.600 --> 00:17:57.980 Now, that sounds really interesting. 00:17:57.980 --> 00:18:05.860 When I was doing some research, it seemed like one of the major pieces used in Atlas was this thing called Athena. 00:18:05.860 --> 00:18:07.080 Right. 00:18:07.080 --> 00:18:07.860 That's right. 00:18:07.860 --> 00:18:16.880 That's the kind of name of the C++ framework that we use that also includes the way that it builds the Python bindings for configuring all the tools. 00:18:17.580 --> 00:18:24.620 And yeah, so that's, yeah, I've spent more hours than I'd like to admit doing programming in that framework. 00:18:24.620 --> 00:18:35.920 But then what's also interesting, I think a lot of your audience will find interesting, is that once you've used that huge, heavyweight data processing pipeline, usually you get to something quite a bit smaller. 00:18:36.260 --> 00:18:41.000 And that's where a lot of the more interactive and exploratory part of the data analysis happens. 00:18:41.000 --> 00:18:47.900 And at that stage, a lot of people, well, people stop using things like Athena for the most part. 00:18:48.140 --> 00:18:53.440 And that's where you start using, see people using Python a lot more in terms of data analysis. 00:18:53.440 --> 00:19:00.340 And so it's an interesting transition because people are always arguing about where do you make that swap, you know? 00:19:00.340 --> 00:19:01.360 Sure. 00:19:01.360 --> 00:19:01.920 Yeah. 00:19:02.640 --> 00:19:05.140 I suspect you guys probably do a lot of IPython. 00:19:05.140 --> 00:19:05.980 Is that true? 00:19:05.980 --> 00:19:08.700 Well, you would think that more people would. 00:19:08.700 --> 00:19:26.420 I guess part of it is that it's still, even at that stage, you still have so much data to process that the kinds of things that people end up wanting to do are, you know, well suited to having like, you know, programs that, you know, that look really like programs that run. 00:19:26.420 --> 00:19:33.140 And they might be Python based, but, you know, you kind of sort of batch systemy, you run over this thing, and then you get some results and look at them. 00:19:33.140 --> 00:19:35.940 There are times when you're doing something very interactive. 00:19:35.940 --> 00:19:51.920 And so years ago, the team at CERN that makes this tool called Root, which is like kind of the dominant data analysis package in high energy physics, came up with something like an interpreter because you want to sit there and have this feedback loop, right? 00:19:51.920 --> 00:19:55.740 You know, like the end where you can, you know, type commands, see plots. 00:19:55.740 --> 00:19:58.560 And that was actually done amazingly. 00:19:58.560 --> 00:20:01.940 They wrote a C++ interpreter many, many years ago. 00:20:01.940 --> 00:20:08.760 And so you actually write these commands in C++, and then they're interpreted and executed on the fly. 00:20:08.760 --> 00:20:11.120 That's actually pretty interesting by itself, isn't it? 00:20:11.120 --> 00:20:12.260 It is interesting. 00:20:12.260 --> 00:20:17.700 It, of course, had all sorts of issues, and C++ wasn't really meant for doing that, but it worked practically. 00:20:18.360 --> 00:20:27.960 And now they've gone through and they have a much heavier duty version of this interpreter that's based on, you know, Cling and more modern, like, compiling, compiler technologies and things. 00:20:27.960 --> 00:20:32.660 But Python, obviously, is another way to go with that, which is nice. 00:20:42.740 --> 00:20:45.460 This episode is brought to you by Hired. 00:20:45.460 --> 00:20:51.940 Hired is a two-sided, curated marketplace that connects the world's knowledge workers to the best opportunities. 00:20:51.940 --> 00:21:01.100 Each offer you receive has salary and equity presented right up front, and you can view the offers to accept or reject them before you even talk to the company. 00:21:01.100 --> 00:21:07.460 Typically, candidates receive five or more offers in just the first week, and there are no obligations, ever. 00:21:07.460 --> 00:21:09.540 Sounds pretty awesome, doesn't it? 00:21:09.540 --> 00:21:11.600 Well, did I mention there's a signing bonus? 00:21:11.600 --> 00:21:15.680 Everyone who accepts a job from Hired gets a $2,000 signing bonus. 00:21:15.680 --> 00:21:20.040 And as Talk Python listeners, it gets way sweeter. 00:21:20.040 --> 00:21:27.600 Use the link Hired.com slash Talk Python To Me, and Hired will double the signing bonus to $4,000. 00:21:27.600 --> 00:21:29.300 Opportunity's knocking. 00:21:29.300 --> 00:21:32.920 Visit Hired.com slash Talk Python To Me and answer the call. 00:21:32.920 --> 00:21:45.300 So people started moving to the Python. 00:21:45.300 --> 00:21:52.220 Well, some people started moving to the Python way of doing things, I don't know, whatever, you know, eight years ago or something like that. 00:21:52.220 --> 00:21:55.760 But the field is kind of split between, you know, which way. 00:21:55.760 --> 00:21:59.640 And then since then, like, things like IPython and the IPython notebook have come around. 00:21:59.640 --> 00:22:03.980 I think that that's great, especially from the point of view of this, like, reproducibility. 00:22:03.980 --> 00:22:07.020 So we're working now that we give tons of talks. 00:22:07.020 --> 00:22:09.220 If you go to CERN, we have this agenda system. 00:22:09.220 --> 00:22:15.040 And you can see that there are, like, hundreds of thousands of presentations happening within these experiments every year. 00:22:15.040 --> 00:22:16.280 That's excellent. 00:22:16.680 --> 00:22:23.340 Yeah, so one of the things, but they're always, like, PowerPoint or, you know, Keynote or whatever, or LaTeX-based PDF presentations. 00:22:23.340 --> 00:22:32.720 And you read about what someone's doing, but it's not very handy for, like, trying to have reproducibility or for another graduate student to pick up where someone left off. 00:22:32.720 --> 00:22:45.600 So we have this effort now to try to make it so that the agenda system can, you know, can basically display notebooks directly so people can upload their IPython notebook directly and visualize it. 00:22:45.600 --> 00:22:49.480 And then if someone else thinks it's interesting, they can, you know, download it and execute it. 00:22:49.940 --> 00:23:04.880 There are efforts about trying to make it so that the whole computing environment associated to that notebook can be, you know, packaged up so that, because they usually aren't just standalone Python, Python notebooks with, like, SciPy dependency. 00:23:04.880 --> 00:23:06.140 They have a bunch of dependencies. 00:23:06.140 --> 00:23:11.000 So if you can package that all up, that's very handy. 00:23:11.000 --> 00:23:12.760 So there are tools like Binder now. 00:23:12.760 --> 00:23:14.560 There's a tool called Everware. 00:23:14.940 --> 00:23:21.060 And previously, there's something like SageMath, which all allowed you to sort of execute a notebook, you know. 00:23:21.060 --> 00:23:26.900 But the problem was how do you get all these, you know, these software dependencies packaged up? 00:23:26.900 --> 00:23:28.460 And now that problem is starting to be solved. 00:23:28.460 --> 00:23:29.200 Right. 00:23:29.200 --> 00:23:30.360 Oh, that's really excellent. 00:23:30.360 --> 00:23:41.260 Because I can imagine you guys have so much data and maybe these back-end systems you've got to reach into to actually work with the data that you're trying to, you know, do physics on. 00:23:41.260 --> 00:23:41.900 That's right. 00:23:41.900 --> 00:23:48.540 You can't just take the program and hand it out, you know, like, oh, and here's our, you know, 50 gigs of data and you've got to get it this way, right? 00:23:48.540 --> 00:23:49.120 Right. 00:23:49.120 --> 00:23:57.080 And not only that, there's also things like databases that say, like, how was the detector aligned on Friday, you know, November 25th or something. 00:23:57.080 --> 00:24:03.460 So there are all these databases involved that you have to connect to for the software to run. 00:24:03.640 --> 00:24:06.760 And that's also through tons of authentication layers. 00:24:06.760 --> 00:24:09.700 So it's a huge pain in the butt, basically. 00:24:09.700 --> 00:24:12.560 But people are solving it. 00:24:12.560 --> 00:24:15.880 And I think that will be a huge change. 00:24:15.880 --> 00:24:23.720 And the Project Jupyter people, you know, luckily had this great foresight to separate the notebook from the background kernel. 00:24:23.920 --> 00:24:29.700 So we're actually also writing kernel based on this C++ interpreter of root. 00:24:29.700 --> 00:24:34.980 So it still looks like notebooks and all the display and everything is the same. 00:24:34.980 --> 00:24:39.720 But in the background, instead of Python, it's the C++ interpreter, which is, you know, interesting. 00:24:40.260 --> 00:24:44.440 Yeah, I mean, that certainly opens it up to a much wider audience. 00:24:44.440 --> 00:24:53.180 Like, you're saying, like, the group that's working directly with Athena and so on, they can just, you know, possibly start using IPython or what do you call them? 00:24:53.180 --> 00:24:54.600 Call them Jupyter notebooks now? 00:24:54.600 --> 00:24:54.960 I don't know. 00:24:54.960 --> 00:24:56.840 I'm not really sure what the naming is. 00:24:56.840 --> 00:25:00.860 Yeah, the front end, kind of the language agnostic part is now Project Jupyter. 00:25:01.980 --> 00:25:18.480 But it's great because we have people like Fernando Perez, who's, you know, leading this effort as part of this advisory board for a project that we got, a grant we got from the National Science Foundation to try to take the tools that have been developed in Iron View Physics, which are mainly very siloed. 00:25:18.480 --> 00:25:20.040 You know, it's like we're trying to solve our problem. 00:25:20.040 --> 00:25:20.960 It's a very hard problem. 00:25:20.960 --> 00:25:22.600 And we don't have a lot of extra time or money. 00:25:22.600 --> 00:25:23.160 Right. 00:25:23.160 --> 00:25:25.520 And then, but now we've done some nice things. 00:25:25.520 --> 00:25:30.800 So let's try to open that up, make it more interoperable with, like, the scientific Python world. 00:25:31.460 --> 00:25:33.160 And it's definitely a two-way street. 00:25:33.160 --> 00:25:35.260 There are lots of other great tools out there that we don't use. 00:25:35.260 --> 00:25:39.080 So we're working on improving the interoperability of all of these things. 00:25:39.080 --> 00:25:41.840 Yeah, I think that's going to be good for science all over. 00:25:41.840 --> 00:25:45.460 And the Jupyter guys just got a huge grant. 00:25:45.460 --> 00:25:50.460 I'm not sure all the folks that contributed, but it was millions, like six million or something like that. 00:25:50.460 --> 00:25:50.960 Do you remember? 00:25:50.960 --> 00:25:57.620 I don't know the number, but they rightfully have been getting some support because they're doing some great things. 00:25:57.620 --> 00:26:01.180 And yeah, I'm really happy to see that. 00:26:01.180 --> 00:26:07.960 So, you know, people will start building C++ and, you know, I guess Ruby and maybe imagine Fortran. 00:26:07.960 --> 00:26:08.420 I don't know. 00:26:08.420 --> 00:26:11.000 That probably is important somewhere in science. 00:26:11.000 --> 00:26:13.080 But I try to not touch that stuff. 00:26:13.080 --> 00:26:13.780 Right. 00:26:14.680 --> 00:26:18.700 You talked about this sort of uniform computing environment. 00:26:18.700 --> 00:26:23.840 And I know just where you're coming from with that, like, just the whole reproducibility and setup and everything. 00:26:23.840 --> 00:26:27.180 Can you talk to, like, are you using Linux? 00:26:27.180 --> 00:26:28.140 What distribution? 00:26:28.140 --> 00:26:29.760 What do things look like there? 00:26:30.260 --> 00:26:30.540 Right. 00:26:30.540 --> 00:26:32.220 So it's definitely Linux-based. 00:26:32.220 --> 00:26:39.040 CERN has a distribution called Scientific Linux that they maintain. 00:26:39.040 --> 00:26:44.240 I've kind of stopped following all the ins and outs of it, to be honest. 00:26:44.240 --> 00:26:50.640 But, you know, at some point it kind of was a derivative of some Red Hat-type package way back in the day. 00:26:50.640 --> 00:26:51.780 But since then it's evolved. 00:26:51.780 --> 00:26:54.840 And now I'm not even sure which distribution it's closest to. 00:26:55.640 --> 00:27:04.600 And then there's been quite a bit of emphasis in virtualization technologies and, you know, OpenStack-related things. 00:27:04.600 --> 00:27:08.080 And what we're starting to see more use of now are also Docker images. 00:27:08.080 --> 00:27:17.380 So I have a student who's working on, you know, making, you know, Docker images that have our very kind of specific computing environment, 00:27:17.380 --> 00:27:21.300 specifically for the issue of reproducibility. 00:27:21.720 --> 00:27:31.800 There, you know, I think you also haven't really seen in high-energy physics, but we're starting to see now as, like, more and more, you know, web-based tools and things like that. 00:27:31.800 --> 00:27:32.800 So you have web services. 00:27:32.800 --> 00:27:38.980 And we have a project going on where we're trying to make not analyses just reproducible but reinterpretable. 00:27:38.980 --> 00:27:44.080 So, you know, you had a team of people analyzing the LHC data, and they were asking a certain question. 00:27:44.080 --> 00:27:47.780 But you can reuse that analysis pipeline to answer other questions. 00:27:47.960 --> 00:27:56.700 So we're trying to wrap up this very heavy computational, you know, pipeline with a very simple web interface, you know, web APIs and things, 00:27:56.700 --> 00:28:04.140 so that people can submit, you know, requests to the system, and they'll be processed through all of this infrastructure. 00:28:04.140 --> 00:28:06.940 And then you'll, you know, come out with a very simple answer. 00:28:06.940 --> 00:28:09.020 That's really cool. 00:28:09.340 --> 00:28:10.460 Yeah, I think it'll be great. 00:28:10.460 --> 00:28:21.940 It also, it addresses a lot of issues with reproducibility when you have, when you can't just, like, say, you know, the typical open data route. 00:28:21.940 --> 00:28:23.400 Like, here it is, do it again. 00:28:23.400 --> 00:28:31.560 Because, you know, there are so many steps involved, and the, you know, the configuration of everything is so heavy that basically almost no one can do that. 00:28:32.160 --> 00:28:35.640 But if the experiments host it, host that service, it's very valuable. 00:28:35.640 --> 00:28:36.000 So. 00:28:36.000 --> 00:28:37.280 Yeah, that makes sense. 00:28:37.280 --> 00:28:45.880 I suspect if it's sufficiently complicated and has enough configuration parameters and variation, like, even the original researchers couldn't reproduce it if they didn't have the details, right? 00:28:45.880 --> 00:28:46.400 Yeah. 00:28:46.400 --> 00:28:47.080 Well, I mean. 00:28:47.080 --> 00:28:50.620 I mean, without redoing the work from scratch, literally. 00:28:50.620 --> 00:28:50.920 Yeah. 00:28:51.360 --> 00:28:52.320 No, exactly. 00:28:52.320 --> 00:28:55.180 I mean, the goal is that we should all be able to do it in practice. 00:28:55.180 --> 00:28:57.880 That's, you know, rarely, rarely checked. 00:28:57.880 --> 00:29:04.200 But that's what we're working on now is to try to make it so we can more confidently say, yes, we can actually reproduce this stuff. 00:29:04.200 --> 00:29:05.500 That's super cool. 00:29:05.500 --> 00:29:08.660 I hadn't even thought about Docker, but that makes perfect sense. 00:29:08.660 --> 00:29:21.180 I mean, I always think of Docker as, here's the way I'm going to, like, horizontally scale my web app so I can do that easier or, like, you know, get higher density on servers and web data center type places. 00:29:21.180 --> 00:29:23.960 But it makes really good sense for scientific computing, doesn't it? 00:29:23.960 --> 00:29:24.920 Yeah, yeah. 00:29:24.920 --> 00:29:26.420 So we have, you know, these experiments. 00:29:26.420 --> 00:29:30.340 At this point, you know, each of the big experiments has put out, you know, several hundred papers. 00:29:30.340 --> 00:29:32.760 So you have, like, thousands of scientific results. 00:29:32.760 --> 00:29:46.440 And now, if associated to each one of these results, you have some sort of, like, Docker image and you can, you know, on demand spin up this service that you want to reproduce or reinterpret what was done. 00:29:46.440 --> 00:29:50.240 You end up making a very powerful, high-level scientific tool. 00:29:50.240 --> 00:29:53.760 So it's been an idea for several years. 00:29:53.760 --> 00:29:57.860 But now, with, you know, these tools that are around, it's really, you know, possible. 00:29:58.120 --> 00:30:00.340 And so, and it's starting to happen. 00:30:00.340 --> 00:30:02.080 Yeah, I think that's fantastic. 00:30:02.080 --> 00:30:07.580 And are you guys, I think, considering or actively putting these into the official Docker repository? 00:30:07.580 --> 00:30:18.080 So right now, that's the sort of model is that that's going to be hosted at CERN, partially because, you know, it's a lot of disk space. 00:30:18.080 --> 00:30:20.380 And we're just working closely with the CERN IT people. 00:30:20.380 --> 00:30:24.180 And there's a lot of trust between the CERN computing and IT and the experiments. 00:30:24.180 --> 00:30:26.760 So that's how that's being developed now. 00:30:27.540 --> 00:30:35.500 But, and some of these things are sensitive, you know, like, yeah, these are big international collaborations. 00:30:35.500 --> 00:30:42.620 And different countries are kind of at different places in terms of their attitudes about open science and things like that. 00:30:42.620 --> 00:30:48.680 So some people still want to keep these things closed, but they're willing to entertain the idea of hosting the service. 00:30:48.680 --> 00:30:52.400 So that's a big political discussion. 00:30:53.000 --> 00:30:56.920 Actually, the more I think about it, that it starts to bring me back to when I was in grad school. 00:30:56.920 --> 00:30:58.380 Yes, I can see that. 00:30:58.380 --> 00:31:09.620 So I suspect once a paper is published in a peer-reviewed journal and approved, pretty much everybody would be pretty happy to have their stuff public. 00:31:09.620 --> 00:31:19.440 But as you're developing that, right, before you've declared the Higgs boson to be found, for example, you would necessarily want to give all your algorithms away and let other people take a shot at it, right? 00:31:19.440 --> 00:31:24.980 You want to keep that until you're ready to publish your papers and publicly do your results, right? 00:31:24.980 --> 00:31:26.620 Yeah, no, that's true. 00:31:26.760 --> 00:31:37.260 And also, you know, I mean, the experiments also worry a lot about, you know, you have to be kind of careful about the scientific message that's coming out of these experiments. 00:31:37.260 --> 00:31:40.800 I mean, they're very, very expensive international projects. 00:31:40.800 --> 00:31:46.480 And you don't want people, like, claiming this and that and adding a lot of noise and drama. 00:31:46.480 --> 00:31:53.080 So, you know, we work very hard to get lots and lots of internal cross-checks of things and then make sure everyone's on the same page. 00:31:53.080 --> 00:31:56.160 And then we want to have kind of one unified voice from an experiment. 00:31:56.160 --> 00:31:58.940 And then we have another experiment to check it. 00:31:58.940 --> 00:32:01.720 But we don't want too much noise. 00:32:01.720 --> 00:32:08.440 So, you know, that's a lot of motivation for keeping these things kind of internal until they're ready. 00:32:08.440 --> 00:32:09.820 Yeah, absolutely. 00:32:09.820 --> 00:32:16.460 I mean, it took several years for the Higgs boson analysis to finally declare, hey, we found it, right? 00:32:17.460 --> 00:32:20.160 Yeah, that had a lot to do with just collecting enough data. 00:32:20.160 --> 00:32:28.240 Once we had enough data, the process was sort of streamlined enough that it was really a matter of like a week and a half or something. 00:32:28.240 --> 00:32:36.440 It was about a week and a half that we went from the last data that we had to the talk that, you know, where the discovery claim was made. 00:32:36.440 --> 00:32:40.760 So, but that was in some sense just like adding the last bits of data. 00:32:40.760 --> 00:32:42.820 But, yeah. 00:32:42.820 --> 00:32:44.660 Yeah, yeah, very cool. 00:32:45.060 --> 00:32:50.300 So, I want to ask you a few more things about LHC and CERN and particle physics. 00:32:50.300 --> 00:32:52.460 And then we could talk about space a little bit maybe. 00:32:52.460 --> 00:32:53.320 Oh, sure. 00:32:53.320 --> 00:32:53.600 Yeah. 00:32:53.600 --> 00:33:04.320 So, you said one of your goals there is that you guys are trying to move towards a more sort of dedicated professional software role there at CERN. 00:33:04.320 --> 00:33:05.520 Can you talk about that a little? 00:33:05.520 --> 00:33:06.880 Oh, right. 00:33:06.960 --> 00:33:13.180 So, I think, you know, the physicists, you know, need to write, obviously, a lot of code and be proficient to be able to do the science they want. 00:33:13.180 --> 00:33:18.800 And, but you also, there's a lot of infrastructure for the processing that's needed. 00:33:18.800 --> 00:33:30.960 So, you know, typically what happened was physicists that were very strong on computing kind of specialized on that and then, you know, became sort of software professionals with a physics background. 00:33:31.520 --> 00:33:34.740 And that's, that model has worked, you know, surprisingly well. 00:33:35.320 --> 00:33:44.640 There are some people, you know, relatively few people that are, don't really have a physics background that are really more just, you know, software professionals. 00:33:44.640 --> 00:33:56.960 But separately, there's been, you know, CERN has had like an IT department that deals with actual computing infrastructure that's going on, but has also developed several different tools that were more services. 00:33:56.960 --> 00:33:57.460 Sure. 00:33:57.540 --> 00:33:59.840 Because you guys have a lot of computers, a lot of network. 00:33:59.840 --> 00:34:01.720 How many computers do you have there? 00:34:01.720 --> 00:34:04.100 Oh, the CERN, I should know the number. 00:34:04.100 --> 00:34:06.520 Hundreds, like a hundred thousand or something, right? 00:34:06.520 --> 00:34:07.580 Like really a lot. 00:34:07.580 --> 00:34:08.600 It's a lot, yeah. 00:34:08.600 --> 00:34:16.440 There's a, there's a great YouTube video that I'll, that you can post that associated to it that has a wonderful overview of the whole processing. 00:34:16.440 --> 00:34:17.320 Oh, yeah, that'd be awesome. 00:34:17.320 --> 00:34:21.100 And lots of little factoids that's nicely produced. 00:34:21.100 --> 00:34:24.860 But I'm not going to say, I won't say a bunch of wrong numbers, so. 00:34:24.860 --> 00:34:25.760 Sure. 00:34:25.760 --> 00:34:25.860 Sure. 00:34:25.860 --> 00:34:42.120 This episode is brought to you by CodeShip. 00:34:42.120 --> 00:34:50.960 CodeShip has launched organizations, create teams, set permissions for specific team members, and improve collaboration in your continuous delivery workflow. 00:34:51.560 --> 00:34:56.860 Maintain centralized control over your organization's projects and teams with CodeShip's new organizations plan. 00:34:56.860 --> 00:35:02.400 And as Talk Python listeners, you can save 20% off any premium plan for the next three months. 00:35:02.400 --> 00:35:05.920 Just use the code TALKPYTHON, all caps, no spaces. 00:35:06.480 --> 00:35:11.660 Check them out at CodeShip.com and tell them thanks for supporting the show on Twitter where they're at CodeShip. 00:35:11.660 --> 00:35:20.760 But the, but there were people that were working on, on that side. 00:35:20.760 --> 00:35:33.340 And then also just general services for like, you know, like the system we use to manage all the papers and the, and the comments to the papers and things like that. 00:35:33.340 --> 00:35:39.720 And like an agenda system where people put up their talks and, you know, schedule meetings and the video services and blah, blah, blah, blah, blah. 00:35:40.160 --> 00:35:47.920 And, and those, the people that were working on that have started branching out into the kinds of services they provide. 00:35:47.920 --> 00:35:54.520 So, so I guess one example that's made me interesting is the very first website in the United States was called Spires. 00:35:54.880 --> 00:36:03.840 And it is, preceded this, the archive where like scientists put the versions of their papers before they're published, this preprint server. 00:36:03.840 --> 00:36:09.680 So the Spires was basically just a database of all of the, high energy physics literature. 00:36:09.680 --> 00:36:13.620 You know, like who wrote what papers and who cited whom and da, da, da, da, da. 00:36:13.620 --> 00:36:23.300 And, and so it was like a, it was the first website in the United States was where physicists would go to see where, you know, where the, who's citing who and which papers are about certain topics. 00:36:23.500 --> 00:36:26.640 And that has evolved into something new called Inspire. 00:36:26.640 --> 00:36:29.360 And that's a, you know, an international effort. 00:36:29.360 --> 00:36:42.800 but the software technology that that's based on, which is mainly Flask based now and, you know, Pythonic, based things, are, has evolved into a set of different tools that are pretty nice. 00:36:42.800 --> 00:36:47.440 So one of them is a data repository where, this is not solving actually CERN's problem. 00:36:47.440 --> 00:36:49.080 It's like the long tail of science. 00:36:49.080 --> 00:36:58.140 So all these small experiments that are out there that have some data, they can put their data onto these servers and then they can refer to that data when they write their papers. 00:36:58.940 --> 00:37:19.620 So there's a service called Zenodo and Zenodo started working with GitHub and made it so that people that now, not just data that you want to point to when you're writing a paper, but the software of your paper, you can, there's a little web hook in, in GitHub that whenever you make a new release, we'll push a copy of the, of the code to the service of Zenodo. 00:37:19.620 --> 00:37:23.340 So, and they'll, they'll, they'll, the jargon is meant a DOI. 00:37:23.340 --> 00:37:28.640 So it's a digital object identifier, which the publishing industry knows about how to use and things like that. 00:37:28.640 --> 00:37:34.280 And it points to a specific version of the, of the code that was used and you can download that version of the code. 00:37:34.280 --> 00:37:37.060 So it's sort of out of the realm of version control. 00:37:37.060 --> 00:37:47.140 It's more, it's just that GitHub is not provided, like people could delete their GitHub repo, but once you get a DOI, it's kind of a, a trust relationship that that code is exactly. 00:37:47.140 --> 00:37:49.060 that code is always going to be there at that link. 00:37:49.060 --> 00:37:49.500 Right. 00:37:49.500 --> 00:37:51.720 Almost like escrow for your code. 00:37:51.720 --> 00:37:53.200 Yeah, exactly. 00:37:53.200 --> 00:37:57.080 So it's redundant in a, in a sense, but you know, there's a link back obviously to the repository. 00:37:57.080 --> 00:38:12.160 So you can see how it's evolved, but, but this is the, the, you know, the basic common denominator for all of the publishing industry and how to track citations to, to different, you know, research outputs, whether they be papers or data or software. 00:38:12.160 --> 00:38:15.580 So that, so that connection has been, you know, is nice. 00:38:15.580 --> 00:38:23.640 And now there are actually thousands of, pieces of code that have been pushed from GitHub to Zenodo where they, and been given these DOIs. 00:38:23.880 --> 00:38:34.700 And the same, that is also, you know, Flask based and, and they have been very nice about how they break up the project, into different, you know, Flask components and things like that. 00:38:34.820 --> 00:38:44.540 And now we're reusing that same kind of infrastructure to try to build these tools for doing a reproducibility and, reinterpretations and things like that. 00:38:44.540 --> 00:38:59.940 So there's a, there's a new effort that's evolving that has a, you know, much more of a like modern, you know, Pythonic programming, you know, web services mentality, which is not necessarily coming from within the experiments, but coming from within the CERN lab. 00:39:00.140 --> 00:39:06.640 And, and I think it's going to change how the experiments sort of approach these problems, in the next few years. 00:39:06.640 --> 00:39:07.480 That's really cool. 00:39:07.480 --> 00:39:10.800 It seems like a huge step down the right path for reproducibility. 00:39:10.800 --> 00:39:14.520 Yeah, no, I think it's very, very important, for what we're doing. 00:39:14.520 --> 00:39:15.780 Yeah, of course. 00:39:15.780 --> 00:39:16.160 Of course. 00:39:16.160 --> 00:39:16.760 Okay. 00:39:16.760 --> 00:39:26.960 The last thing I wanted to ask you about, was this project called Recast that you guys have going that, not you necessarily, but folks at CERN have an LHC of going. 00:39:27.480 --> 00:39:33.540 So Recast is the system that I was referring to that is, for trying to make the analyses reinterpretable. 00:39:33.540 --> 00:39:46.260 and that's the one where we bring in these sort of Docker images for the, for the different analysis pipelines and try to make this web service, for being able to either reproduce or reinterpret, one of the published LHC analyses. 00:39:46.260 --> 00:39:54.700 So that one is, is definitely within CERN, but using a lot of these more modern, you know, web service, type approaches. 00:39:54.700 --> 00:40:00.580 There was another, another project that maybe, you also wanted to talk about, which was the space one. 00:40:00.580 --> 00:40:01.480 I don't know if that's what. 00:40:01.480 --> 00:40:02.260 Yeah, absolutely. 00:40:02.260 --> 00:40:02.900 So. 00:40:02.900 --> 00:40:05.340 Or I don't know if there's a, we're running out of time. 00:40:05.340 --> 00:40:07.240 No, no, that, yeah, no, we definitely have time. 00:40:07.240 --> 00:40:08.100 Let's talk about that one. 00:40:08.100 --> 00:40:09.660 And that's, Kreyfis. 00:40:09.660 --> 00:40:10.240 Yeah. 00:40:10.340 --> 00:40:10.680 That's right. 00:40:10.680 --> 00:40:11.420 That's right. 00:40:11.420 --> 00:40:16.080 So, Kreyfis is this project, which is, we started up, which is a lot of fun. 00:40:16.080 --> 00:40:20.480 and it's a very, very sort of small team of people right now. 00:40:20.480 --> 00:40:23.520 So it's, it's kind of the opposite of these LHC experiments. 00:40:23.520 --> 00:40:30.540 and our basic idea is that we, we want to try to detect very, you know, ultra high energy cosmic rays. 00:40:30.540 --> 00:40:34.900 So these are particles coming from outer space that have much higher energies than the LHC. 00:40:34.900 --> 00:40:39.500 And they smack the atmosphere and create a shower of new particles. 00:40:39.500 --> 00:40:42.800 So you get, you get tons and tons of particles hitting the ground. 00:40:42.800 --> 00:40:54.400 And the idea is that if you get a bunch of people running an app on their phone, the camera of your phone, the little CMOS sensor in your camera is like a miniature particle detector. 00:40:54.400 --> 00:40:59.180 So if one of these particles happens to fly through your camera, it will light your camera up. 00:40:59.180 --> 00:41:09.480 And so basically the idea is that if you get a bunch of people at night running this app, like while their phone's plugged in and they're sleeping, and you see a bunch of these phones light up at the same time, 00:41:09.480 --> 00:41:14.820 then that's, like an indication they just, you just got hit by a high energy cosmic ray. 00:41:15.220 --> 00:41:17.260 And, and, these things are very rare. 00:41:17.260 --> 00:41:24.160 So if you want to see them, like the, one of the best shots you have is to try to cover as much of the earth with a detector as you can. 00:41:24.160 --> 00:41:35.780 And, and it looks like if we have, you know, in the order of a million people running this app, which is a big number, but you know, not crazy big number, that we might be able to do some like really cutting edge science. 00:41:35.780 --> 00:41:42.040 And, and be able to study like the very highest energy particles that people have ever seen in the universe. 00:41:42.040 --> 00:41:49.020 So, so it's a really fun, you know, like citizen science project and it has a lot of data science challenges. 00:41:49.020 --> 00:41:50.260 I'm sure it does. 00:41:50.260 --> 00:41:51.140 That's, that's really cool. 00:41:51.140 --> 00:41:55.480 So does it run on the major phone platforms, iPhone, Android, things like that? 00:41:55.480 --> 00:41:56.280 Yeah. 00:41:56.280 --> 00:42:01.360 So we have an Android app and an, and an iPhone app that they're both kind of in beta right now. 00:42:01.440 --> 00:42:06.120 and, and so they collect data and, you know, one phone can't say anything. 00:42:06.120 --> 00:42:15.220 So they're uploading their data about, sort of when they saw flashes and, and where they were when they saw the flashes to, you know, a central server. 00:42:15.220 --> 00:42:26.540 And, we've, we've just, so it's been fun because we, the people working on this all kind of have a LHC type of background, but we now get a fresh set of choices about what we're going to use. 00:42:26.540 --> 00:42:39.700 So, so, you know, we have Elasticsearch and Django and Spark and, you know, you know, just much, you know, more kind of modern stack of, of, of tools for being able to process the data. 00:42:39.820 --> 00:42:46.900 And then we're working on trying to make it so that the final stages of the data analysis are, are more like IPython notebooks. 00:42:46.900 --> 00:42:53.080 And that we have this ability to, not just view the notebook, but to launch the notebook and recompute it. 00:42:53.080 --> 00:43:03.060 So that's using things like SageMath or this binder project, and, and, and Everware, which have, you know, with Docker, all of the computing environment. 00:43:03.260 --> 00:43:09.740 And this is going to, I think, be just awesome for doing outreach because they're, this has really grabbed the public's imagination. 00:43:09.740 --> 00:43:13.640 We got covered by NPR and Wired and some people. 00:43:13.640 --> 00:43:15.220 And so the, and IFL science. 00:43:15.220 --> 00:43:22.080 So we have like almost a hundred thousand people that have signed up and we're, we haven't even launched the app yet. 00:43:22.080 --> 00:43:22.980 That's really cool. 00:43:22.980 --> 00:43:25.280 So if people want to get started, how do they do it? 00:43:25.280 --> 00:43:27.160 Is that a thing they can do yet? 00:43:27.160 --> 00:43:28.080 Or do we have to wait a while? 00:43:28.760 --> 00:43:30.820 we're going to have to wait a little bit longer. 00:43:30.820 --> 00:43:34.680 we're, we're working as fast as we can, with our ragtag team. 00:43:34.680 --> 00:43:42.440 But the, but if you, you can sign up for it right now, if you go to the website, it's, crayfish.io. 00:43:42.440 --> 00:43:44.580 So it's like crayfish without the H. 00:43:44.580 --> 00:43:45.060 Right. 00:43:45.060 --> 00:43:47.120 So see, yeah, I'll definitely add a link. 00:43:47.120 --> 00:43:50.220 C R A Y F I S dot IO. 00:43:50.220 --> 00:43:51.120 That's right. 00:43:51.120 --> 00:43:51.540 That's right. 00:43:51.540 --> 00:43:52.320 Fantastic. 00:43:52.600 --> 00:44:02.360 So this, you know, reminds me a little bit of the whole protein folding stuff that people try to do with like running it on idle computers and the study at home. 00:44:02.360 --> 00:44:06.400 But, but it seems even cooler because this is actually the science. 00:44:06.400 --> 00:44:07.060 That's right. 00:44:07.060 --> 00:44:07.440 Yeah. 00:44:07.440 --> 00:44:08.300 Happening. 00:44:08.300 --> 00:44:11.260 Not just the post analysis computational bits. 00:44:11.260 --> 00:44:12.000 That's right. 00:44:12.000 --> 00:44:12.160 Yeah. 00:44:12.160 --> 00:44:15.280 I think there's sort of three types of citizen science projects like this. 00:44:15.280 --> 00:44:23.960 There are ones like study at home and the protein folding where you're basically donating compute cycles, you know, to solve a problem, but you're not really involved in otherwise. 00:44:23.960 --> 00:44:33.440 then there are projects like, there's a great platform called the Zooniverse, which I encourage people to go check out if you like these things. 00:44:33.440 --> 00:44:35.120 So they are the very interactive. 00:44:35.120 --> 00:44:44.860 You're like looking at pictures of galaxies and trying to, to classify them or, you know, like pictures of the ocean and trying to say if you see certain things in them. 00:44:44.920 --> 00:44:52.620 And it's, it's a lot like mechanical Turk, you know, you, so you, you're farming out, these things that computers might not be great at. 00:44:52.620 --> 00:44:56.960 We don't have, we're not good at writing algorithms for doing and having humans do it. 00:44:56.960 --> 00:45:00.480 and this one, this project is neither of those. 00:45:00.480 --> 00:45:02.520 The citizen science is actually the instrument. 00:45:02.520 --> 00:45:04.860 I mean, you are, you are doing the science. 00:45:04.860 --> 00:45:10.340 And, so that's, that's, yeah, I think, you know, that makes our project really unique. 00:45:10.340 --> 00:45:11.100 Yeah. 00:45:11.100 --> 00:45:11.700 That's really awesome. 00:45:11.700 --> 00:45:15.840 So I encourage everyone out there to go sign up, participate. 00:45:15.840 --> 00:45:16.920 I'll definitely do it. 00:45:16.920 --> 00:45:17.800 This is exciting. 00:45:17.800 --> 00:45:18.180 Great. 00:45:18.180 --> 00:45:18.600 Thank you. 00:45:18.600 --> 00:45:19.700 Yeah. 00:45:19.700 --> 00:45:25.400 And then you said this is somewhat, done with a, a grant from Amazon web services. 00:45:25.400 --> 00:45:26.600 That's right. 00:45:26.600 --> 00:45:26.800 Yeah. 00:45:26.840 --> 00:45:31.080 So, so we talked to Amazon and they thought it was an exciting, you know, an exciting project. 00:45:31.080 --> 00:45:43.800 And so they, they've been great about trying to support some, in different, you know, scientific projects that, that, need some computing, that, you know, have, that are still in a prototyping stage. 00:45:44.040 --> 00:45:50.580 And, so they, yeah, they've given us a grant for some of the, you know, AWS, sort of budget. 00:45:50.580 --> 00:45:57.520 And, and we're using that for, you know, collecting the data and some of our, you know, you know, offline processing of the data. 00:45:57.520 --> 00:46:07.240 and, and, and we try to make it so that there's a kind of a feedback loop so that the, you know, the Django server is showing stats on what the current status of the network is. 00:46:07.240 --> 00:46:11.820 and, so it's, it, that's been very nice of them. 00:46:11.820 --> 00:46:23.960 And also Yandex, which is like a sort of the Russian Google is, has a group of people that are data scientists, and part of a lab that they run that are very well integrated at CERN. 00:46:23.960 --> 00:46:33.680 And some of them have joined our project and they're, you know, helping us, you know, build the, the, the, sort of offline spark based, you know, data processing system and things like that. 00:46:33.680 --> 00:46:39.160 So it's a lot of fun to have, you know, these different mix of people and a very fresh project. 00:46:39.160 --> 00:46:40.100 We move very fast. 00:46:40.100 --> 00:46:40.520 It's nice. 00:46:40.520 --> 00:46:40.880 Yeah. 00:46:40.880 --> 00:46:41.440 How exciting. 00:46:41.440 --> 00:46:45.960 I mean, I know how it is to pick up a project that's been around for 10 or 20 years. 00:46:45.960 --> 00:46:52.400 It's kind of stale in the technology and what you can do because you don't want to break stuff, but it's nice to start over and do exciting things, right? 00:46:52.400 --> 00:46:53.380 Absolutely. 00:46:53.380 --> 00:46:53.740 Yeah. 00:46:53.740 --> 00:46:54.720 It's a breath of fresh air. 00:46:54.720 --> 00:46:55.540 Cool. 00:46:55.540 --> 00:46:57.220 So we're kind of getting short on time. 00:46:57.220 --> 00:47:00.080 Let me ask you just a few more questions before we wrap things up. 00:47:00.080 --> 00:47:08.840 What do you think the most amazing or interesting thing about the Large Hadron Collider and all the stuff you guys did there that people might not really know about? 00:47:08.840 --> 00:47:10.140 Oh boy. 00:47:10.140 --> 00:47:18.960 Part of it, I think that's really great about the whole project is just that, you know, it, it, we have people from all around the world, right? 00:47:18.960 --> 00:47:26.000 And they're not all employed by one, you know, one, you know, central, you know, corporation or something like that. 00:47:26.000 --> 00:47:28.180 And so, and we're all contributing. 00:47:28.180 --> 00:47:36.140 So I guess in that sense, it has a lot of the flavor of like a successful open science, I mean, it's open software project where people are contributing from all around. 00:47:36.400 --> 00:47:45.400 And I think the other part is that, you know, everyone should feel proud about it because, you know, it's, you know, it's not a huge amount of your tax dollars. 00:47:45.400 --> 00:47:52.920 You know, if you look at like, I forget for every thousand dollars of taxes you pay, it's just a few cents or something is going to this kind of project. 00:47:52.920 --> 00:47:54.740 But, you know, everyone made this happen. 00:47:54.740 --> 00:47:59.320 And I think everyone in the world should be proud of the fact that this, this happened. 00:47:59.320 --> 00:48:02.060 Yeah, it's definitely a global effort, which is awesome. 00:48:02.060 --> 00:48:03.480 It is a global effort. 00:48:03.480 --> 00:48:10.840 And it's, you know, this is a, I mean, it's a really big accomplishment what's happened in terms of our understanding of the universe. 00:48:10.840 --> 00:48:18.220 So, and there's obviously all of the software and computing and the technical challenges of pulling it off. 00:48:18.220 --> 00:48:22.920 But, you know, it's a big multi-decade global thing. 00:48:22.920 --> 00:48:25.100 And, you know, I don't know, humanity should be proud of itself. 00:48:25.100 --> 00:48:27.220 Yeah, absolutely. 00:48:27.220 --> 00:48:27.960 That's great. 00:48:27.960 --> 00:48:36.040 How do you see Python in science sort of evolving, growing, changing, shrinking? 00:48:36.040 --> 00:48:39.960 I mean, right now, I think it's growing very fast. 00:48:40.080 --> 00:48:47.220 I mean, you know, the whole scientific Python side of things and, you know, distributions that are working, working better. 00:48:47.220 --> 00:48:52.160 And GitHub is changing, you know, how people are doing science very, very quickly. 00:48:52.160 --> 00:49:01.980 If you look in, like, the astronomy and astrophysics community, there have been, you know, some big efforts to create new analysis platforms. 00:49:01.980 --> 00:49:17.040 There's a great project called AstroPy, which is a, you know, just a poster boy for, you know, for different, for, you know, kind of open science mentality of the community contributing to making the tools they need to do the science that they want. 00:49:18.040 --> 00:49:25.340 And the, that's working pretty well because, you know, a lot of those experiments, they have a lot of new experiments coming up and they want a fresh start. 00:49:26.120 --> 00:49:35.180 So, you know, we have this issue that we have a lot of legacy code and, you know, a lot of, you know, a lot of inertia about how we've done things. 00:49:35.260 --> 00:49:43.040 But we're, we're trying to move things into a more, you know, a less siloed software picture that looks more like that model. 00:49:43.040 --> 00:49:53.320 And, and Python is a big, I think Python is like the dominant kind of language of people that have that mentality and at least in, you know, particle physics and astrophysics. 00:49:53.320 --> 00:49:55.680 Yeah, that, that's my feeling as well. 00:49:55.680 --> 00:49:57.560 But obviously I do a lot less science than you do. 00:49:57.560 --> 00:49:59.000 So that's, that's great to hear. 00:50:00.200 --> 00:50:12.880 Yeah, I mean, I think the other place that you see it is just that to the extent that data science is attached to doing sciences, as opposed to like business analytics, you know, you see just a big influx of Python and R. 00:50:12.880 --> 00:50:25.860 And so I think that, you know, there are a lot of, a lot of other scientific fields that are undergoing a similar, that are undergoing a similar transformation, but, but maybe a little bit heavier on the R side than on the Python side. 00:50:25.860 --> 00:50:28.420 Yeah, it's definitely down to those two, I think. 00:50:28.820 --> 00:50:31.300 And modern analysis for the most part. 00:50:31.300 --> 00:50:31.940 Right. 00:50:31.940 --> 00:50:34.980 Kyle, thank you for being on the show and taking the time. 00:50:34.980 --> 00:50:38.340 I know you're super busy and this, this is really, really interesting. 00:50:38.340 --> 00:50:39.340 Thanks for sharing it with everyone. 00:50:39.340 --> 00:50:40.140 No, absolutely. 00:50:40.140 --> 00:50:41.020 Thank you for inviting me. 00:50:41.020 --> 00:50:41.380 It's great. 00:50:41.380 --> 00:50:42.060 Yeah, you bet. 00:50:42.060 --> 00:50:42.860 Take care. 00:50:42.860 --> 00:50:43.120 Bye. 00:50:43.120 --> 00:50:47.180 This has been another episode of Talk Python To Me. 00:50:47.180 --> 00:50:51.380 Today's guest was Kyle Cranmer and this episode has been sponsored by Hired and CodeShip. 00:50:51.380 --> 00:50:53.280 Thank you guys for supporting the show. 00:50:53.280 --> 00:50:55.920 Hired wants to help you find your next big thing. 00:50:55.920 --> 00:51:03.940 Visit Hired.com slash Talk Python To Me to get five or more offers with salary and equity presented right up front and a special listener signing bonus of $4,000. 00:51:03.940 --> 00:51:07.480 CodeShip wants you to always keep shipping. 00:51:07.480 --> 00:51:11.340 Check them out at CodeShip.com and thank them on Twitter via at CodeShip. 00:51:11.600 --> 00:51:13.300 Don't forget the discount code for listeners. 00:51:13.300 --> 00:51:13.860 It's easy. 00:51:13.860 --> 00:51:14.720 Talk Python. 00:51:14.720 --> 00:51:15.420 All caps. 00:51:15.420 --> 00:51:16.120 No spaces. 00:51:16.120 --> 00:51:23.220 You can find the links from today's show at talkpython.fm/episodes slash show slash 29. 00:51:23.220 --> 00:51:25.500 Be sure to subscribe to the show. 00:51:25.500 --> 00:51:27.780 Open your favorite podcatcher and search for Python. 00:51:27.780 --> 00:51:28.960 We should be right at the top. 00:51:28.960 --> 00:51:33.140 You can also find the iTunes and direct RSS feeds in the footer of the website. 00:51:33.900 --> 00:51:38.280 Our theme music is Developers, Developers, Developers by Corey Smith, who goes by Smix. 00:51:38.280 --> 00:51:41.700 You can hear the entire song at talkpython.fm. 00:51:41.700 --> 00:51:43.840 This is your host, Michael Kennedy. 00:51:43.840 --> 00:51:44.880 Thanks for listening. 00:51:44.880 --> 00:51:46.900 Smix, take us out of here. 00:51:47.900 --> 00:51:49.040 Staying with my voice. 00:51:49.040 --> 00:51:50.840 There's no norm that I can fill within. 00:51:50.840 --> 00:51:52.060 Haven't been sleeping. 00:51:52.060 --> 00:51:53.660 I've been using lots of rest. 00:51:53.660 --> 00:51:56.500 I'll pass the mic back to who rocked it best. 00:51:56.500 --> 00:52:07.960 Developers, Developers, Developers, Developers, Developers, Developers, Developers, Developers, 00:52:07.960 --> 00:52:08.460 you 00:52:08.460 --> 00:52:10.460 .