WEBVTT 00:00:00.020 --> 00:00:02.080 Do you know what's actually slow in your Python app? 00:00:02.480 --> 00:00:03.200 Or are you just guessing? 00:00:03.780 --> 00:00:04.140 Until now, 00:00:04.380 --> 00:00:07.000 profiling Python meant a tracing profiler 00:00:07.070 --> 00:00:09.740 that made your code two to three times slower, 00:00:10.460 --> 00:00:12.660 or a third-party tool that broke 00:00:12.670 --> 00:00:14.480 with every new release of the Python runtime. 00:00:15.180 --> 00:00:17.000 Python 3.15 fixes that. 00:00:17.340 --> 00:00:19.680 It ships with Tachyon, a sampling profiler 00:00:20.160 --> 00:00:21.500 built into the standard library. 00:00:22.100 --> 00:00:23.820 It attaches to live production apps 00:00:23.970 --> 00:00:25.300 with almost zero overhead. 00:00:26.040 --> 00:00:28.000 My guests are Pablo Galindo Salgado, 00:00:28.600 --> 00:00:30.980 a CPython core developer and steering council member, 00:00:31.560 --> 00:00:35.240 and Laszlo Kiskoller from Bloomberg's Python infrastructure team. 00:00:35.880 --> 00:00:38.840 Their first prototype ran at two samples a second. 00:00:39.380 --> 00:00:42.320 Now it can do over a million hertz sampling. 00:00:42.760 --> 00:00:46.120 The best news is this lands in Python 3.15 today. 00:00:47.360 --> 00:00:48.480 This is Talk Python To Me, 00:00:48.680 --> 00:00:50.580 episode 565, 00:00:51.040 --> 00:00:52.700 recorded September 16th, 00:00:53.000 --> 00:00:53.660 2026. 00:01:11.900 --> 00:01:16.220 Welcome to Talk Python To Me, the number one Python podcast for developers and data scientists. 00:01:16.770 --> 00:01:17.480 This is your host, 00:01:17.700 --> 00:01:18.120 Michael Kennedy. 00:01:18.320 --> 00:01:22.060 I'm a PSF fellow who's been coding for over 25 years. 00:01:22.720 --> 00:01:23.760 Let's connect on social media. 00:01:24.140 --> 00:01:26.400 You'll find me and Talk Python on Mastodon, Bluesky, 00:01:26.760 --> 00:01:27.280 and X. 00:01:27.560 --> 00:01:29.400 The social links are all in your show notes. 00:01:30.160 --> 00:01:33.680 You can find over 10 years of past episodes at talkpython.fm. 00:01:33.860 --> 00:01:37.060 And if you want to be part of the show, you can join our recording live streams. 00:01:37.420 --> 00:01:37.880 That's right. 00:01:38.140 --> 00:01:41.340 We live stream the raw uncut version of each episode on YouTube. 00:01:41.940 --> 00:01:46.260 Just visit talkpython.fm/youtube to see the schedule of upcoming events. 00:01:46.580 --> 00:01:50.220 Be sure to subscribe there and press the bell so you'll get notified anytime we're recording. 00:01:50.480 --> 00:01:51.980 This episode is brought to you by Sentry. 00:01:52.260 --> 00:01:56.360 Sentry's new MCP server connects your error reports straight to your coding agent. 00:01:56.880 --> 00:01:57.960 Claude Code reads the errors, 00:01:58.400 --> 00:01:58.940 finds the bugs, 00:01:59.300 --> 00:01:59.800 and fixes them. 00:02:00.340 --> 00:02:03.320 Get started at talkpython.fm/sentry. 00:02:03.600 --> 00:02:06.400 Pablo Laszlo, welcome to Talk Python Week. 00:02:06.640 --> 00:02:06.720 Pablo, 00:02:06.880 --> 00:02:07.280 welcome back. 00:02:07.380 --> 00:02:08.740 Always good to have you on the show. 00:02:08.899 --> 00:02:09.580 Laszlo, hello. 00:02:09.979 --> 00:02:10.700 Yeah, thank you very much. 00:02:11.039 --> 00:02:12.000 Yeah, it's good to see you both. 00:02:12.200 --> 00:02:13.760 I'm really excited to talk about profilers. 00:02:14.040 --> 00:02:16.540 I've had very extreme examples. 00:02:18.060 --> 00:02:18.860 Extreme examples. 00:02:19.860 --> 00:02:21.620 Extreme experiences with profiling. 00:02:21.880 --> 00:02:28.300 experiences. Because I was certain, Pablo, I was certain something was happening with my program, 00:02:28.500 --> 00:02:29.220 making it slow. 00:02:29.570 --> 00:02:34.780 And then I attached a profiler and it wasn't even closely related. It was just 00:02:35.080 --> 00:02:36.360 something completely else. 00:02:36.540 --> 00:02:40.420 I was using the wrong data structure instead of the 6,000 lines of 00:02:40.800 --> 00:02:46.120 complex mathematical stuff that I thought was slow. And it's really saved the day because I would 00:02:46.540 --> 00:02:50.100 have tried to rewrite stuff that was super complicated. And really all I needed to do 00:02:50.100 --> 00:02:53.920 with something silly like switch from a dictionary or a list to a set or a list to a dictionary, 00:02:54.220 --> 00:02:54.940 something like this, right? 00:02:55.040 --> 00:02:55.420 And it's like, 00:02:55.760 --> 00:02:55.880 wow, 00:02:56.220 --> 00:02:59.520 it's just like list find or list index sort of thing. 00:02:59.600 --> 00:03:00.100 Oh, my gosh. 00:03:00.720 --> 00:03:01.880 Yeah, many such cases. 00:03:03.140 --> 00:03:03.240 Yes, 00:03:03.400 --> 00:03:03.500 yes, 00:03:03.700 --> 00:03:03.760 yes. 00:03:05.000 --> 00:03:07.340 I think it's more common than you will think, 00:03:07.580 --> 00:03:07.660 right? 00:03:07.720 --> 00:03:07.940 Like, 00:03:08.280 --> 00:03:09.460 I know that when this happens, 00:03:09.720 --> 00:03:11.300 it's going to feel a bit bad because it's like, 00:03:11.320 --> 00:03:11.420 oh, 00:03:11.580 --> 00:03:12.100 man, I'm stupid. 00:03:12.300 --> 00:03:12.520 But like, 00:03:12.760 --> 00:03:16.039 no, it's extremely anti-intuitive in the world of modern software 00:03:16.160 --> 00:03:18.800 when there is a bazillion layers of things 00:03:18.940 --> 00:03:22.280 and now it's very much unclear what is costing things. 00:03:22.560 --> 00:03:23.820 And that happens everywhere. 00:03:24.160 --> 00:03:27.020 That happens in the smallest shops and the biggest shops. 00:03:27.780 --> 00:03:29.980 So the tools are very important. 00:03:30.380 --> 00:03:31.360 Yeah, they absolutely are. 00:03:31.400 --> 00:03:36.280 And you all are behind some really cool new profiling capabilities 00:03:36.520 --> 00:03:37.160 coming into Python. 00:03:37.680 --> 00:03:38.060 Yes, yes. 00:03:38.320 --> 00:03:40.560 Looking forward to diving into that with you. 00:03:40.860 --> 00:03:43.700 So let's just do a quick reintroduction for you, 00:03:43.900 --> 00:03:44.100 Pablo. 00:03:44.120 --> 00:03:45.640 Just tell people about yourself a little bit, 00:03:45.740 --> 00:03:46.460 what you've been up to. 00:03:46.940 --> 00:03:48.500 It's been a while since you've been on the show. 00:03:48.780 --> 00:03:49.000 Yes, 00:03:49.160 --> 00:03:49.560 but always, 00:03:49.720 --> 00:03:51.440 always been a lot of fun to be here. 00:03:51.620 --> 00:03:52.900 So like, let's do more, let's do more. 00:03:52.930 --> 00:03:55.620 But let me keep going and do more profilers 00:03:55.840 --> 00:03:58.260 so we can come here even more time. 00:03:58.440 --> 00:03:59.800 That's my goal, 00:03:59.980 --> 00:04:01.780 just doing more profilers to come here. 00:04:02.480 --> 00:04:03.060 So yes, 00:04:03.260 --> 00:04:04.340 I think last time we talked about 00:04:04.550 --> 00:04:06.200 perhaps PyStack or Membray, one of the two. 00:04:06.210 --> 00:04:07.400 I don't remember the last one. 00:04:08.020 --> 00:04:08.980 We spoke about both. 00:04:08.980 --> 00:04:09.800 I don't remember the order. 00:04:10.500 --> 00:04:11.000 Yeah, exactly. 00:04:11.470 --> 00:04:11.560 Yeah, 00:04:11.850 --> 00:04:12.740 but yeah. 00:04:12.890 --> 00:04:14.500 So yeah, since then, I think, 00:04:14.920 --> 00:04:15.900 Yeah, a bunch of things have changed. 00:04:15.950 --> 00:04:18.799 I think, you know, a bunch of Python releases since then. 00:04:19.250 --> 00:04:22.079 We are now in very close to 3.15. 00:04:22.520 --> 00:04:23.120 It's very hard. 00:04:23.170 --> 00:04:25.400 You will think, like, people know it is the last version, 00:04:25.540 --> 00:04:27.360 but in my mind, 3.15 is all news 00:04:27.660 --> 00:04:29.020 because the main branch is 3.16. 00:04:29.230 --> 00:04:30.740 So, like, it's like all stuff. 00:04:30.960 --> 00:04:32.200 It's like, oh, is that released already? 00:04:32.720 --> 00:04:34.340 Anyway, no, 3.15 on October. 00:04:34.560 --> 00:04:36.820 I'm sure you will do a very good episode about that 00:04:36.850 --> 00:04:36.940 so, 00:04:37.180 --> 00:04:38.340 you know, people should listen to it. 00:04:38.500 --> 00:04:38.680 Yeah, 00:04:38.850 --> 00:04:38.980 yeah. 00:04:39.580 --> 00:04:40.480 We've done before, 00:04:40.750 --> 00:04:40.840 like, 00:04:41.100 --> 00:04:43.340 kind of a launch event thing 00:04:43.640 --> 00:04:44.240 around Python. 00:04:44.580 --> 00:04:46.800 I think it was 3.13, 3.12. 00:04:47.090 --> 00:04:48.640 I can't remember which one, but that was a lot of fun. 00:04:48.780 --> 00:04:50.500 Are you involved in the release this year? 00:04:51.460 --> 00:04:53.580 I mean, from the sidelines, I still release my versions, 00:04:53.770 --> 00:04:54.560 3.10 and 3.11, 00:04:54.900 --> 00:04:57.460 but the release manager right now is Hugo Pankeminade, 00:04:57.960 --> 00:05:00.060 and the next is going to be Savannah, 00:05:01.700 --> 00:05:04.940 which is going to be the new release manager for 3.16 and 3.17. 00:05:05.320 --> 00:05:06.520 So I just help in. 00:05:06.990 --> 00:05:07.520 I mean, there's always, 00:05:07.680 --> 00:05:07.800 like, 00:05:07.980 --> 00:05:08.620 things to fix, 00:05:08.770 --> 00:05:12.780 and certainly since I'm the author of a bunch of stabbing 3.15, 00:05:13.300 --> 00:05:14.340 quite a lot of stuff actually, 00:05:14.620 --> 00:05:16.280 like, you know, lazy imports and the profiler. 00:05:16.540 --> 00:05:18.060 So like there's a bunch of broken stuff. 00:05:19.340 --> 00:05:21.100 So yeah, that does, that's, 00:05:21.700 --> 00:05:22.440 you know, I'll help it. 00:05:22.460 --> 00:05:22.540 Yeah. 00:05:22.960 --> 00:05:23.020 Yeah. 00:05:23.120 --> 00:05:24.000 So apart from that, I don't know, 00:05:24.160 --> 00:05:24.560 I, 00:05:24.760 --> 00:05:25.600 you know, same things. 00:05:25.840 --> 00:05:27.540 I think the last time are still true, 00:05:27.800 --> 00:05:28.420 you know, core developer. 00:05:28.800 --> 00:05:29.840 I was still in the stream console. 00:05:30.080 --> 00:05:31.440 I think I'm furniture at this point. 00:05:31.620 --> 00:05:33.400 I feel like I've been there for like six years already. 00:05:33.560 --> 00:05:34.500 So I don't know. 00:05:34.860 --> 00:05:37.160 Maybe it's time to, to, to, to reconsider. 00:05:38.500 --> 00:05:41.000 But yeah, I think the only thing is new is that I changed employers. 00:05:41.260 --> 00:05:42.360 I used to work for Bloomberg, 00:05:42.720 --> 00:05:43.860 which is with Laszlo. 00:05:43.910 --> 00:05:44.600 I used to work there. 00:05:45.030 --> 00:05:45.600 And this actually, 00:05:45.760 --> 00:05:48.240 the provider work started when I was at Bloomberg. 00:05:48.730 --> 00:05:50.380 But now I'm, I betrayed Bloomberg. 00:05:50.800 --> 00:05:52.800 And now I'm in Hudson River trading. 00:05:53.300 --> 00:05:53.940 Hudson River trading. 00:05:54.240 --> 00:05:54.440 Bam. 00:05:55.100 --> 00:05:57.440 Doing like fast trading. 00:05:57.860 --> 00:05:58.340 Lots of money. 00:05:58.560 --> 00:05:58.780 Yes, yes. 00:05:59.120 --> 00:05:59.960 And that's very exciting. 00:06:00.120 --> 00:06:00.300 Also, 00:06:00.580 --> 00:06:01.580 probably a little high stress, 00:06:01.770 --> 00:06:02.220 but that's okay. 00:06:02.540 --> 00:06:03.160 No, I'm chill. 00:06:03.600 --> 00:06:05.180 Like, you know, they let me record this podcast. 00:06:06.020 --> 00:06:06.100 Well, 00:06:06.280 --> 00:06:08.260 I'm just meaning like, if you remove a lot of money, 00:06:08.460 --> 00:06:10.300 that is a nerve wracking thing. 00:06:10.360 --> 00:06:23.920 It's more like the stress I think is more like what happens when you screw up because like a bloomer I think you screw up and you know maybe your manager says bad bad bad developer you know but like it's fine and I can talk about bloomer this way because I'm not worried anymore. 00:06:25.400 --> 00:06:26.500 But sorry guys. 00:06:26.560 --> 00:06:28.320 You're going to make this interesting for me Pablo. 00:06:28.410 --> 00:06:28.540 Yes. 00:06:28.960 --> 00:06:29.680 Sorry Chime. 00:06:30.960 --> 00:06:31.480 This guy. 00:06:31.900 --> 00:06:31.980 Anyway. 00:06:33.080 --> 00:06:38.640 But yeah here is like when you screw up there is a very scary number that comes with the build you know. 00:06:38.700 --> 00:06:41.000 and like your little deployment failure 00:06:41.580 --> 00:06:44.280 caused like sort of big numbers. 00:06:44.540 --> 00:06:45.780 So you're like, shit, 00:06:46.740 --> 00:06:46.980 damn. 00:06:47.680 --> 00:06:48.320 So when that number, 00:06:48.620 --> 00:06:49.560 you know, it's very high, 00:06:49.720 --> 00:06:49.900 then, 00:06:50.160 --> 00:06:50.780 you know, so it's scary. 00:06:51.040 --> 00:06:51.120 So, 00:06:51.280 --> 00:06:52.340 yeah, I don't know. 00:06:52.360 --> 00:06:53.060 It keeps interesting. 00:06:53.520 --> 00:06:53.940 I find it. 00:06:53.940 --> 00:06:53.980 Yeah, 00:06:54.180 --> 00:06:55.780 but after you release Python 00:06:56.200 --> 00:06:59.580 to how many tens of millions or more people 00:06:59.900 --> 00:07:02.220 and platforms and it's low state. 00:07:02.320 --> 00:07:02.740 I mean, 00:07:03.040 --> 00:07:04.360 that's also a big deal. 00:07:04.360 --> 00:07:04.880 I'm used to it. 00:07:04.980 --> 00:07:05.480 I'm used to it. 00:07:05.680 --> 00:07:08.340 This is just like high speed. 00:07:08.640 --> 00:07:09.000 high speed. 00:07:09.190 --> 00:07:09.960 Like in Python, 00:07:10.140 --> 00:07:10.820 when you screw up, 00:07:11.240 --> 00:07:13.260 you know, it's after a year that people use your thing 00:07:13.270 --> 00:07:13.940 and they say, oh, 00:07:14.210 --> 00:07:14.400 you know, 00:07:14.680 --> 00:07:16.680 lazy imports is bad. And then you say, damn. 00:07:17.080 --> 00:07:18.000 They're not going to say 00:07:18.030 --> 00:07:19.420 that. Lazy imports are going to be awesome. 00:07:19.780 --> 00:07:20.580 I'm for it. 00:07:20.920 --> 00:07:21.900 That's it. That's it. 00:07:22.280 --> 00:07:23.560 One of us. One of us. 00:07:24.080 --> 00:07:24.400 But here, 00:07:24.600 --> 00:07:24.960 you know, like, 00:07:25.180 --> 00:07:27.520 they will tell you the millisecond it fails. 00:07:27.680 --> 00:07:28.440 So you will know. 00:07:28.650 --> 00:07:28.940 But yeah, 00:07:29.280 --> 00:07:30.040 last look. And, 00:07:30.250 --> 00:07:30.440 you know, 00:07:30.980 --> 00:07:31.580 also, 00:07:32.050 --> 00:07:34.860 quick shout out to core.py. That's a fun. Oh, thank you. I forgot. 00:07:34.920 --> 00:07:35.940 I only have one job. 00:07:36.100 --> 00:07:36.880 I only have one job. 00:07:37.000 --> 00:07:37.760 I failed at the job. 00:07:38.100 --> 00:07:38.260 Sorry, 00:07:38.460 --> 00:07:38.540 Lucas. 00:07:38.700 --> 00:07:38.860 Sorry, 00:07:39.020 --> 00:07:39.100 Lucas. 00:07:40.060 --> 00:07:40.700 Yeah, Lucas, 00:07:40.980 --> 00:07:41.200 run that. 00:07:41.320 --> 00:07:42.860 That's a great, that's a fun deep dive. 00:07:43.100 --> 00:07:44.740 Like core developer internals. 00:07:45.020 --> 00:07:45.200 Yes, 00:07:45.360 --> 00:07:45.440 yes. 00:07:45.900 --> 00:07:46.500 Listen to us. 00:07:46.580 --> 00:07:47.300 We're still alive. 00:07:47.700 --> 00:07:52.960 Lucas just like joined Meta and Mark Zuckerberg himself doesn't allow him to record that often. 00:07:53.240 --> 00:07:57.200 Like that, when he joined Meta, Mark called him and was like, you cannot just, you cannot 00:07:57.320 --> 00:07:58.580 just do this podcast often. 00:07:58.760 --> 00:07:59.800 So like now we cannot do it. 00:07:59.800 --> 00:08:01.960 It's like, if you're not going to have me on the show to talk about AI, 00:08:02.120 --> 00:08:02.820 we can do it. 00:08:03.000 --> 00:08:03.260 Yeah, 00:08:03.480 --> 00:08:03.620 exactly. 00:08:03.920 --> 00:08:04.900 So, but we have integrity. 00:08:04.970 --> 00:08:07.060 We say, no, Mark, you cannot be on Corel.py 00:08:07.100 --> 00:08:08.720 because you don't talk about Python. 00:08:09.900 --> 00:08:11.460 It's not a PHP show, man. 00:08:11.550 --> 00:08:11.800 Come on. 00:08:11.980 --> 00:08:12.200 Exactly. 00:08:12.330 --> 00:08:12.420 Yeah, 00:08:12.580 --> 00:08:12.760 yeah. 00:08:13.280 --> 00:08:13.740 Those guys. 00:08:14.880 --> 00:08:15.340 So, yeah, yeah. 00:08:15.450 --> 00:08:17.280 So we're still alive, 00:08:17.700 --> 00:08:18.380 just slower. 00:08:18.960 --> 00:08:20.920 But listen to Corel.py in all platforms. 00:08:21.480 --> 00:08:21.740 Solid. 00:08:22.040 --> 00:08:23.020 Lazlo, welcome to the show. 00:08:23.280 --> 00:08:23.660 Tell us about yourself. 00:08:24.000 --> 00:08:24.360 Thanks. 00:08:24.950 --> 00:08:25.640 Happy to be here. 00:08:25.950 --> 00:08:26.040 So, 00:08:27.210 --> 00:08:28.400 yeah, so we've been working. 00:08:29.220 --> 00:08:31.240 I have worked with Pablo for quite a few years. 00:08:31.820 --> 00:08:31.860 Originally, 00:08:33.360 --> 00:08:34.280 I think since 2018, 00:08:34.840 --> 00:08:38.340 we started in the Python team together in Lundberg. 00:08:38.780 --> 00:08:42.020 And up until the end of last year, we've worked together. 00:08:43.320 --> 00:08:48.100 I actually participated in some of the projects like PyStack and Memory internally, 00:08:49.180 --> 00:08:51.860 not nearly as much to the extent as in Tachyon. 00:08:52.540 --> 00:08:55.820 But then we started collaborating with Pablo last year. 00:08:56.620 --> 00:08:57.520 And I remember this because, 00:08:57.720 --> 00:09:01.380 you know, probably just mentioned, like, it seems like 315 years old. 00:09:01.580 --> 00:09:04.540 I remember that we started working on this in the Sprint last year. 00:09:04.540 --> 00:09:05.440 Picon last year. 00:09:05.540 --> 00:09:05.780 Yeah, yeah. 00:09:06.240 --> 00:09:06.320 Crazy. 00:09:06.720 --> 00:09:07.060 April, 00:09:07.300 --> 00:09:07.580 May. 00:09:08.020 --> 00:09:10.820 And it just seems like so far away. 00:09:11.140 --> 00:09:13.040 And it's crazy that it's still not out. 00:09:14.100 --> 00:09:15.880 But yeah, so that's where everything started. 00:09:16.140 --> 00:09:18.320 But now that you have this thing fresh, 00:09:18.500 --> 00:09:19.180 like now, right? 00:09:19.240 --> 00:09:20.140 Like the last year, 00:09:20.300 --> 00:09:21.740 you know, we were young and full of hope. 00:09:21.920 --> 00:09:26.100 But like imagine the time when Python releases were every two years. 00:09:26.540 --> 00:09:26.840 Oh, yeah. 00:09:26.980 --> 00:09:27.380 Damn. 00:09:27.840 --> 00:09:27.920 Damn. 00:09:28.220 --> 00:09:29.020 Because you think, oh, 00:09:29.200 --> 00:09:29.300 man, 00:09:29.460 --> 00:09:29.720 312, 00:09:29.900 --> 00:09:30.260 313. 00:09:30.460 --> 00:09:30.580 No, 00:09:30.760 --> 00:09:30.940 man. 00:09:31.300 --> 00:09:32.740 from 3.8 to 3.9. 00:09:33.280 --> 00:09:34.260 Well, 3.8 was the first one, 00:09:34.360 --> 00:09:36.540 but 3.7 to 3.8 was two years. 00:09:36.900 --> 00:09:38.460 You forget everything about the feature. 00:09:38.920 --> 00:09:42.060 You introduced ramp by the time it's released. 00:09:42.620 --> 00:09:42.780 Exactly. 00:09:43.100 --> 00:09:44.240 Do you guys think it should go faster? 00:09:44.640 --> 00:09:44.820 Yeah. 00:09:45.700 --> 00:09:47.220 Do you think there should be a six months 00:09:47.280 --> 00:09:49.640 or every three months version of Python that comes out? 00:09:50.760 --> 00:09:51.160 Or is it not? 00:09:51.900 --> 00:09:54.560 I think from the usability thing, 00:09:55.000 --> 00:09:57.420 I think I will do rolling releases if I could, 00:09:57.660 --> 00:10:00.079 but the problem is that CPython is too big 00:10:00.100 --> 00:10:01.220 to like do that. 00:10:01.460 --> 00:10:03.920 In the sense that stability is really hard. 00:10:04.300 --> 00:10:05.880 In the sense that ideally in the ideal world, 00:10:06.000 --> 00:10:07.280 you merge things that aren't ready, 00:10:07.500 --> 00:10:07.600 right? 00:10:07.660 --> 00:10:08.360 And then you don't like, 00:10:08.700 --> 00:10:10.760 you know, merge half bike stuff. 00:10:10.880 --> 00:10:11.080 But like, 00:10:11.440 --> 00:10:11.600 you know, 00:10:12.020 --> 00:10:13.000 for a software this complex, 00:10:13.500 --> 00:10:14.960 I like that affects so much things 00:10:15.260 --> 00:10:16.600 and like so many people depend on it. 00:10:16.680 --> 00:10:17.960 Of course, like people are going to run 00:10:17.980 --> 00:10:19.240 on the latest one, right? 00:10:19.340 --> 00:10:21.360 But like, it's very unfortunate. 00:10:21.600 --> 00:10:23.380 And from this thought that is obviously wrong 00:10:23.839 --> 00:10:25.700 to like the one year, two year release, 00:10:25.760 --> 00:10:26.440 there is a continuum, 00:10:26.660 --> 00:10:26.780 right? 00:10:26.860 --> 00:10:28.040 So like, where do you put the line? 00:10:28.480 --> 00:10:35.100 But in our experience in the core team so far is that one year is already like running to the edges of, 00:10:35.880 --> 00:10:37.320 you know, how much we can ensure. 00:10:38.180 --> 00:10:41.460 Because like there's always some last time scare. 00:10:41.800 --> 00:10:42.040 You know, 00:10:42.260 --> 00:10:43.220 there's always like, oh, 00:10:43.600 --> 00:10:46.180 you know, for example, in Python 3.12, 00:10:46.440 --> 00:10:50.340 we have to revert an entire new GC implementation. 00:10:50.800 --> 00:10:50.900 Right. 00:10:51.120 --> 00:10:53.360 And now in Python 3.14, 00:10:54.080 --> 00:10:54.900 sorry, that was 3.13. 00:10:55.040 --> 00:10:55.920 In Python 3.14, 00:10:56.180 --> 00:10:57.340 we have to revert it again. 00:10:57.800 --> 00:11:02.640 So like, because it was like having a huge amount of memory impact and that was only 00:11:02.860 --> 00:11:04.260 known after people started to use it. 00:11:04.620 --> 00:11:09.640 So that gave us like some information that perhaps we are, or either way we need to look 00:11:09.640 --> 00:11:10.060 at the process, 00:11:10.340 --> 00:11:11.080 which we are of course, 00:11:11.300 --> 00:11:14.180 but like, or one year is already like on the line 00:11:14.280 --> 00:11:15.160 of what is responsible. 00:11:15.560 --> 00:11:15.720 So, 00:11:16.120 --> 00:11:19.040 you know, it would be amazing if it's like rolling release and then you can get like 00:11:19.120 --> 00:11:22.520 the features the next day, but not a possibility. 00:11:23.140 --> 00:11:23.220 Yeah. 00:11:23.500 --> 00:11:24.880 And I guess it doesn't change. 00:11:25.300 --> 00:11:26.880 One of the things that's really nice about Python, 00:11:27.360 --> 00:11:27.560 honestly, 00:11:28.040 --> 00:11:32.340 is there's almost never a reason not to run on the new lease. 00:11:32.780 --> 00:11:33.900 It's very, 00:11:34.120 --> 00:11:34.580 very stable. 00:11:34.780 --> 00:11:40.300 Only once have I had a major release of Python break any of my apps. 00:11:40.560 --> 00:11:45.020 And the reason was the async coroutine decorator was removed 00:11:45.860 --> 00:11:49.660 in lieu of async and await only for async funks, something like that. 00:11:50.080 --> 00:11:51.280 Oh, I also didn't look at it. 00:11:51.580 --> 00:11:51.840 Yeah. 00:11:52.560 --> 00:11:58.220 And one of the libraries I was using for the database way deep down still used that. 00:11:58.400 --> 00:11:59.400 And I didn't know. 00:11:59.560 --> 00:12:00.820 But when I rolled out the new version, 00:12:00.980 --> 00:12:02.700 it's like, what are you talking about, decorator 00:12:03.300 --> 00:12:03.720 async code routine? 00:12:03.780 --> 00:12:05.240 I'm like, it just doesn't run. 00:12:05.370 --> 00:12:05.780 I don't know. 00:12:05.900 --> 00:12:06.360 This isn't great. 00:12:07.020 --> 00:12:07.980 But that's not what happens. 00:12:08.050 --> 00:12:10.560 And I guess the faster you go, the more likely that is to happen, right? 00:12:10.880 --> 00:12:11.080 Yeah. 00:12:11.190 --> 00:12:12.180 I think you're also lucky. 00:12:12.270 --> 00:12:13.320 I think like for, 00:12:13.880 --> 00:12:14.300 I mean, I don't know. 00:12:14.400 --> 00:12:15.340 I don't want to assume. 00:12:15.350 --> 00:12:17.080 But I suppose my experience, 00:12:17.490 --> 00:12:17.780 for instance, 00:12:17.960 --> 00:12:20.120 working at Bloomberg and also here is not 00:12:20.120 --> 00:12:20.540 really the same. 00:12:21.160 --> 00:12:22.420 when your software is big enough, 00:12:22.770 --> 00:12:25.860 you will run into the sync decorators all the time. 00:12:26.080 --> 00:12:26.480 You know what I mean? 00:12:27.500 --> 00:12:28.760 And in the weirdest cases, 00:12:29.020 --> 00:12:30.920 normally it's not even just Python. 00:12:31.090 --> 00:12:32.000 It's like, oh, 00:12:32.340 --> 00:12:37.240 this Python 3.15 requires you to upgrade your Pandas data science stack 00:12:37.330 --> 00:12:41.700 because the NumPy only supports 3.15 in version 2.5, right? 00:12:41.810 --> 00:12:42.960 So if you're running version 2, 00:12:43.180 --> 00:12:45.540 that forces you to move 0.5, 00:12:45.860 --> 00:12:47.980 sorry, five versions of NumPy or whatever it is. 00:12:48.090 --> 00:12:50.540 And then, I don't know, you run into all sorts of things, 00:12:50.700 --> 00:12:50.780 right? 00:12:50.860 --> 00:12:51.340 Like here, 00:12:51.540 --> 00:12:56.560 the work I do right now in HRT, we have a lot of numeric work, for instance. 00:12:56.890 --> 00:13:01.640 And there is times when we upgrade the interpreter and nothing breaks in the sense that, 00:13:01.900 --> 00:13:04.340 you know, this function doesn't exist or this doesn't work. 00:13:04.440 --> 00:13:04.900 It's like, no, 00:13:05.340 --> 00:13:07.360 like the floating points are slightly off, 00:13:07.700 --> 00:13:10.700 you know, like so the simulation is slightly off. 00:13:11.020 --> 00:13:11.100 And, 00:13:11.770 --> 00:13:16.940 you know, you chase that down and it's like pandas changed slightly the algorithm and like nothing, 00:13:17.400 --> 00:13:18.240 you know. 00:13:19.240 --> 00:13:25.820 like maybe the compiler is slightly different or something or like a compiler optimization flag 00:13:26.840 --> 00:13:32.980 so people you know they still are relatively major events even if in the core team we've been working 00:13:32.980 --> 00:13:37.960 with for instance numpy and pandas to ensure that they have wheels now on release so when you you 00:13:37.960 --> 00:13:43.800 know when you start 315 uh and they won we won for can use no buy and it used to be not true right 00:13:43.940 --> 00:13:48.660 3.8 that was not a thing but now it is yeah it's certainly that uh the deploying of packages is 00:13:49.040 --> 00:13:52.440 I want to talk about a couple things as a way to kind of introduce, 00:13:52.900 --> 00:13:56.980 work our way into performance and profiling and so on. 00:13:57.380 --> 00:14:00.540 And since you brought it up, let's talk real quick about lazy imports, 00:14:00.700 --> 00:14:02.500 explicit lazy imports. 00:14:03.260 --> 00:14:03.620 Explicit. 00:14:03.960 --> 00:14:04.040 Yeah. 00:14:04.690 --> 00:14:06.600 Well, by the way, did you consider not explicit? 00:14:07.000 --> 00:14:07.540 Of course, yeah. 00:14:07.950 --> 00:14:08.840 And that was rejected. 00:14:09.520 --> 00:14:10.360 I wonder why. 00:14:11.060 --> 00:14:12.800 So what does the syntax even look like? 00:14:13.220 --> 00:14:13.640 Yeah, yeah. 00:14:13.800 --> 00:14:16.700 So you just put lazy import to your import and bam, 00:14:16.880 --> 00:14:17.620 your problem is faster. 00:14:17.920 --> 00:14:18.400 Isn't that cool? 00:14:18.680 --> 00:14:18.760 Well, 00:14:19.060 --> 00:14:19.280 asterisk, 00:14:19.440 --> 00:14:22.220 no, but that's the selling point. 00:14:23.040 --> 00:14:23.140 Well, 00:14:23.360 --> 00:14:23.460 okay. 00:14:23.720 --> 00:14:25.120 The guess is that, you know, 00:14:26.060 --> 00:14:26.400 so Python, 00:14:26.660 --> 00:14:29.120 one of the powers of Python is very unsurprising 00:14:29.220 --> 00:14:29.840 is packaging, 00:14:30.220 --> 00:14:30.320 right? 00:14:30.400 --> 00:14:32.020 Like you have a package for everything, 00:14:32.760 --> 00:14:33.900 but guess what? 00:14:34.220 --> 00:14:39.780 Those packages also use packages for everything and so on and so forth until you download is 00:14:40.060 --> 00:14:40.160 even, 00:14:40.340 --> 00:14:43.280 you know, and like just to check if a number is even, you just download a package 00:14:43.320 --> 00:14:43.660 or something. 00:14:44.020 --> 00:14:47.560 So what happens is that because this library is packaged a bunch of functionality, 00:14:48.000 --> 00:14:51.520 sometimes when you only want to reach for a tiny function in some package, 00:14:51.760 --> 00:14:54.040 you end importing the entire thing, right? 00:14:54.180 --> 00:14:58.700 And the entire thing will end importing the entire thing of other packages and so on and so forth. 00:14:59.000 --> 00:15:02.320 So it's very common that just by importing a tiny function, 00:15:02.880 --> 00:15:05.080 you end paying for the entire world. 00:15:05.240 --> 00:15:06.840 And some of those imports are really slow, 00:15:07.180 --> 00:15:07.340 unfortunately, 00:15:07.780 --> 00:15:14.480 because like they need to load big files or anyone that has dealt with Torch knows how chunky the video wheels are and whatnot. 00:15:15.020 --> 00:15:15.320 So, 00:15:15.740 --> 00:15:17.800 you know, that's bad because you should only pay for what you use. 00:15:17.950 --> 00:15:21.320 So the idea of explicit lazy imports is that, you know, 00:15:21.600 --> 00:15:23.400 if people start to mark the imports as lazy, 00:15:23.780 --> 00:15:26.400 that you will only pay for the imports that you really need. 00:15:26.470 --> 00:15:28.220 So only when you actually need that thing, 00:15:28.410 --> 00:15:29.320 then it will be imported. 00:15:29.780 --> 00:15:35.460 And that transforms your mega graph of imports of things that you don't really need into only the ones that you need. 00:15:35.510 --> 00:15:37.340 And that saves you a lot of time and memory, 00:15:37.600 --> 00:15:37.680 right? 00:15:37.780 --> 00:15:39.540 Because loading those things is not free. 00:15:40.080 --> 00:15:40.620 Not only in time, 00:15:40.780 --> 00:15:40.860 right? 00:15:40.900 --> 00:15:41.540 It's in memory as well. 00:15:41.760 --> 00:15:42.000 Yeah. 00:15:42.510 --> 00:15:42.600 Honestly, 00:15:43.410 --> 00:15:44.940 I think, I mean, there are, 00:15:45.120 --> 00:15:46.280 speaking of profilers, 00:15:46.440 --> 00:15:47.380 there are some tools. 00:15:47.780 --> 00:15:48.960 Now, this is not a very popular one. 00:15:49.020 --> 00:15:49.360 It's very old. 00:15:49.390 --> 00:15:50.300 I don't know if it even works. 00:15:50.500 --> 00:15:55.400 But there are some tools to do things like profile import time and so on. 00:15:55.760 --> 00:15:57.080 Certainly the time can matter. 00:15:57.400 --> 00:15:58.140 I guess it matters. 00:15:58.680 --> 00:15:59.720 It depends on what you're doing. 00:16:00.070 --> 00:16:01.240 If you're doing serverless, 00:16:01.440 --> 00:16:04.600 if you're doing CLIs, if you're doing subprocessing, 00:16:04.990 --> 00:16:05.100 those 00:16:05.360 --> 00:16:08.700 kinds of things really matter because it's spin up the thing, do a little bit of work 00:16:08.800 --> 00:16:09.260 and go away. 00:16:09.580 --> 00:16:10.400 If it's a web app, 00:16:10.640 --> 00:16:12.640 you know, startup and then it just chills, 00:16:12.880 --> 00:16:13.000 right? 00:16:13.060 --> 00:16:14.680 It doesn't really matter in terms of time. 00:16:14.850 --> 00:16:17.340 But I'll tell you where it does matter a lot is memory. 00:16:17.640 --> 00:16:17.880 Right, 00:16:18.080 --> 00:16:18.160 right. 00:16:18.260 --> 00:16:23.180 I was on a mission to reduce how much memory it takes to run like the Talk Python courses 00:16:23.360 --> 00:16:24.460 site and all the different things. 00:16:24.920 --> 00:16:30.140 And I was surprised a couple hundred megabytes were ports that were rarely used. 00:16:30.700 --> 00:16:34.120 So maybe if I run some back-end admin, 00:16:34.640 --> 00:16:37.940 it would pull up NumPy, Matplotlib, and other types of stuff, right? 00:16:38.300 --> 00:16:39.380 But otherwise it wouldn't. 00:16:39.600 --> 00:16:41.900 And that added at least 100 megabytes to the runtime. 00:16:42.480 --> 00:16:44.700 And because the way the imports worked, everything was eager. 00:16:45.120 --> 00:16:47.220 It basically imported everything at startup. 00:16:47.480 --> 00:16:49.760 So just by having those import statements there, 00:16:49.920 --> 00:16:53.000 the app was using 100 megabytes more than it needed. 00:16:53.200 --> 00:16:55.780 I changed them to the best I could of lazy imports, 00:16:55.960 --> 00:16:59.000 where you just import in the function instead of the top of the file. 00:16:59.040 --> 00:16:59.940 Lazy imports at home. 00:17:00.600 --> 00:17:00.700 Yeah, 00:17:01.120 --> 00:17:02.140 like homegrown, 00:17:02.320 --> 00:17:03.320 homebrew lazy imports. 00:17:04.199 --> 00:17:07.500 And it really dropped the memory by 100 megabytes. 00:17:07.620 --> 00:17:10.740 And on the server, that's one of the most expensive things is RAM, 00:17:11.000 --> 00:17:11.600 not compute. 00:17:12.339 --> 00:17:15.640 I was going to say that there's one more, I think, super cool area 00:17:15.800 --> 00:17:17.920 where lazy imports help a lot, 00:17:18.079 --> 00:17:19.060 which is CLI tools, 00:17:19.860 --> 00:17:22.180 which typically can be, and we've seen a lot of this, 00:17:22.360 --> 00:17:26.640 And I know with Pablo, we actually even looked at some stuff at work in the past where once 00:17:26.640 --> 00:17:28.820 you have a complex enough application, 00:17:29.420 --> 00:17:31.360 but you literally just want to see the help page 00:17:31.600 --> 00:17:34.940 or just spin something up and it can take seconds. 00:17:35.400 --> 00:17:39.200 And lazy imports can make a massive difference in that sort of the responsiveness of that 00:17:39.300 --> 00:17:39.680 CLI tool. 00:17:40.040 --> 00:17:40.220 Right. 00:17:40.480 --> 00:17:45.640 I was going to just add that there is also second and third order effects that people 00:17:45.800 --> 00:17:46.380 think about, 00:17:46.540 --> 00:17:49.180 but depending on the size of your thing can be quite dire. 00:17:49.480 --> 00:17:50.440 Like one story, for instance, 00:17:50.660 --> 00:17:55.240 for my current employer is that even if you don't care about memory, 00:17:55.400 --> 00:17:57.040 let's say you have all the memory in the world, right? 00:17:57.680 --> 00:17:59.620 And let's say you don't care about the startup tag, 00:17:59.870 --> 00:18:00.740 because like you said, like, 00:18:01.360 --> 00:18:02.380 you know, you have a web app, 00:18:02.640 --> 00:18:02.740 right? 00:18:02.740 --> 00:18:04.500 And like, you just start your thing and that's fine. 00:18:05.220 --> 00:18:13.900 What can happen still is that if you import a lot of stuff that you don't use, all those imports will bring into existence a bunch of like a lot of Python objects. 00:18:14.140 --> 00:18:15.300 And those objects can be classes, 00:18:15.760 --> 00:18:16.920 functions, 00:18:17.600 --> 00:18:18.280 constants, 00:18:19.480 --> 00:18:21.180 and, you know, like all sorts of things. 00:18:21.270 --> 00:18:23.360 And those things will be more and more and more and more. 00:18:23.500 --> 00:18:26.160 And so you end with a lot of Python objects in your, 00:18:26.420 --> 00:18:28.920 you know, in your program that you don't use, right? 00:18:29.340 --> 00:18:33.600 And those things have non-zero costs because every time the garbage collector runs, 00:18:33.930 --> 00:18:38.140 it will, like, look at all those things and then every run of the garbage collector will be slower. 00:18:38.760 --> 00:18:42.899 Of course, like, you know, you have a web app that is a solution like the Instagram GC Freeze 00:18:43.340 --> 00:18:44.220 that they implemented, 00:18:44.520 --> 00:18:46.360 but this will happen also, you don't have web apps. 00:18:46.470 --> 00:18:50.500 Like you have like even like the case of my employee, like scientific computing, 00:18:51.100 --> 00:18:54.440 and you are crunching out of numbers just by importing stuff that you don't use, 00:18:54.490 --> 00:18:55.820 those numbers will be crunched slower. 00:18:55.990 --> 00:19:00.560 And you cannot just DC freeze because you don't have like a fork model 00:19:00.760 --> 00:19:02.460 that then you run your workers, 00:19:02.740 --> 00:19:02.820 right? 00:19:03.450 --> 00:19:07.820 So importing less will make these objects not exist, 00:19:08.020 --> 00:19:10.080 and therefore the app will run more linear. 00:19:10.520 --> 00:19:11.140 That's the key, right? 00:19:11.340 --> 00:19:14.640 So there is like very interesting second and third order effects to this. 00:19:15.340 --> 00:19:18.040 This portion of Talk Python is brought to you by Sentry. 00:19:18.280 --> 00:19:20.860 Now, I've told you about Sentry many times on the show, 00:19:21.340 --> 00:19:24.980 but this time I have something new that should be very exciting to you. 00:19:25.540 --> 00:19:28.180 Sentry has recently released their MCP server. 00:19:28.680 --> 00:19:29.560 And for me personally, 00:19:29.800 --> 00:19:32.060 this has truly unlocked a new level of integration 00:19:32.300 --> 00:19:33.340 and workflow with Sentry. 00:19:33.820 --> 00:19:34.180 You see, 00:19:34.720 --> 00:19:36.660 Sentry is integrated into Talk Python, 00:19:37.260 --> 00:19:37.560 courses, 00:19:38.020 --> 00:19:38.220 website, 00:19:38.640 --> 00:19:39.340 the mobile app, 00:19:39.440 --> 00:19:42.280 and a whole bunch of things behind the scenes no one ever sees. 00:19:43.240 --> 00:19:44.980 When something goes wrong in any of these apps, 00:19:45.160 --> 00:19:46.560 I'm alerted immediately over email, 00:19:46.740 --> 00:19:49.480 and I can log into Sentry and see all the details of what happened, 00:19:49.780 --> 00:19:51.380 the parameters that were passed to functions, 00:19:51.960 --> 00:19:53.900 what user encountered this error, and much more. 00:19:54.700 --> 00:19:55.100 Traditionally, 00:19:55.480 --> 00:19:57.780 what I would do is I would read the stack trace, 00:19:58.340 --> 00:19:59.100 open up the source code, 00:19:59.280 --> 00:20:01.180 try to piece together the detective story 00:20:01.200 --> 00:20:03.140 that would solve what was wrong with my code. 00:20:03.540 --> 00:20:04.240 Once I figured it out, 00:20:04.400 --> 00:20:06.560 I would fix the code and push out a new version. 00:20:07.180 --> 00:20:07.980 But you know what's coming, 00:20:08.160 --> 00:20:08.240 right? 00:20:08.800 --> 00:20:09.460 It's 2026. 00:20:09.820 --> 00:20:13.440 This is the kind of work that AI is spectacularly good at. 00:20:14.340 --> 00:20:18.120 Now when there's an error, I simply open up Claude Code with the project that had the error, 00:20:18.560 --> 00:20:21.060 and I tell Claude, we have new errors in Sentry. 00:20:21.280 --> 00:20:23.260 Please investigate them, and if you can, 00:20:23.630 --> 00:20:23.960 solve them. 00:20:24.320 --> 00:20:28.500 Claude immediately knows to use Sentry's MCP server because I connected it to 00:20:28.640 --> 00:20:29.180 Claude Code. 00:20:29.400 --> 00:20:31.020 It investigates all the errors, 00:20:31.580 --> 00:20:32.920 uses the information I discussed, 00:20:33.180 --> 00:20:33.740 and much more, 00:20:33.900 --> 00:20:35.660 as well as the code base it's working in. 00:20:36.360 --> 00:20:41.120 And 99% of the time, it fixes this without another instruction from me. 00:20:41.390 --> 00:20:46.140 The magic of Sentry's MCP server is that it gives you full Sentry error reporting 00:20:46.190 --> 00:20:50.620 and management directly connected to Frontier models running inside your code base. 00:20:51.160 --> 00:20:53.260 If you use Agentic AI for your code at all, 00:20:53.370 --> 00:20:57.120 you definitely need to check out Sentry and connect Sentry's MCP server 00:20:57.340 --> 00:20:58.540 straight to your favorite coding agent. 00:20:59.060 --> 00:21:00.280 Get started today at Sentry. 00:21:00.370 --> 00:21:02.500 Be sure to use our link and discount code. 00:21:03.040 --> 00:21:05.700 It's talkpython.fm/sentry, 00:21:05.920 --> 00:21:07.980 code talkpython26. 00:21:08.880 --> 00:21:10.640 That's talkpython.fm/sentry. 00:21:10.880 --> 00:21:12.100 It gives you extra credits, 00:21:12.320 --> 00:21:12.880 but more importantly, 00:21:12.920 --> 00:21:14.480 it lets Sentry know you came from us. 00:21:14.980 --> 00:21:17.120 Thank you to Sentry for supporting Talk Python To Me. 00:21:17.940 --> 00:21:20.320 Python does a bunch to try to keep the memory together, 00:21:21.039 --> 00:21:23.380 arenas and blocks and all that kind of business. 00:21:23.700 --> 00:21:25.880 But yeah, if you allocate a whole bunch of pointers, 00:21:26.320 --> 00:21:28.400 all of a sudden the GC has to count them, right? 00:21:28.660 --> 00:21:28.740 Right. 00:21:28.880 --> 00:21:30.220 The Gen 2 collections, yeah. 00:21:30.560 --> 00:21:30.660 Okay. 00:21:31.120 --> 00:21:32.120 Well, I'm excited for it. 00:21:32.240 --> 00:21:38.340 So everyone get ready to write lazy import module instead of just import module and see where it takes you. 00:21:38.680 --> 00:21:38.800 Yes. 00:21:39.200 --> 00:21:39.480 All right. 00:21:39.880 --> 00:21:40.060 Also, 00:21:40.440 --> 00:21:43.440 let's talk faster CPython a little bit. 00:21:43.740 --> 00:21:43.960 All right. 00:21:44.170 --> 00:21:46.100 I mean, we're talking profiling. 00:21:46.350 --> 00:21:48.080 We're talking about making code faster. 00:21:48.530 --> 00:21:50.060 The Faster CPython initiative, 00:21:50.300 --> 00:21:52.999 even though it's gone through a little bit of organizational challenges. 00:21:54.100 --> 00:21:54.640 Drop my life, 00:21:54.820 --> 00:21:54.880 yeah. 00:21:55.180 --> 00:21:55.340 Yeah, 00:21:55.700 --> 00:22:00.300 like with the Microsoft pulling the support or the people off of the project or whatever. 00:22:00.960 --> 00:22:02.120 I think it's been a huge success. 00:22:02.320 --> 00:22:05.220 I think Python is so much faster in a lot of ways. 00:22:05.540 --> 00:22:06.160 And yeah, 00:22:06.340 --> 00:22:07.340 what are your thoughts? 00:22:07.680 --> 00:22:07.960 Either of you. 00:22:08.240 --> 00:22:08.760 Yeah, I think, 00:22:09.240 --> 00:22:09.460 I mean, 00:22:10.040 --> 00:22:11.140 it's pretty obvious 00:22:11.360 --> 00:22:13.140 if you look at like the performance of like, 00:22:13.360 --> 00:22:14.000 let's say 39, 00:22:14.600 --> 00:22:15.600 that was when this started, 00:22:15.780 --> 00:22:16.740 or like 310 perhaps, 00:22:17.280 --> 00:22:18.940 like on the performance of today's Python, 00:22:19.420 --> 00:22:20.840 like that has been a clear success. 00:22:21.580 --> 00:22:22.740 And this kind of continues. 00:22:22.880 --> 00:22:26.140 It's just that like we just run out of like obvious things. 00:22:26.260 --> 00:22:27.060 Obvious doesn't mean easy, 00:22:27.240 --> 00:22:27.320 right? 00:22:27.340 --> 00:22:28.780 But like obvious things to look at. 00:22:29.360 --> 00:22:29.780 And right now, 00:22:29.940 --> 00:22:32.940 the biggest efforts are into big chunks of this work, 00:22:33.080 --> 00:22:35.780 like JIT, which is like, you know, a big piece of software. 00:22:35.920 --> 00:22:37.540 I think you got Brandon in the show, 00:22:37.860 --> 00:22:38.040 perhaps, 00:22:38.960 --> 00:22:39.380 in the past, 00:22:39.680 --> 00:22:40.340 to talk about it. 00:22:40.680 --> 00:22:41.040 So now, 00:22:41.780 --> 00:22:44.660 you know, that's something that you cannot just commit in a couple of months. 00:22:45.340 --> 00:22:47.020 Pips are needed and all sorts of things. 00:22:47.180 --> 00:22:47.860 So right now, 00:22:48.240 --> 00:22:48.400 you know, 00:22:48.600 --> 00:22:50.900 the successor of that project is still happening. 00:22:51.340 --> 00:22:56.120 It's just that now those optimizations are much harder and require more careful consideration. 00:22:56.480 --> 00:23:03.540 And as you very well know, we are also very deep into the weeds of free threading and free thread Python dropping the GIL. 00:23:04.040 --> 00:23:08.360 And, you know, it turns out that those things kind of like don't play necessarily well together. 00:23:08.780 --> 00:23:12.080 So there is also like, you know, a lot of considerations there. 00:23:12.800 --> 00:23:14.460 And going to the topic at hand, 00:23:15.180 --> 00:23:18.120 profiling is really important to ensure that like none of this gets wrong. 00:23:18.220 --> 00:23:18.380 Right. 00:23:18.580 --> 00:23:22.800 And to also know that indeed your application became faster. 00:23:23.120 --> 00:23:23.480 Why? 00:23:23.780 --> 00:23:23.980 Because, 00:23:24.160 --> 00:23:24.260 like, 00:23:24.520 --> 00:23:24.780 you know, 00:23:25.000 --> 00:23:29.380 is this because Python 3.15 is going to be faster or is it because, like, NumPy is faster? 00:23:29.560 --> 00:23:30.180 Like, how do you know? 00:23:30.340 --> 00:23:31.620 Well, you need to use one of these tools, 00:23:31.820 --> 00:23:31.900 right? 00:23:32.100 --> 00:23:32.220 Yeah. 00:23:32.540 --> 00:23:32.860 Vaslo? 00:23:33.080 --> 00:23:33.300 Yeah, 00:23:33.460 --> 00:23:41.060 I mean, I watched the sort of the fastest SuperTor project from the sidewings for the past couple of years. 00:23:41.320 --> 00:23:42.400 And I think the, 00:23:42.560 --> 00:23:43.320 I actually remember, 00:23:43.480 --> 00:23:44.240 I think the biggest jump, 00:23:44.500 --> 00:23:45.960 was it 3.12? 00:23:46.020 --> 00:23:47.840 there was a very significant jump there. 00:23:48.190 --> 00:23:53.220 And it was very exciting because when we introduced that version, 00:23:53.290 --> 00:23:55.460 there was such a significant jump that whereas, 00:23:55.760 --> 00:23:56.340 you know, 00:23:57.400 --> 00:23:58.880 oftentimes you would have to convince folks, 00:23:58.970 --> 00:24:00.400 like you need to upgrade your Python versions, 00:24:00.640 --> 00:24:00.760 right? 00:24:01.000 --> 00:24:02.060 But when you have this carrot, 00:24:02.480 --> 00:24:06.560 where you have literally a 30-40% performance jump just for free, 00:24:06.730 --> 00:24:06.840 basically, 00:24:07.380 --> 00:24:08.780 that just made it a lot more appealing. 00:24:08.830 --> 00:24:11.320 And I think it opened up a lot more areas 00:24:11.430 --> 00:24:14.200 where Python became competitive and more appealing. 00:24:14.280 --> 00:24:14.940 So I think, 00:24:15.140 --> 00:24:22.600 yeah, I mean, I shared sort of what Paul mentioned about the complexity, 00:24:22.840 --> 00:24:24.720 sort of the JIT coming in, 00:24:25.460 --> 00:24:26.040 no-gil Python. 00:24:26.300 --> 00:24:29.340 I think it's quite a bit challenging now, 00:24:29.500 --> 00:24:31.940 I think, to look at these performance improvements. 00:24:32.900 --> 00:24:35.420 I mean, I have a lot of faith in the JIT compiler, 00:24:35.600 --> 00:24:38.520 you know, making significant performance improvements. 00:24:39.760 --> 00:24:43.600 I'm not sure exactly how exactly it's going to work. 00:24:44.000 --> 00:24:46.480 I have a lot of compilers as well, I think. 00:24:46.680 --> 00:24:47.460 Compilers are pretty awesome. 00:24:47.980 --> 00:24:50.420 Some stuff out of the C eval, 00:24:50.660 --> 00:24:51.940 which for statement, 00:24:52.220 --> 00:24:52.980 that would be good, you know? 00:24:53.100 --> 00:24:53.320 Yep. 00:24:53.400 --> 00:24:55.420 Yeah, and I think at the same time, 00:24:55.640 --> 00:25:03.480 it's also not forgetting that free trading Python has the potential to provide very significant speedups, 00:25:03.640 --> 00:25:03.740 right? 00:25:03.820 --> 00:25:06.600 It really depends on the application and the domain. 00:25:07.020 --> 00:25:17.900 But I think we can actually see very significant speedups and we can see like certain extensions and performance sensitive libraries can produce very significant speedups. 00:25:18.280 --> 00:25:22.620 And I think that's like sort of, I guess what we're seeing is now it's going into in multiple different directions, 00:25:22.940 --> 00:25:23.900 depending on your application. 00:25:24.530 --> 00:25:27.820 You can benefit from free threading potentially a lot more than, 00:25:28.280 --> 00:25:30.980 for example, waiting for a JIT profiler to go further. 00:25:31.320 --> 00:25:31.500 Yeah, 00:25:32.000 --> 00:25:33.460 I'm really excited about the free threading stuff. 00:25:33.580 --> 00:25:35.300 I think it has tons of potential, 00:25:35.760 --> 00:25:36.679 but I'm also concerned. 00:25:37.270 --> 00:25:43.700 I feel like a lot of the people building libraries and packages for Python have not deeply thought about race conditions. 00:25:44.300 --> 00:25:48.140 And I think just because they've assumed, well, the gil and generally single threaded, 00:25:48.420 --> 00:25:49.940 at least effectively, 00:25:50.150 --> 00:25:51.520 even if they have threads, so whatever. 00:25:52.020 --> 00:25:53.240 And then they're going to say, oh, 00:25:53.680 --> 00:25:54.080 it's free. 00:25:54.190 --> 00:25:56.560 We got these bug reports from free threading. 00:25:56.660 --> 00:25:57.480 So we're going to add locks. 00:25:57.570 --> 00:25:59.220 And then you're going to have deadlocks in your application. 00:25:59.420 --> 00:26:04.280 I just, I feel like there's going to be a couple of waves of growing pains and I don't know. 00:26:04.580 --> 00:26:04.760 Yeah. 00:26:05.520 --> 00:26:05.800 Yeah. 00:26:05.940 --> 00:26:09.360 And part of that is like intrinsic to the porn in particular, 00:26:09.820 --> 00:26:10.560 you know, there is a lot 00:26:10.560 --> 00:26:13.220 of jokes in the internet about like threads being complicated. 00:26:13.700 --> 00:26:17.500 And I think people in the Python ecosystem have not been exposed to what that really 00:26:17.660 --> 00:26:18.920 means because sure. 00:26:19.020 --> 00:26:19.080 Yeah. 00:26:19.190 --> 00:26:22.340 I mean, you, you, you have some problems with threads in Python, but like most of the real 00:26:22.440 --> 00:26:23.980 problems are not really exposed to you. 00:26:24.500 --> 00:26:25.180 While now, 00:26:25.680 --> 00:26:27.400 you know, there is two problems that you need to suffer. 00:26:27.460 --> 00:26:28.240 One of them is like, 00:26:28.640 --> 00:26:28.760 you know, 00:26:29.180 --> 00:26:32.860 you are not tall enough to do multiple threading, which was the joke, 00:26:33.020 --> 00:26:33.080 right? 00:26:33.480 --> 00:26:39.240 And then the other one is that, you know, free thread may be buggy just because there's bugs in the interpreter, 00:26:39.440 --> 00:26:39.920 hopefully not, 00:26:40.120 --> 00:26:42.200 but also maybe bugs in the packages, 00:26:42.500 --> 00:26:42.580 right? 00:26:43.139 --> 00:26:51.280 Or like the other problem that we can see a lot of people running into sometimes is that it doesn't scale linearly. 00:26:51.660 --> 00:26:52.240 Like it's not magic. 00:26:52.460 --> 00:26:52.580 Again, 00:26:52.780 --> 00:26:54.380 this is a matter of threading, it's not easy. 00:26:54.980 --> 00:27:00.440 and sometimes you know the cost of not crashing and the multiple threads is that extra checks 00:27:00.500 --> 00:27:04.940 need to be done and those extra checks make your use of multiple threading not work and 00:27:05.180 --> 00:27:11.640 people are not getting used to profile that thing as well so you know like it's very hard to 00:27:11.800 --> 00:27:15.840 understand why your algorithm that should be scaling with number of threads is not scaling 00:27:16.540 --> 00:27:20.659 because now you need different tools and not even the tool that we're going to talk about today 00:27:20.680 --> 00:27:25.860 works in that case right so you need to like people people need to be really careful with that i like 00:27:25.860 --> 00:27:30.760 your analogy of like yeah you're using threadings but you've kind of been sheltered it's like oh yeah 00:27:30.960 --> 00:27:37.300 i i traversed the desert car versus on a bicycle right it's yeah you it's not the same experience 00:27:37.700 --> 00:27:43.180 of going through the desert in a car like oh it is hot exactly exactly yeah yeah yeah and people 00:27:43.250 --> 00:27:48.499 don't know you know they don't know how fresh that the car was when they were towards in the desert 00:27:48.580 --> 00:27:58.860 right uh but the deal it made so everything so easy right with uh right is your ac ac in your uh 00:27:59.020 --> 00:28:06.480 onda you know desert on that all right exactly yeah all right so so then let's talk profiling so 00:28:06.820 --> 00:28:12.860 i think maybe a quick history of profilers in python we've got the more advanced sort of external ones 00:28:13.020 --> 00:28:17.540 like memory and so on but let's let's just focus on what i always thought was a little bit of a 00:28:17.320 --> 00:28:20.260 weird distinction that there's two profilers, 00:28:20.680 --> 00:28:23.520 the C profile and the just profile in Python. 00:28:23.800 --> 00:28:23.900 Yeah. 00:28:24.360 --> 00:28:26.200 I just pick one, but tell us, 00:28:26.540 --> 00:28:28.060 this is what we've had for quite a while. 00:28:28.220 --> 00:28:29.780 I've done a lot with these. 00:28:30.340 --> 00:28:30.460 So, 00:28:31.220 --> 00:28:35.100 yeah, this is like, it's kind of weird to say today where like, 00:28:35.260 --> 00:28:35.400 you know, 00:28:35.620 --> 00:28:36.040 so many 00:28:36.120 --> 00:28:39.480 people are using Python and Python is so big that what we're going to explain here is a 00:28:39.480 --> 00:28:39.900 bit like, 00:28:40.180 --> 00:28:40.380 you know, 00:28:41.060 --> 00:28:41.860 it sounds strange, 00:28:42.180 --> 00:28:44.540 but so it used to be a time where like, 00:28:44.820 --> 00:28:44.920 you know, 00:28:44.980 --> 00:28:47.340 the bar for the modules was much lower. 00:28:48.190 --> 00:28:50.220 So the profile module, 00:28:50.380 --> 00:28:51.900 I mean, you know, it's a different 00:28:52.140 --> 00:28:52.200 time. 00:28:52.400 --> 00:28:53.880 I'm not saying I will not do that, 00:28:54.060 --> 00:28:56.440 right? But like the quality of the tools that we're using 00:28:56.610 --> 00:28:56.720 today, 00:28:56.960 --> 00:28:58.140 not just profilers in general, 00:28:58.320 --> 00:29:00.720 it's under much more scrutiny because, 00:29:01.020 --> 00:29:01.540 you know, it serves 00:29:01.590 --> 00:29:04.240 more people and those people have like more demanding workflows. 00:29:04.390 --> 00:29:04.860 And this is true, 00:29:05.040 --> 00:29:05.140 right? 00:29:05.200 --> 00:29:09.240 The software that was written long, long ago had less complexity to deal with because, 00:29:09.520 --> 00:29:09.620 you know, 00:29:10.040 --> 00:29:14.340 computers were not that, you know, powerful and like the stack was like saluer, 00:29:14.520 --> 00:29:14.600 right? 00:29:14.980 --> 00:29:19.300 So the profile module was the first attempt that people tried to do profiling in Python. 00:29:19.860 --> 00:29:21.680 It's a very good attempt, so like, oh, it's actually good. 00:29:21.760 --> 00:29:23.140 The problem is architecturally speaking, 00:29:23.300 --> 00:29:24.940 it's not really a good profiler in general. 00:29:25.180 --> 00:29:30.180 The reason is because it's written in Python and it runs inside application that is being profiled. 00:29:30.420 --> 00:29:37.340 So you don't need to be a profiler expert to know that you're doing extra work to measure the work that is going to skew the results, right? 00:29:37.480 --> 00:29:39.600 Because now you're not measuring your application. 00:29:39.800 --> 00:29:41.920 You're measuring your application and the profiler at the same time, 00:29:42.120 --> 00:29:42.240 right? 00:29:42.640 --> 00:29:44.180 So that was bad. 00:29:44.300 --> 00:29:50.540 And then this profiler module kind of like does a bunch of tricks to try to measure how much itself is contributing to the problem. 00:29:50.690 --> 00:29:53.200 But like you can imagine how flaky that is. 00:29:54.080 --> 00:29:59.160 And then the other problem is that it makes your application much slower because it's being measured, right? 00:29:59.260 --> 00:30:01.100 And the other slowness was like huge. 00:30:01.280 --> 00:30:04.080 Like we're talking about like 20 times slower and things like that. 00:30:04.130 --> 00:30:04.500 Or like, 00:30:04.760 --> 00:30:04.920 you know, 00:30:05.180 --> 00:30:05.860 depending on what you're doing. 00:30:06.380 --> 00:30:08.280 So part of the reason as well, again, 00:30:08.460 --> 00:30:10.300 is because the profiler itself was written in Python. 00:30:10.640 --> 00:30:11.120 So perhaps, 00:30:11.820 --> 00:30:12.000 you know, 00:30:12.260 --> 00:30:16.180 that was not a great experience. 00:30:16.660 --> 00:30:17.960 So the C profile module, 00:30:18.320 --> 00:30:18.400 which, 00:30:19.120 --> 00:30:21.120 you know, is very explicit about what is a trick, 00:30:21.440 --> 00:30:21.700 but like, 00:30:21.960 --> 00:30:22.120 unfortunately, 00:30:22.900 --> 00:30:27.500 it's a leaky name because like the C in the C profile is the C language, right? 00:30:27.700 --> 00:30:28.220 And also, 00:30:29.100 --> 00:30:34.580 it has one of the biggest problems that I don't like this module is that the C is lowercase and the P is uppercase. 00:30:34.680 --> 00:30:36.520 And the name of the stupid module is like this. 00:30:36.700 --> 00:30:38.480 So writing the thing is very annoying. 00:30:39.380 --> 00:30:41.820 I always forget which one is uppercase and lowercase. 00:30:41.980 --> 00:30:42.060 Anyway, 00:30:42.320 --> 00:30:46.220 so the C profile is basically a re-implementation of the profile module in C, 00:30:46.540 --> 00:30:47.300 the illegal language. 00:30:47.620 --> 00:30:50.560 And then many people don't know why I say this. 00:30:50.620 --> 00:30:55.760 This is because the White House released this hyper thing that C was now dangerous because, 00:30:55.940 --> 00:30:57.120 you know, it's like memory unsafe. 00:30:57.480 --> 00:31:03.540 So I find very funny that the White House said that C is illegal and CPython is, 00:31:04.360 --> 00:31:05.060 we are all, 00:31:05.700 --> 00:31:08.700 you know, like bandits writing in the illegal language. 00:31:08.720 --> 00:31:08.760 Exactly. 00:31:09.380 --> 00:31:09.740 Yeah, yeah. 00:31:10.050 --> 00:31:10.140 Okay, 00:31:10.340 --> 00:31:10.860 so- 00:31:10.860 --> 00:31:11.620 Living underground, 00:31:12.220 --> 00:31:16.620 hiding from everything with your GCC and your clang. 00:31:17.100 --> 00:31:17.460 Exactly. 00:31:17.640 --> 00:31:19.460 Give food to your core developer underground. 00:31:19.860 --> 00:31:20.080 Yeah, yeah. 00:31:20.220 --> 00:31:25.060 At least Seth and team were able to get Python on the good list. 00:31:25.440 --> 00:31:26.120 Oh, yeah, yeah, yeah. 00:31:26.820 --> 00:31:28.180 We both like that also. 00:31:28.220 --> 00:31:28.600 Memory-safe. 00:31:29.140 --> 00:31:32.100 And I was like, everybody's like, yeah, yeah, of course, Python is memory-safe. 00:31:32.180 --> 00:31:32.780 And I was like thinking, 00:31:33.150 --> 00:31:33.380 is it? 00:31:33.450 --> 00:31:34.380 They're like, I don't know, man. 00:31:35.820 --> 00:31:37.640 I see a lot of bugs, but like, okay, 00:31:37.820 --> 00:31:39.040 okay, we are allowed to have bugs, 00:31:39.240 --> 00:31:41.220 you know, like Rust as well can have bugs and crash, 00:31:41.560 --> 00:31:41.800 maybe less, 00:31:42.040 --> 00:31:42.140 but, 00:31:42.460 --> 00:31:42.660 you know, 00:31:43.100 --> 00:31:46.380 in theory, the language is very safe if you assume that there's no bugs. 00:31:46.820 --> 00:31:46.900 Anyway, 00:31:47.890 --> 00:31:51.900 so C profile is this like implementation of the whole tree bank in C. 00:31:52.300 --> 00:31:54.400 So it's faster than profile, 00:31:54.700 --> 00:31:54.920 actually, 00:31:55.100 --> 00:31:56.180 it's 10 times faster or so. 00:31:56.700 --> 00:31:57.440 But it still, 00:31:57.740 --> 00:31:57.800 again, 00:31:57.980 --> 00:31:58.740 has the same problem, 00:31:59.000 --> 00:32:01.940 just less. So it makes your application still two or three times slower, 00:32:02.720 --> 00:32:04.500 something like that, which is already huge, right? 00:32:04.620 --> 00:32:06.900 Like thinking that your application takes five minutes, 00:32:07.060 --> 00:32:08.060 now it takes 15 minutes, right? 00:32:08.100 --> 00:32:08.760 So not great. 00:32:09.640 --> 00:32:13.160 And also the problem is that it also runs in the process. 00:32:13.680 --> 00:32:15.080 So I don't want to get the impression 00:32:15.180 --> 00:32:16.160 that this is just negative. 00:32:17.100 --> 00:32:18.700 So because now Cprofile is actually, 00:32:19.340 --> 00:32:20.380 it's well architecture, 00:32:21.540 --> 00:32:22.300 it's now in C, 00:32:22.460 --> 00:32:23.260 so it's all good. 00:32:23.760 --> 00:32:26.020 And now the problems in the slowdowns 00:32:26.160 --> 00:32:28.220 are an inherited property of these kind of tools, 00:32:28.440 --> 00:32:28.520 right? 00:32:28.580 --> 00:32:30.100 Because that run inside the profiler. 00:32:30.260 --> 00:32:31.840 These are called tracing profilers. 00:32:32.320 --> 00:32:33.980 The reason they are called tracing profilers 00:32:34.000 --> 00:32:36.900 because they see every single call that is happening, 00:32:37.150 --> 00:32:37.260 right? 00:32:37.440 --> 00:32:39.340 So every time you call a function, they see that 00:32:39.370 --> 00:32:40.360 and they need to record that. 00:32:40.460 --> 00:32:41.900 So it's a lot of work to do, 00:32:42.720 --> 00:32:45.060 but also because they run in the process, 00:32:45.250 --> 00:32:46.080 every time you call a function, 00:32:46.220 --> 00:32:47.100 now you do this extra work 00:32:47.350 --> 00:32:49.680 and then therefore your thing is worse. 00:32:49.980 --> 00:32:51.180 The bright side, of course, 00:32:51.620 --> 00:32:52.600 is that you collect everything. 00:32:53.060 --> 00:32:54.300 So if you want to know how many times 00:32:54.370 --> 00:32:55.660 your progress is calling, 00:32:56.220 --> 00:32:57.260 send money to employees, 00:32:57.840 --> 00:32:59.600 you will know how much money are you selling to employees. 00:32:59.880 --> 00:33:00.840 You are not missing it. 00:33:01.220 --> 00:33:03.600 If it says seven times, it was called seven times. 00:33:03.840 --> 00:33:06.140 Yeah, it wasn't just a statistical analysis, 00:33:06.380 --> 00:33:06.440 yeah. 00:33:06.600 --> 00:33:06.860 Exactly, 00:33:07.040 --> 00:33:07.140 exactly. 00:33:07.220 --> 00:33:08.980 So you can tell your employees that were not paid 00:33:09.080 --> 00:33:09.940 that that was not the reason, 00:33:10.140 --> 00:33:10.200 right? 00:33:10.340 --> 00:33:12.680 So in that case, then, 00:33:13.040 --> 00:33:13.840 it's really good for that. 00:33:13.980 --> 00:33:15.300 The same way Membray, for example, 00:33:15.520 --> 00:33:15.900 if you recall, 00:33:16.480 --> 00:33:17.820 Membray is also a tracing profiler 00:33:18.160 --> 00:33:21.680 because, you know, imagine that you lose a one gigabyte allocation. 00:33:21.980 --> 00:33:22.860 Can you imagine that, oh, 00:33:23.020 --> 00:33:24.040 well, we just lose that one. 00:33:24.220 --> 00:33:24.500 And, like, 00:33:24.860 --> 00:33:26.560 you know, I suppose your application is fine. 00:33:26.820 --> 00:33:27.640 Like, you don't want that, right? 00:33:27.700 --> 00:33:30.240 Like, you want Membray to tell you exactly everything that happened. 00:33:30.400 --> 00:33:31.340 In time, it's different, right? 00:33:31.420 --> 00:33:33.100 because like we will talk about that in a second, 00:33:33.640 --> 00:33:34.280 Laszlo will explain, 00:33:34.460 --> 00:33:35.500 but, you know, 00:33:35.720 --> 00:33:35.960 here, 00:33:36.360 --> 00:33:39.100 that's the advantage of Zprofile still even today 00:33:39.390 --> 00:33:39.500 because, 00:33:39.780 --> 00:33:39.980 you know, 00:33:40.290 --> 00:33:41.680 perhaps people will hear today's episode 00:33:41.750 --> 00:33:41.940 and say, 00:33:42.100 --> 00:33:43.760 well, Zprofile is not good anymore. 00:33:43.920 --> 00:33:44.660 We should throw it away, 00:33:44.860 --> 00:33:46.640 but it still has its own use cases. 00:33:47.120 --> 00:33:48.720 So that is, I think, a brief story 00:33:48.920 --> 00:33:50.440 of like why we have to... 00:33:50.580 --> 00:33:51.660 Laszlo, what do you want to add to that? 00:33:52.060 --> 00:33:54.100 Yeah, so I think it's definitely, 00:33:54.230 --> 00:33:57.340 I think, a problem with the ergonomics as well. 00:33:57.480 --> 00:33:58.420 And if you're a developer 00:33:58.510 --> 00:34:00.180 and you want to provide a Python application, 00:34:00.480 --> 00:34:02.280 And I think if you look at the documentation, 00:34:03.040 --> 00:34:03.140 I, 00:34:03.340 --> 00:34:04.280 when I first looked at this, 00:34:04.840 --> 00:34:05.760 found it very confusing. 00:34:06.240 --> 00:34:08.480 How do I decide whether it's Cprofile or Profile, right? 00:34:08.580 --> 00:34:11.960 And then I think it just became sort of the defacto standard. 00:34:12.050 --> 00:34:13.580 You just always use Cprofile, right? 00:34:14.320 --> 00:34:15.440 Why exactly it's unclear, 00:34:15.820 --> 00:34:15.919 right? 00:34:16.190 --> 00:34:20.440 So I think, but I also think that there haven't been, 00:34:20.820 --> 00:34:23.360 like CPI didn't have obviously something profiling 00:34:23.450 --> 00:34:25.600 and it didn't have powerful profiling tools. 00:34:25.740 --> 00:34:28.179 So you always had to look for external tools 00:34:28.200 --> 00:34:32.659 to get actually like higher quality sort of profiling results. 00:34:32.770 --> 00:34:33.260 And of course, 00:34:33.500 --> 00:34:37.100 there's an entire family of situations 00:34:37.190 --> 00:34:39.419 where you cannot use a tracing profiler 00:34:39.860 --> 00:34:41.780 because effectively for a tracing profiler, 00:34:41.790 --> 00:34:44.120 you have to reconstruct a small sample test. 00:34:44.620 --> 00:34:47.240 You cannot run that on your application, 00:34:47.580 --> 00:34:47.639 right? 00:34:47.800 --> 00:34:49.440 Because it would just be too slow. 00:34:50.300 --> 00:34:51.120 It's too time consuming. 00:34:51.230 --> 00:34:53.760 So generally you would just have a test case or something 00:34:53.950 --> 00:34:54.780 and run it on that. 00:34:55.060 --> 00:34:56.480 Yeah, the worst of that is 00:34:56.500 --> 00:34:58.020 It doesn't work great in production, 00:34:58.580 --> 00:35:02.220 but I profiled it locally and it seems okay. 00:35:02.520 --> 00:35:03.580 So what's going on, right? 00:35:03.900 --> 00:35:04.040 Exactly. 00:35:04.700 --> 00:35:07.420 You can't really reproduce the exact same situation oftentimes. 00:35:07.800 --> 00:35:12.220 And then you're basically left with no options or you have to start downloading, 00:35:12.680 --> 00:35:14.480 finding some third-party option. 00:35:14.760 --> 00:35:20.260 I can't, Pablo, I got to circle back to this a little bit because I think it's really interesting. 00:35:20.340 --> 00:35:25.180 So you talked about the tracing profilers, which C profile and profile is, 00:35:25.420 --> 00:35:26.560 and how it makes, 00:35:26.830 --> 00:35:31.860 they're very accurate, but they make a big impact on basically what is happening with your code. 00:35:32.120 --> 00:35:34.880 And it reminds me of quantum mechanics a little bit. 00:35:35.230 --> 00:35:35.360 Wow. 00:35:35.920 --> 00:35:36.580 Let's go. 00:35:36.700 --> 00:35:37.060 Let's go. 00:35:37.160 --> 00:35:39.080 It's like there might've been a function. 00:35:39.580 --> 00:35:40.320 Let's take two functions. 00:35:40.520 --> 00:35:43.420 They each take half a second in practice on their own, 00:35:43.620 --> 00:35:43.720 right? 00:35:43.940 --> 00:35:48.220 One of them has to do something with a file system a million times in a loop. 00:35:48.320 --> 00:35:50.100 Another one of them calls an API, 00:35:50.680 --> 00:35:52.000 which takes half a second to return. 00:35:52.280 --> 00:35:53.980 You run that in a tracing profiler. 00:35:54.320 --> 00:35:56.620 The one that calls the API is barely changed. 00:35:57.040 --> 00:35:58.080 It's tracing one call, 00:35:58.480 --> 00:35:58.680 sort of, 00:35:58.960 --> 00:35:59.040 right? 00:35:59.300 --> 00:36:00.180 You do the other one. 00:36:00.500 --> 00:36:05.600 It's now a million plus little intercepts of like a function call over and over and over. 00:36:05.720 --> 00:36:09.260 And it makes that one that's chatty in Python seem really, 00:36:09.420 --> 00:36:09.880 really slow. 00:36:10.220 --> 00:36:12.000 By observing it, you've changed it. 00:36:12.080 --> 00:36:12.660 You know what I mean? 00:36:12.900 --> 00:36:13.160 Yeah, 00:36:13.340 --> 00:36:13.540 yeah, yeah. 00:36:13.560 --> 00:36:14.020 In this sense. 00:36:15.300 --> 00:36:17.740 I'm very sad to report that, unfortunately, 00:36:18.060 --> 00:36:19.400 because you're talking to a physicist, 00:36:19.620 --> 00:36:20.780 I'm going to be a pain here. 00:36:21.080 --> 00:36:22.320 Do it. Let's do it. Let's get it right. 00:36:22.700 --> 00:36:25.700 I'm going to put my pedantic hat here. 00:36:27.080 --> 00:36:27.700 So, yes, 00:36:27.880 --> 00:36:28.280 you're right. 00:36:28.620 --> 00:36:29.000 Also, 00:36:31.000 --> 00:36:32.700 technically speaking, it's not really correct. 00:36:33.020 --> 00:36:33.660 The reason is because, 00:36:33.980 --> 00:36:36.320 yes, that is indeed what happens in quantum mechanics. 00:36:36.440 --> 00:36:36.880 When you observe, 00:36:37.120 --> 00:36:38.080 you modify the system. 00:36:38.260 --> 00:36:39.300 That is correct. That part is correct. 00:36:39.760 --> 00:36:44.720 But the reason the metaphor kind of like doesn't fully work is because the important part of quantum mechanics 00:36:44.840 --> 00:36:45.420 is that before, 00:36:45.780 --> 00:36:46.860 if this was quantum mechanics, 00:36:47.230 --> 00:36:49.860 before you actually measure your program, 00:36:50.320 --> 00:36:52.660 your program is both fast and slow at the same time. 00:36:52.940 --> 00:36:55.160 But that's the key, 00:36:55.320 --> 00:36:55.380 right? 00:36:55.780 --> 00:36:58.100 We got the cat in the box as the profiler now. 00:36:58.430 --> 00:36:59.000 Yeah, right, right. 00:36:59.290 --> 00:36:59.380 Here, 00:36:59.640 --> 00:37:00.540 the profiler, 00:37:01.140 --> 00:37:02.440 sorry, your program was the cat. 00:37:02.770 --> 00:37:03.780 So your cat is dead, 00:37:04.120 --> 00:37:04.220 right? 00:37:04.360 --> 00:37:06.080 And then you say, well, 00:37:06.320 --> 00:37:06.540 you know, 00:37:06.930 --> 00:37:10.980 I can move the box and then suddenly, 00:37:11.180 --> 00:37:11.340 you know, 00:37:11.420 --> 00:37:13.640 I open the box and it's dead, right? 00:37:13.760 --> 00:37:15.400 So you say, well, you modify the system. 00:37:15.940 --> 00:37:16.360 But here, 00:37:16.780 --> 00:37:16.900 you know, 00:37:17.640 --> 00:37:18.520 for being quantum mechanics, 00:37:18.800 --> 00:37:19.200 you, 00:37:19.670 --> 00:37:20.800 I mean, it would be a very good excuse 00:37:20.990 --> 00:37:23.100 to tell your boss and then say, no, no, no, you don't understand. 00:37:23.420 --> 00:37:25.440 This program is fast and slow at the same time. 00:37:25.760 --> 00:37:27.520 If we measure, we would screw it up, 00:37:27.760 --> 00:37:27.920 you know? 00:37:28.020 --> 00:37:28.240 So like, 00:37:28.610 --> 00:37:30.260 we must not measure the program. 00:37:31.240 --> 00:37:31.700 Well, here, 00:37:31.960 --> 00:37:35.240 of course, you can imagine that what happens here is that the program is either 00:37:35.460 --> 00:37:36.220 fast or slow. 00:37:36.940 --> 00:37:39.900 And what happens is that when you measure, you are making it much slower. 00:37:40.210 --> 00:37:40.300 So, 00:37:40.790 --> 00:37:42.600 but it was already fast or slow. 00:37:42.660 --> 00:37:43.940 It's just you're making it worse. 00:37:44.050 --> 00:37:46.040 So it was slow, now it's very slow. 00:37:46.340 --> 00:37:47.740 And if it's less fast, now it's slow. 00:37:48.020 --> 00:37:49.240 But it's not like, oh, 00:37:49.460 --> 00:37:51.900 wow, it was in this superposition of states. 00:37:52.360 --> 00:37:53.820 I know I'm not a fan person, 00:37:53.990 --> 00:37:56.280 so don't feel too bad. 00:37:56.380 --> 00:37:57.240 It's still a good metaphor. 00:37:58.600 --> 00:38:04.060 My warning is that the tracing profilers make it unevenly slower. 00:38:04.740 --> 00:38:05.820 Some parts get away. 00:38:06.120 --> 00:38:11.240 Like parts that do a lot of internal in Python function calls are affected a lot. 00:38:11.360 --> 00:38:16.700 stuff that works with external systems that don't require much execution of python but themselves 00:38:16.890 --> 00:38:23.100 are slow are not very affected by tracing profilers that was my warning my my quantum mechanics 00:38:23.380 --> 00:38:27.460 analogy wasn't perfect i understand so let's try this sorry sorry sorry no what about string theory 00:38:28.359 --> 00:38:34.580 no no i'm just kidding let's let's talk about let's talk about the peps let's talk about what's 00:38:34.600 --> 00:38:35.760 coming here. 00:38:36.060 --> 00:38:36.380 Oh, 00:38:36.630 --> 00:38:36.940 right, right. 00:38:37.290 --> 00:38:39.500 And we got the PEP 799, 00:38:39.900 --> 00:38:41.500 a dedicated profiling package 00:38:41.700 --> 00:38:45.140 for organizing Python profiling tools by the two of you. 00:38:45.400 --> 00:38:45.540 Awesome. 00:38:45.980 --> 00:38:46.360 Tell us about this. 00:38:46.580 --> 00:38:48.780 And then we're going to talk about 768, 00:38:48.980 --> 00:38:51.680 which is some of the actual changes. 00:38:52.130 --> 00:38:52.240 Yeah. 00:38:52.240 --> 00:38:54.240 So how did we start the back story, 00:38:54.480 --> 00:38:54.620 Pablo, 00:38:54.840 --> 00:38:56.720 that I think we actually were planning 00:38:56.790 --> 00:38:59.440 to put the something profiler in the profile package, 00:38:59.800 --> 00:38:59.960 right? 00:39:00.070 --> 00:39:02.300 I think that was the original conning plan. 00:39:02.540 --> 00:39:05.660 And then it turned out we cannot do that. 00:39:06.240 --> 00:39:07.340 And that led to this path. 00:39:08.240 --> 00:39:09.040 Because basically, 00:39:09.440 --> 00:39:10.740 let's do the right thing, right? 00:39:10.920 --> 00:39:11.740 Let's clean up this. 00:39:12.260 --> 00:39:14.400 I don't want to say mess, but let's clean things up. 00:39:14.560 --> 00:39:18.240 And that would mean that you're going to have a single tracing profiler. 00:39:19.100 --> 00:39:20.760 Eventually, we can talk about how that's going to go. 00:39:20.840 --> 00:39:22.640 And then you're going to have one sampling profiler. 00:39:22.740 --> 00:39:25.480 And this is all going to be grouped into the profiling package, 00:39:25.800 --> 00:39:28.640 which should provide much better sort of ergonomics. 00:39:28.720 --> 00:39:31.080 It should make it much easier for developers to decide 00:39:31.100 --> 00:39:32.820 what tool they should reach for. 00:39:32.820 --> 00:39:34.480 And it also means that there are not going to be 00:39:34.660 --> 00:39:36.080 two different tracing profilers, 00:39:36.180 --> 00:39:37.180 which makes it difficult to, 00:39:37.740 --> 00:39:37.880 again, 00:39:38.020 --> 00:39:39.220 like to choose the right thing. 00:39:39.360 --> 00:39:41.820 So profiling the tracing effectively becomes, 00:39:42.820 --> 00:39:44.760 that's basically C-profile. 00:39:45.160 --> 00:39:47.020 And C-profile is going to continue living on 00:39:47.200 --> 00:39:49.740 as the C-profile import as well for compatibility. 00:39:51.080 --> 00:39:51.980 But profile, 00:39:52.240 --> 00:39:54.540 so that the old Python-only version 00:39:54.900 --> 00:39:59.180 is effectively deprecated and will be removed eventually. 00:39:59.680 --> 00:40:01.640 And then profiling with sampling is the new profiler, 00:40:01.860 --> 00:40:02.080 Tachium. 00:40:02.380 --> 00:40:03.220 It makes it super obvious. 00:40:03.740 --> 00:40:07.680 You just import profiling.tracing or profiling.sampling. 00:40:08.080 --> 00:40:09.980 If you understand the nomenclature of profiling, 00:40:10.140 --> 00:40:11.580 it tells you exactly what you're getting, right? 00:40:11.840 --> 00:40:13.880 Yeah, it gets a bit long with time, 00:40:14.140 --> 00:40:14.240 right? 00:40:14.420 --> 00:40:15.600 Because like you mostly, 00:40:16.040 --> 00:40:16.640 as we will discuss, 00:40:16.940 --> 00:40:19.240 mostly all the time you want profiling.sampling. 00:40:19.560 --> 00:40:20.380 So now it's like, 00:40:20.540 --> 00:40:20.640 damn, 00:40:21.220 --> 00:40:22.940 profiling.sampling. 00:40:23.080 --> 00:40:24.660 Like, this is very, very boring. 00:40:25.440 --> 00:40:25.520 But, 00:40:25.720 --> 00:40:28.360 you know, I propose to just call it Tachium. 00:40:28.580 --> 00:40:28.740 Like, 00:40:28.990 --> 00:40:31.160 you know, maybe in 3.16, 00:40:31.500 --> 00:40:35.560 I can convince people to let me have a top level binary name. 00:40:35.960 --> 00:40:38.920 You know, the same way when you install Python in a VAMF, you get, 00:40:39.340 --> 00:40:40.660 so you can get Tachyon. 00:40:41.080 --> 00:40:42.300 Like, we'll see, we'll see. 00:40:43.440 --> 00:40:44.480 Maybe you should, 00:40:44.680 --> 00:40:44.960 sorry, 00:40:45.200 --> 00:40:48.740 maybe you should propose to have a C-Tachyon and a Tachyon module. 00:40:49.280 --> 00:40:49.800 Confuse everyone. 00:40:50.530 --> 00:40:51.580 Oh, C-Tachyon. 00:40:51.880 --> 00:40:52.940 But Tachyon is C already. 00:40:53.400 --> 00:40:55.940 And then just make a compromise to have just Tachyon, 00:40:56.260 --> 00:40:56.340 right? 00:40:56.920 --> 00:40:57.240 No, no, no. 00:40:57.360 --> 00:41:00.040 Because Tachyon is C, then it will be P-Python in Python. 00:41:00.310 --> 00:41:02.280 Like a pure Python implementation of Tachyon. 00:41:02.480 --> 00:41:02.940 P-Python. 00:41:04.020 --> 00:41:05.480 I've got a fix for you, Pablo. 00:41:06.060 --> 00:41:08.760 Just create a Python package called Tachyon. 00:41:09.060 --> 00:41:10.060 Oh, already exists, 00:41:10.220 --> 00:41:10.320 man. 00:41:10.390 --> 00:41:14.000 And then all it does is import profiling, 00:41:14.280 --> 00:41:14.780 tracing, 00:41:15.570 --> 00:41:17.780 or sampling as Tachyon. 00:41:18.120 --> 00:41:18.840 But apparently we cannot. 00:41:18.840 --> 00:41:19.980 And then you'll be exit. 00:41:20.170 --> 00:41:20.960 Yeah, you're good to go. 00:41:21.280 --> 00:41:27.060 Okay, or I can bribe Charlie Marsh that they add UB-Tachyon. 00:41:27.300 --> 00:41:28.340 You know, like uv profile. 00:41:29.090 --> 00:41:29.660 UV profile? 00:41:30.000 --> 00:41:35.160 How many megadollars, how many megadollars are needed for this? 00:41:35.360 --> 00:41:36.020 Like, we will see. 00:41:36.540 --> 00:41:36.920 Open AI. 00:41:37.360 --> 00:41:38.040 Sam Alman. 00:41:38.180 --> 00:41:38.560 Make it happen, 00:41:38.730 --> 00:41:38.820 yes. 00:41:39.010 --> 00:41:39.680 Allow this to happen. 00:41:40.060 --> 00:41:40.380 Yes, yes. 00:41:41.100 --> 00:41:43.320 You will be so much popular on the Anthropics, those guys. 00:41:44.040 --> 00:41:45.320 You just need a little prompt injection. 00:41:45.370 --> 00:41:45.960 You'll be good to go. 00:41:46.080 --> 00:41:46.960 Yes, exactly, exactly. 00:41:47.960 --> 00:41:48.300 All right, so, 00:41:48.600 --> 00:41:53.060 Oslo, you mentioned the old Python profile module 00:41:53.480 --> 00:41:55.460 is under deprecation warning. 00:41:56.500 --> 00:41:58.960 And that will be in 15 and 16. 00:41:59.300 --> 00:42:00.580 And actually two years from now, 00:42:00.920 --> 00:42:01.460 two years from now, 00:42:01.620 --> 00:42:02.140 plus a month, 00:42:02.820 --> 00:42:03.500 it will be out. 00:42:03.700 --> 00:42:05.160 It will be removed in 317. 00:42:05.560 --> 00:42:06.680 So just heads up for people. 00:42:07.380 --> 00:42:07.700 Obliterated. 00:42:08.240 --> 00:42:08.740 That's right. 00:42:08.960 --> 00:42:11.340 It will be recated into history for real. 00:42:11.440 --> 00:42:11.980 Oblivion. 00:42:12.300 --> 00:42:12.580 Oblivion. 00:42:12.940 --> 00:42:12.980 Okay. 00:42:13.100 --> 00:42:16.400 So this is really just kind of an organizational deal, right? 00:42:16.520 --> 00:42:16.900 This PEP. 00:42:17.180 --> 00:42:17.300 Okay. 00:42:17.700 --> 00:42:19.720 And then we have PEP 768, 00:42:20.180 --> 00:42:21.720 which sort of makes this possible, 00:42:21.880 --> 00:42:22.000 right? 00:42:22.080 --> 00:42:23.360 It makes Tachyon, 00:42:23.800 --> 00:42:24.720 the new profiler, 00:42:24.960 --> 00:42:25.480 accessible, 00:42:25.780 --> 00:42:25.840 right? 00:42:26.100 --> 00:42:26.360 Yes. 00:42:26.540 --> 00:42:29.060 So the original of that PEP was to, 00:42:30.120 --> 00:42:33.000 that's the previous work we did with my colleagues from Bloomberg, 00:42:33.160 --> 00:42:33.760 Ivana and Matt. 00:42:34.040 --> 00:42:37.080 And the idea here is that we were chasing this goal, 00:42:37.240 --> 00:42:40.840 like North Star, of allowing PDB, the Python debugger, 00:42:41.040 --> 00:42:44.920 and technically any other debugger, but also the one that we ship in the standard library. 00:42:45.380 --> 00:42:48.000 So we wanted that to be able to attach to their programs. 00:42:48.200 --> 00:42:51.560 So you could have a program running in production that is doing weird stuff. 00:42:51.900 --> 00:42:55.560 And instead of having to stop it and then running under the debugger, 00:42:55.860 --> 00:42:56.780 you could just say, hey, 00:42:57.160 --> 00:42:57.380 you know, 00:42:57.620 --> 00:42:58.500 touch to that program 00:42:58.680 --> 00:42:59.760 and like, tell me what's wrong, 00:42:59.920 --> 00:43:00.000 right? 00:43:00.000 --> 00:43:00.920 So we wanted that. 00:43:01.120 --> 00:43:01.720 And then we added, 00:43:01.940 --> 00:43:02.420 therefore, 00:43:02.700 --> 00:43:03.400 to achieve this, 00:43:03.780 --> 00:43:06.680 we added all these capabilities to CPython 00:43:07.040 --> 00:43:08.760 such that you could attach to a running process 00:43:09.000 --> 00:43:10.860 and then you could inspect the running process, right? 00:43:10.920 --> 00:43:12.720 Because the debugger needs to inspect the running process. 00:43:12.940 --> 00:43:14.860 So we have all this machinery there. 00:43:15.200 --> 00:43:15.320 And, 00:43:15.540 --> 00:43:18.280 you know, that was this PEP and that's shipping 3.14. 00:43:19.000 --> 00:43:22.180 And 3.14 has a bunch of utilities that use this machinery 00:43:22.820 --> 00:43:25.740 and allow you to attach with PDV and much more, 00:43:25.920 --> 00:43:25.980 right? 00:43:26.220 --> 00:43:29.240 But then we were at PyCon US 2025, 00:43:30.080 --> 00:43:31.180 and then I went to Laszlo, 00:43:31.400 --> 00:43:32.380 and then I told Laszlo, 00:43:32.540 --> 00:43:35.460 hey, Laszlo, would it be cool to call this in a loop? 00:43:35.840 --> 00:43:37.300 And then we have a profiler, 00:43:38.120 --> 00:43:40.240 and then Laszlo is like, yes, let's just do it. 00:43:40.240 --> 00:43:41.160 So in the sprint, 00:43:41.520 --> 00:43:43.980 we did this mini program 00:43:44.260 --> 00:43:45.760 that was calling this thing in a loop, 00:43:45.960 --> 00:43:46.040 right? 00:43:46.040 --> 00:43:48.820 So this thing allows you to get the stack of the application, 00:43:49.200 --> 00:43:50.760 kind of, and then at the end, 00:43:50.860 --> 00:43:55.200 the profiler is just doing it very quickly and dump the information somewhere. 00:43:55.380 --> 00:43:57.340 But that is at the point not important. 00:43:57.400 --> 00:44:01.820 The most important part is how fast can you see what the program is doing, right? 00:44:01.900 --> 00:44:03.040 That's what a sampling profiler does. 00:44:03.760 --> 00:44:05.400 And just for reference, 00:44:05.560 --> 00:44:08.080 because many people may not be aware of these tools, 00:44:09.160 --> 00:44:12.280 the minimum, minimum, minimum number that you are looking at 00:44:12.440 --> 00:44:14.580 may be something like 100 times per second, 00:44:15.000 --> 00:44:15.060 right? 00:44:15.120 --> 00:44:18.120 That would be on the realm of useful already. 00:44:18.560 --> 00:44:20.960 And it goes from there to more than that, 00:44:21.120 --> 00:44:21.200 right? 00:44:21.620 --> 00:44:24.380 So we put the loop and then we measure, okay, 00:44:24.560 --> 00:44:27.580 how fast is this loop here? 00:44:27.940 --> 00:44:29.380 And it was like two times per second, 00:44:29.980 --> 00:44:31.020 like two times per second. 00:44:31.100 --> 00:44:33.240 That was how fast it was, 00:44:34.000 --> 00:44:36.460 which as you can imagine is not fantastic, not fantastic. 00:44:36.700 --> 00:44:40.260 So we sat down together and for the course of the sprint, 00:44:40.480 --> 00:44:46.500 we basically moved from two times per second to an illegal number, 00:44:47.420 --> 00:44:49.040 like a month of frames per second, 00:44:49.240 --> 00:44:52.560 which I think even today is more than a million times per second. 00:44:53.220 --> 00:44:53.440 But yeah, 00:44:53.620 --> 00:44:55.960 how was your experience during that time, Lasslo? 00:44:56.280 --> 00:44:59.000 Well, yeah, I remember that we were chatting about this 00:44:59.520 --> 00:45:00.640 in the beginning of the sprints 00:45:00.820 --> 00:45:02.980 and Pablo laid out the plan. 00:45:02.980 --> 00:45:03.780 This is what we're going to do. 00:45:04.040 --> 00:45:06.820 I'm going to figure out how fast this thing is 00:45:07.200 --> 00:45:08.840 and then I will see if I can make it faster 00:45:08.900 --> 00:45:11.160 because it might not be that good yet. 00:45:11.540 --> 00:45:15.660 And you're going to work on building UI around this, the CLI, right? 00:45:15.780 --> 00:45:15.860 Like, 00:45:16.020 --> 00:45:17.380 let's build up all the machinery. 00:45:18.100 --> 00:45:19.900 And then we sat down and probably just said, 00:45:20.020 --> 00:45:21.720 hey, this thing is so very slow. 00:45:22.020 --> 00:45:22.540 This is, like, 00:45:22.920 --> 00:45:23.360 not useful. 00:45:23.820 --> 00:45:26.800 And then we literally spent the entire rest of the sprint working on that. 00:45:26.810 --> 00:45:29.860 I think we actually ended up with no code committed, 00:45:30.540 --> 00:45:30.780 I think, 00:45:30.830 --> 00:45:31.280 during the sprint. 00:45:31.520 --> 00:45:35.000 All of the actual commits went in later on. 00:45:35.320 --> 00:45:38.300 But the sort of going through these motions of, 00:45:38.490 --> 00:45:38.600 like, 00:45:38.960 --> 00:45:41.900 basically every single change, getting a 10x. 00:45:42.070 --> 00:45:44.540 I mean, initially, it was easy to get a 10x improvement 00:45:44.560 --> 00:45:46.500 when you have like two hertz, 00:45:47.420 --> 00:45:48.900 two samples a second, right? 00:45:49.300 --> 00:45:51.600 But then just making it faster and faster. 00:45:51.660 --> 00:45:53.680 And actually I remember that even after that, 00:45:54.840 --> 00:45:57.220 Pablo made a couple of improvements later on 00:45:57.220 --> 00:45:58.820 and maybe we can go into details later, 00:45:58.980 --> 00:46:00.240 but he made a couple of improvements 00:46:00.620 --> 00:46:02.080 which really just made it. 00:46:02.120 --> 00:46:04.140 It was already, I think around 200,000, 00:46:04.340 --> 00:46:06.060 maybe 100,000, 200,000 hertz 00:46:06.280 --> 00:46:08.140 at the end of that on the sprints 00:46:08.300 --> 00:46:10.300 and then still like a 4X, 00:46:10.460 --> 00:46:12.220 5X improvement later on. 00:46:12.260 --> 00:46:12.940 So I think That's 00:46:12.940 --> 00:46:13.200 crazy. 00:46:13.760 --> 00:46:16.740 Yeah, and I think it was a headline later on. 00:46:16.860 --> 00:46:19.820 It's the fastest sampling profiler, 00:46:20.770 --> 00:46:20.900 basically. 00:46:21.440 --> 00:46:21.620 Yeah, 00:46:21.800 --> 00:46:22.740 just to be clear also, 00:46:23.000 --> 00:46:25.480 I think it's a good moment to mention this. 00:46:25.720 --> 00:46:29.860 This is not the first sampling profiler of this kind there is by far, right? 00:46:29.980 --> 00:46:31.500 There is a couple of them. 00:46:32.280 --> 00:46:33.760 There's quite a lot of tooling in this space. 00:46:34.280 --> 00:46:36.840 Perhaps the most famous existing tooling there is 00:46:36.840 --> 00:46:39.100 is another profiler called Pyospy, 00:46:39.720 --> 00:46:42.140 which has been for a while. 00:46:42.220 --> 00:46:45.220 and it's a very good piece of software. 00:46:45.280 --> 00:46:46.040 It's written in Rust. 00:46:46.840 --> 00:46:49.140 And this is quite efficient. 00:46:49.900 --> 00:46:51.480 The thing, so you may ask, 00:46:51.560 --> 00:46:53.960 okay, why to have one in the standard library, 00:46:54.160 --> 00:46:54.260 right? 00:46:54.520 --> 00:46:57.500 So the answer to that is that, first of all, 00:46:58.860 --> 00:47:01.400 unfortunately, PySpy and any other provider of this kind, 00:47:01.520 --> 00:47:03.040 because it doesn't run inside your application, 00:47:03.500 --> 00:47:05.360 the trade-off is that it needs to learn 00:47:05.860 --> 00:47:07.380 how to analyze your application, 00:47:07.680 --> 00:47:07.800 right? 00:47:07.900 --> 00:47:10.360 And that is understanding the Python interpreter. 00:47:10.420 --> 00:47:13.100 But these parts of the Python interpreter are not public, 00:47:13.600 --> 00:47:13.680 right? 00:47:13.760 --> 00:47:14.680 Like we don't expose that. 00:47:14.730 --> 00:47:16.820 It needs to illegally access to them. 00:47:17.140 --> 00:47:22.220 And that means that there is a competing set of circumstances here 00:47:22.450 --> 00:47:23.320 because the core developers, 00:47:23.600 --> 00:47:24.060 first of all, 00:47:24.160 --> 00:47:28.600 we used to not care about this use case because it was not part of our APIs. 00:47:29.130 --> 00:47:31.880 So we keep changing things in ways that make it really, really hard 00:47:32.000 --> 00:47:35.440 for Biospy and other profiles like Austin to catch up with this. 00:47:35.650 --> 00:47:36.460 To the point that, 00:47:36.790 --> 00:47:38.179 for instance, it used to be the case 00:47:38.200 --> 00:47:41.580 that profiling 3.11 was much slower than profiling 3.10 00:47:41.860 --> 00:47:44.340 because 3.11 made changes that make this really, really hard. 00:47:45.060 --> 00:47:46.020 And sometimes, 00:47:46.460 --> 00:47:48.640 I was keeping an eye to help these tools, 00:47:48.700 --> 00:47:50.340 but sometimes people were making changes 00:47:50.920 --> 00:47:52.420 that were making it impossible to profile 00:47:52.700 --> 00:47:54.460 because it was hiding this so much 00:47:54.640 --> 00:47:55.920 that you couldn't have access to it. 00:47:56.860 --> 00:47:59.280 So one advantage to have this on the standard library, 00:47:59.380 --> 00:48:03.160 of course, is that now this is something that we offer. 00:48:03.240 --> 00:48:06.959 And this profile is also like the reference implementation 00:48:06.980 --> 00:48:07.860 of such a profiler. 00:48:08.340 --> 00:48:10.460 So if we can do it, then anyone can do it, right? 00:48:10.620 --> 00:48:12.860 So that we ensure that this is a feature 00:48:13.100 --> 00:48:15.000 that Python offers and will never break. 00:48:15.300 --> 00:48:16.100 And if you want to, 00:48:16.260 --> 00:48:16.720 you know, 00:48:16.940 --> 00:48:18.520 make your profiler externally, 00:48:19.100 --> 00:48:21.100 then you just need to reimplement what we do, 00:48:21.340 --> 00:48:21.420 right? 00:48:21.520 --> 00:48:22.660 And then you always have a way. 00:48:22.700 --> 00:48:23.820 You can look at how we do it 00:48:23.840 --> 00:48:24.860 and then you can do it as well. 00:48:25.020 --> 00:48:26.160 So that is very important. 00:48:26.460 --> 00:48:27.640 The second part is that, 00:48:28.040 --> 00:48:28.220 you know, 00:48:28.440 --> 00:48:28.760 as many, 00:48:29.040 --> 00:48:30.100 there's many examples 00:48:30.280 --> 00:48:32.740 and this is a very delicate aspect of it, 00:48:32.780 --> 00:48:34.600 is that having this tool in the standard library, 00:48:34.760 --> 00:48:39.760 we consider now that this was like very important to have it in the set of like batteries included 00:48:39.920 --> 00:48:44.780 right because you know you need to like learn for a external tool like is that external tool going to 00:48:44.840 --> 00:48:50.700 be as with the quality that we expect and is going to like have all the edge cases that can have so 00:48:50.840 --> 00:48:55.740 because we are dealing with internal soft cpython we don't know in real time all the little 00:48:55.980 --> 00:49:00.479 intricacies intricacies of like handling these problems so we want to share that the tools 00:49:00.500 --> 00:49:03.640 that we offer are of the utmost quality that we can provide, 00:49:03.900 --> 00:49:04.000 right? 00:49:04.340 --> 00:49:06.960 Which is to say that it's not that we don't trust the others, 00:49:07.220 --> 00:49:07.300 right? 00:49:07.300 --> 00:49:11.600 But we want to ensure that users at least have a very good reason 00:49:11.860 --> 00:49:15.020 to use something that is built in. 00:49:15.380 --> 00:49:16.380 And then the third reason, 00:49:16.680 --> 00:49:19.700 which is piggybacking on what Laszlo was saying, 00:49:20.140 --> 00:49:24.380 is that at some point, we managed to achieve the same speed 00:49:24.500 --> 00:49:25.800 as any other tool outside, 00:49:26.300 --> 00:49:27.220 including PySpy and Austin, 00:49:27.460 --> 00:49:27.520 right? 00:49:27.820 --> 00:49:31.040 But then because we have an extra trick that they don't have, 00:49:31.380 --> 00:49:32.780 which is that we also control the interpreter, 00:49:33.220 --> 00:49:33.300 right? 00:49:33.360 --> 00:49:34.560 Because this is now in CPython. 00:49:34.820 --> 00:49:39.980 So now we could do changes to the interpreter with the only purpose to ensure the profiling is faster, 00:49:40.420 --> 00:49:40.520 right? 00:49:40.960 --> 00:49:43.400 And also, by the way, this means that anyone can also do that, 00:49:43.580 --> 00:49:43.720 right? 00:49:43.860 --> 00:49:44.600 So now anyone, 00:49:45.240 --> 00:49:46.500 like those other profilers can also, 00:49:46.880 --> 00:49:49.500 you know, now we say that this is the faster Python profiler, 00:49:49.800 --> 00:49:49.940 but, 00:49:50.240 --> 00:49:52.100 you know, I'm pretty sure this is not going to be for long. 00:49:52.820 --> 00:49:53.520 Spoilers in a second. 00:49:53.700 --> 00:49:58.260 but like part of the reason is because anyone can use the tricks is but it's just like we did 00:49:58.580 --> 00:50:02.120 you know we can ensure that those tricks work and those tricks are there so part of the reason we 00:50:02.220 --> 00:50:06.720 could go past what was possible before is because we can cheat a bit and we can like change the 00:50:06.900 --> 00:50:11.680 interpreter such that like it exposes easier information and some of those tricks are really 00:50:11.840 --> 00:50:17.800 really cool and complicated like the way you know it can go faster is really interesting 00:50:18.300 --> 00:50:22.999 perhaps i don't know if we can explain it here but like uh they're quite advanced but like anyone 00:50:23.020 --> 00:50:25.520 can pull those tricks if they know how it works, right? 00:50:25.620 --> 00:50:29.660 It's just that they need to ensure that they implement those things. 00:50:29.960 --> 00:50:30.120 Yeah, 00:50:30.340 --> 00:50:30.700 very interesting. 00:50:31.720 --> 00:50:34.060 I don't know if we actually called out the title, 00:50:34.260 --> 00:50:38.540 but PEP768 is a safe external bugger interface for CPython. 00:50:38.740 --> 00:50:40.980 And debugger means like, hey, what are you doing? 00:50:41.220 --> 00:50:43.700 But that's also kind of a profiling thing of your only purpose 00:50:43.920 --> 00:50:44.740 just to say, what are you doing? 00:50:45.200 --> 00:50:45.640 Where are you? 00:50:45.760 --> 00:50:46.000 Where are you? 00:50:46.060 --> 00:50:46.480 Where are you? 00:50:46.640 --> 00:50:48.320 Like over and over a million times a second, 00:50:48.780 --> 00:50:50.380 whatever frequency you want to run at, right? 00:50:50.680 --> 00:50:50.820 Right. 00:50:51.140 --> 00:50:57.320 There's another interesting aspect where a built-in sampling profiler in CPython is super useful, 00:50:57.470 --> 00:51:01.620 which is that, and I ran into this myself as well, that when you're working on CPython itself, 00:51:02.010 --> 00:51:04.960 you don't have a profiler which you can use to measure performance. 00:51:05.550 --> 00:51:08.400 Because typically the third-party profilers will, 00:51:08.620 --> 00:51:12.340 when you're in an alpha phase of the changes, 00:51:12.800 --> 00:51:17.660 typically those profilers wouldn't support that version of CPython. 00:51:17.920 --> 00:51:20.740 Now you will have a something profiler which works, 00:51:21.290 --> 00:51:23.140 which is within NCPython itself. 00:51:23.960 --> 00:51:26.880 I run into this working on the profiler itself. 00:51:27.420 --> 00:51:32.560 How do you measure the performance of the profiling code, 00:51:32.660 --> 00:51:33.860 like all of those Python, 00:51:34.140 --> 00:51:36.820 fairly complex Python code? 00:51:37.340 --> 00:51:41.140 You can't really do that without having a profiler built in. 00:51:41.500 --> 00:51:43.960 So that's another nice effect of having a something 00:51:44.140 --> 00:51:46.980 profile built into the standard library itself. 00:51:47.280 --> 00:51:51.440 Yeah. And as Pablo said, you can optimize for it so that now that you have a stable interface, 00:51:51.950 --> 00:51:54.860 we can say, okay, this is the thing we're going to optimize with everyone else just randomly 00:51:55.200 --> 00:51:58.140 poking around the bytes and my code and all that kind of stuff. 00:51:58.340 --> 00:51:58.640 It's like, 00:51:58.760 --> 00:52:01.500 well, you still need to poke around the bytes on the bytecode. It's just like, 00:52:01.620 --> 00:52:02.140 at least now, 00:52:02.460 --> 00:52:03.040 you know, it's possible. 00:52:03.240 --> 00:52:04.640 You know, you can follow the trail, 00:52:04.790 --> 00:52:08.340 you can follow the soldiers of giants kind of thing, you know? 00:52:08.600 --> 00:52:08.880 Yeah, 00:52:09.180 --> 00:52:09.420 exactly. 00:52:10.300 --> 00:52:10.360 Exactly. 00:52:11.140 --> 00:52:11.880 It is interesting. 00:52:12.040 --> 00:52:14.319 You could use it to figure out how stuff runs on the 00:52:14.340 --> 00:52:14.860 newer versions, 00:52:15.340 --> 00:52:17.240 because even though maybe the shape of stuff has changed, 00:52:17.460 --> 00:52:18.660 it's still the same API. 00:52:18.760 --> 00:52:18.820 Oh, 00:52:19.000 --> 00:52:19.560 actually, 00:52:20.020 --> 00:52:20.800 yeah. 00:52:21.000 --> 00:52:23.080 And also just to clarify another thing, 00:52:23.240 --> 00:52:25.260 like right now, which is also 00:52:25.480 --> 00:52:29.000 important and it's also an advantage of the other profilers is that, of course, 00:52:29.420 --> 00:52:30.300 this is using this 00:52:30.480 --> 00:52:32.360 extra technology that we added to 3.15, 00:52:32.700 --> 00:52:34.680 which means that it only runs in 3.15. 00:52:34.880 --> 00:52:35.680 So you can only 00:52:35.840 --> 00:52:38.380 analyze 3.15 code from 3.15, 00:52:39.080 --> 00:52:40.700 while PySpy works with any Python. 00:52:41.080 --> 00:52:43.439 So right now you want to run 00:52:43.460 --> 00:52:44.780 PiSpy on 3.12, 00:52:44.960 --> 00:52:45.280 you can. 00:52:45.580 --> 00:52:46.920 Or 3.13 or 3.14. 00:52:47.340 --> 00:52:49.140 You know, with different performance 00:52:49.460 --> 00:52:51.280 characteristics for sure, but you can 00:52:51.380 --> 00:52:51.720 do that. 00:52:52.120 --> 00:52:53.440 Or Austin or whatever 00:52:53.480 --> 00:52:54.280 of the other ones, right? 00:52:55.820 --> 00:52:57.320 But right now, this one 00:52:57.520 --> 00:52:59.220 only works from 3.15 forward, 00:52:59.340 --> 00:52:59.460 right? 00:52:59.780 --> 00:53:01.140 And between the same version. 00:53:01.500 --> 00:53:03.440 So that's an important 00:53:03.660 --> 00:53:03.720 thing. 00:53:03.880 --> 00:53:05.340 It's not a limitation for the 00:53:05.580 --> 00:53:05.660 future. 00:53:05.860 --> 00:53:07.280 It's possible that we can 00:53:07.500 --> 00:53:09.000 allow 3.15 to analyze forward. 00:53:09.600 --> 00:53:09.760 Sorry, 00:53:11.120 --> 00:53:12.180 backward for 3.15, 00:53:12.440 --> 00:53:13.420 so 3.16 can 00:53:13.440 --> 00:53:15.480 3.15 and 3.16 or 00:53:15.540 --> 00:53:17.700 3.17 can do 3.15, 3.16 00:53:17.780 --> 00:53:18.440 and 3.17 00:53:19.480 --> 00:53:21.480 but still we don't know. We don't know 00:53:21.480 --> 00:53:23.280 if we want to get into that 00:53:24.060 --> 00:53:25.320 but yeah, that's another advantage 00:53:26.120 --> 00:53:27.500 but if you go to the 00:53:27.800 --> 00:53:28.480 I don't know if you, 00:53:28.800 --> 00:53:29.520 I can see that 00:53:29.600 --> 00:53:31.400 Hold on really quick before we move on 00:53:31.580 --> 00:53:33.520 what about this Python backport 00:53:33.640 --> 00:53:33.780 stuff? 00:53:34.120 --> 00:53:34.840 Are you familiar with this? 00:53:35.040 --> 00:53:35.640 Python profiling? 00:53:36.940 --> 00:53:38.540 Backporting the new profiling library 00:53:38.940 --> 00:53:39.100 to, 00:53:39.520 --> 00:53:40.620 I don't know what version goes 00:53:40.760 --> 00:53:41.420 to 3.14. 00:53:41.960 --> 00:53:45.740 Like a Clanker made version of this, like, or something like a version? 00:53:45.740 --> 00:53:46.460 Let's see. 00:53:47.100 --> 00:53:50.200 Yeah, possibly it was made over a one-week period. 00:53:50.440 --> 00:53:50.560 So, 00:53:50.760 --> 00:53:51.400 yeah, probably. 00:53:51.860 --> 00:53:52.000 Yeah. 00:53:52.540 --> 00:53:54.520 Well, I mean, it is possible because, 00:53:54.760 --> 00:53:55.240 technically speaking, 00:53:55.740 --> 00:53:57.520 you can, 00:53:58.360 --> 00:53:59.820 like, you go up in this repo. 00:53:59.820 --> 00:54:01.340 We are looking at the GitHub repo. 00:54:01.380 --> 00:54:02.760 So, you go up in this page. 00:54:03.140 --> 00:54:05.500 You can see that there's this underscore remote debugging. 00:54:05.900 --> 00:54:06.020 So, 00:54:06.180 --> 00:54:08.240 that is the C-Python folder, 00:54:08.440 --> 00:54:08.560 right? 00:54:08.760 --> 00:54:11.240 So probably what happened here is that they copy this code 00:54:11.560 --> 00:54:13.360 and they strip all the new cool, 00:54:13.540 --> 00:54:14.040 you know, 00:54:14.600 --> 00:54:15.860 bells and whistles for 3.15. 00:54:16.460 --> 00:54:18.380 And that still works very well for 3.14. 00:54:19.680 --> 00:54:21.080 I don't know if it works for less. 00:54:21.290 --> 00:54:22.060 So I will be interested. 00:54:22.200 --> 00:54:23.240 Can you check the Python version, 00:54:23.580 --> 00:54:23.700 perhaps, 00:54:23.790 --> 00:54:25.120 in the set of py or something? 00:54:25.500 --> 00:54:25.640 Yeah, 00:54:25.800 --> 00:54:25.920 sure. 00:54:26.100 --> 00:54:27.120 Py project. 00:54:27.380 --> 00:54:28.400 Oh, they will set py. 00:54:28.540 --> 00:54:29.140 Wow, how about that? 00:54:29.270 --> 00:54:29.380 Okay. 00:54:30.820 --> 00:54:31.240 Yeah, okay. 00:54:31.500 --> 00:54:31.600 Well, 00:54:32.140 --> 00:54:32.360 interesting. 00:54:32.570 --> 00:54:33.500 Yeah, 3.14, you see? 00:54:34.279 --> 00:54:34.640 So, 00:54:35.050 --> 00:54:37.060 yeah, it supports 3.14 only because, 00:54:37.500 --> 00:54:37.620 you know, 00:54:37.700 --> 00:54:39.640 It's the 3.0 version minus the cool things. 00:54:39.900 --> 00:54:42.440 So it's still using the debugging interface. 00:54:43.580 --> 00:54:46.300 You can see the bag obsessed validation.h, 00:54:46.660 --> 00:54:48.280 which is clearly the bag stuff. 00:54:49.660 --> 00:54:51.040 So that is possible. 00:54:51.200 --> 00:54:51.520 Unfortunately, 00:54:51.980 --> 00:54:52.940 as you really well know, 00:54:52.940 --> 00:54:54.980 we cannot backport features in CPython. 00:54:55.120 --> 00:54:56.520 So this is good. 00:54:56.570 --> 00:54:57.020 This is good. 00:54:57.140 --> 00:54:59.000 I mean, I'm happy people are doing this. 00:54:59.000 --> 00:55:01.660 So maybe it gives you a little bit of a lever of today and tomorrow. 00:55:02.600 --> 00:55:06.100 Yeah, and this makes me happy, 00:55:06.300 --> 00:55:06.400 right? 00:55:06.600 --> 00:55:12.100 Like, look, man, if people are willing to like pull tricks to use our stuff in all versions, 00:55:12.390 --> 00:55:12.860 like amazing. 00:55:13.050 --> 00:55:13.580 You know what I mean? 00:55:13.800 --> 00:55:14.840 Like that's success to me. 00:55:15.860 --> 00:55:17.880 What's worse is they don't care and they don't pay any attention. 00:55:17.990 --> 00:55:18.720 They just leave it, right? 00:55:19.000 --> 00:55:19.400 Yeah, yeah. 00:55:19.770 --> 00:55:22.160 Well, you know, who the bugs are going to go to? 00:55:22.340 --> 00:55:23.380 Like that's an interesting question. 00:55:23.860 --> 00:55:24.160 But like, 00:55:24.640 --> 00:55:24.920 you know, 00:55:25.300 --> 00:55:27.320 if you report a 3.14 bug in Cpython, 00:55:27.540 --> 00:55:28.660 I may tell you, hey, man, what 00:55:28.660 --> 00:55:29.060 are you doing? 00:55:29.820 --> 00:55:30.200 But like, 00:55:30.860 --> 00:55:31.040 you know, 00:55:31.560 --> 00:55:34.000 if they are willing to bug for the bugs as well, 00:55:34.220 --> 00:55:34.840 then let's go. 00:55:34.910 --> 00:55:36.260 You know, like that's good to me. 00:55:36.560 --> 00:55:38.340 So I actually did this. 00:55:38.480 --> 00:55:39.580 So I tried doing this. 00:55:39.740 --> 00:55:40.560 This is not my project, 00:55:41.180 --> 00:55:45.200 of course, but I tried this because this seemed like a good idea. 00:55:45.360 --> 00:55:47.180 Hey, how can we make this work in 3.14? 00:55:47.780 --> 00:55:48.680 And that was the idea. 00:55:48.780 --> 00:55:53.420 You can just literally take the profiling package and just copy that over. 00:55:53.640 --> 00:55:54.800 But it's not that simple. 00:55:54.980 --> 00:56:01.320 So there are some changes in internal CPython structures in CLO. 00:56:03.360 --> 00:56:06.400 It's not a lot of changes, but you do get some changes there. 00:56:06.770 --> 00:56:10.200 You may be able to get away without those, but the patch is enormous. 00:56:10.580 --> 00:56:11.840 It's absolutely gigantic. 00:56:11.970 --> 00:56:15.720 I did not realize how much code there is in Tachyon until I was doing this. 00:56:16.100 --> 00:56:18.620 And that just means that every single patch version, every single change, 00:56:18.780 --> 00:56:20.440 you will have to maintain this. 00:56:20.470 --> 00:56:21.740 There may be significant maintenance. 00:56:22.380 --> 00:56:24.240 So the idea was rejected, 00:56:25.090 --> 00:56:27.580 sort of fact, to do this and maintain it as a separate package, 00:56:28.620 --> 00:56:29.380 at least for me. 00:56:29.640 --> 00:56:31.380 I don't see this as a long-term, 00:56:31.660 --> 00:56:33.660 this back thing as a long-term useful thing, 00:56:33.940 --> 00:56:35.300 but maybe in the first year, 00:56:35.700 --> 00:56:35.900 you know, 00:56:36.020 --> 00:56:37.520 as people are searching for 3.15. 00:56:37.720 --> 00:56:38.480 Of course, of course, 00:56:38.720 --> 00:56:38.800 yeah. 00:56:39.340 --> 00:56:40.380 And then in three years, you tell them, 00:56:40.380 --> 00:56:41.180 like, what are you doing 3.14? 00:56:41.290 --> 00:56:41.660 Get out of here. 00:56:42.040 --> 00:56:43.440 But, but, but, 00:56:43.740 --> 00:56:43.920 but, 00:56:44.080 --> 00:56:44.520 ooh, 00:56:44.860 --> 00:56:46.120 maybe we can talk about this later. 00:56:46.400 --> 00:56:48.820 But like, one interesting thing, 00:56:48.940 --> 00:56:51.340 well, actually, let me segue into it smoothly, 00:56:51.540 --> 00:56:51.740 smoothly. 00:56:51.900 --> 00:56:53.880 Let's do some smooth segue into it. 00:56:54.440 --> 00:56:58.360 So one interesting thing that is also important to mention, 00:56:58.900 --> 00:57:02.500 And I can see that you have the profiling.sampling documentation in the sidebar. 00:57:02.660 --> 00:57:06.180 So let's open that for the people viewing this in YouTube or whatever. 00:57:06.700 --> 00:57:09.060 So the documentation here is huge, 00:57:09.280 --> 00:57:09.360 huge, 00:57:09.620 --> 00:57:10.140 huge documentation. 00:57:10.580 --> 00:57:11.040 Why is that? 00:57:11.260 --> 00:57:12.660 Because this does a lot of things. 00:57:12.880 --> 00:57:15.980 It does a lot of things, a lot of things, by the way, that only profiler can do. 00:57:16.120 --> 00:57:17.500 So it's not just about being fast. 00:57:17.600 --> 00:57:20.060 It does all sorts of other things. 00:57:20.380 --> 00:57:24.380 So as opposed to like cprofile that like has like, 00:57:24.560 --> 00:57:25.799 you know, the PSTAT format 00:57:25.820 --> 00:57:26.920 and can tell you like basic, 00:57:27.300 --> 00:57:28.920 you know, statistics about your program or whatever. 00:57:29.410 --> 00:57:31.920 Like this thing is a full beast. 00:57:32.030 --> 00:57:32.860 It comes with everything, 00:57:33.080 --> 00:57:33.160 right? 00:57:33.410 --> 00:57:37.480 So here Tachyon can create like, like flame graphs. 00:57:37.770 --> 00:57:39.500 It can create all sorts of visualizations. 00:57:39.770 --> 00:57:41.060 Like, so it has that top model. 00:57:41.170 --> 00:57:44.040 So like, like when you do top and you see the processes life. 00:57:44.420 --> 00:57:46.580 So Tachyon has this like, like Tachyon life. 00:57:47.060 --> 00:57:49.740 Well, I can show you like the functions life of your application, 00:57:50.180 --> 00:57:51.100 like almost like a top, 00:57:51.310 --> 00:57:52.940 it can create like heat maps. 00:57:53.200 --> 00:58:01.040 So you can see lines with different colors depending if they are more hot or cold, if they are more slow or fast. 00:58:01.860 --> 00:58:05.120 You can do differential frame graphs, which is very cool. 00:58:05.300 --> 00:58:12.900 You can run your program and then change a bit and run it again and then compare what became faster and slower. 00:58:13.080 --> 00:58:16.840 So it can tell you what functions were faster and what functions got slower. 00:58:17.300 --> 00:58:20.420 Another very cool thing that this does is that it does async audio, 00:58:20.640 --> 00:58:21.180 which is the first 00:58:21.420 --> 00:58:25.920 profiler right now that like can also do async audio because like, you know, 00:58:26.080 --> 00:58:26.560 the even loop 00:58:26.780 --> 00:58:28.060 does all sorts of weird things. 00:58:28.900 --> 00:58:29.260 Right. 00:58:29.400 --> 00:58:30.680 And it's often one thread, 00:58:31.110 --> 00:58:32.780 basically kind of async call stack. 00:58:32.890 --> 00:58:33.700 So it's like, what are you doing? 00:58:33.820 --> 00:58:34.740 Like, well, we're dispatching. 00:58:35.320 --> 00:58:35.540 Yeah, 00:58:35.720 --> 00:58:35.800 exactly. 00:58:35.960 --> 00:58:36.060 Exactly. 00:58:36.100 --> 00:58:36.660 What is that? 00:58:36.720 --> 00:58:36.980 What is that? 00:58:37.110 --> 00:58:37.200 Exactly. 00:58:37.520 --> 00:58:37.580 Yeah. 00:58:38.040 --> 00:58:38.940 Or like, you know, it's like, 00:58:39.160 --> 00:58:39.280 oh, 00:58:40.040 --> 00:58:40.200 you know, 00:58:40.570 --> 00:58:42.140 but it's also important because I think 00:58:42.200 --> 00:58:45.660 I do changes a bit the, what means slow or fast. 00:58:45.820 --> 00:58:50.480 Like, for example, imagine that you're downloading a 3GB file, 00:58:50.810 --> 00:58:51.820 but the server is slow. 00:58:52.220 --> 00:58:57.680 So you're spending a lot of time waiting for that 3GB file, not because you are crunching 00:58:57.730 --> 00:58:58.200 your CPU, 00:58:58.440 --> 00:58:59.680 it's because your server is slow. 00:58:59.920 --> 00:59:02.340 But your application is slow because the server is slow. 00:59:02.660 --> 00:59:04.540 So how do you know that that is happening? 00:59:04.980 --> 00:59:09.200 Well, in the normal world, if that was the case, you would be like, I don't know, reading 00:59:09.290 --> 00:59:09.720 or something. 00:59:10.040 --> 00:59:13.300 Like, you will see that the profile will show you that you are reading, reading, reading, 00:59:13.540 --> 00:59:14.880 so you can't imagine what's going on. 00:59:15.200 --> 00:59:15.820 But in this case, 00:59:16.000 --> 00:59:20.220 because the event loop will only be waking up when the server has something to tell you. 00:59:20.640 --> 00:59:20.860 Meanwhile, 00:59:21.060 --> 00:59:22.220 that will never appear. 00:59:22.540 --> 00:59:25.140 So like if you can only see the coroutines that are running, 00:59:25.600 --> 00:59:30.100 the fact that you are waiting for the server to send your data will never appear in your flame graph because it's not happening. 00:59:30.480 --> 00:59:37.040 So you will never know that what is slow about your program is that your program is waiting for a three-hour file to appear, 00:59:37.360 --> 00:59:37.460 right? 00:59:37.820 --> 00:59:38.060 Yeah, 00:59:38.820 --> 00:59:41.100 this is especially important for a sampling profiler, 00:59:41.300 --> 00:59:41.380 right? 00:59:41.680 --> 00:59:43.320 Exactly, exactly, because you are not tracing it. 00:59:43.400 --> 00:59:48.600 Also tracing in the even loop is very weird because you keep changing for task to task, so it's very confusing. 00:59:49.100 --> 00:59:51.760 But here in Taki has these two modes. 00:59:52.240 --> 00:59:54.780 One of them is like tasks that are running. 00:59:55.000 --> 00:59:56.680 So you can see only running tasks. 00:59:56.840 --> 00:59:59.900 So in that case, for instance, if you're waiting for the 3D guy file, it will not appear. 01:00:00.300 --> 01:00:02.200 But you can see that meanwhile you're doing other things. 01:00:02.600 --> 01:00:05.720 But also it has this mode, which is all the tasks that I'm waiting for. 01:00:06.180 --> 01:00:07.980 So every time you sample, 01:00:08.140 --> 01:00:10.320 you see every single task that is going on. 01:00:10.660 --> 01:00:11.660 So you can see that like, 01:00:12.000 --> 01:00:12.080 oh, 01:00:12.300 --> 01:00:13.740 I'm waiting for this 3G file. 01:00:14.140 --> 01:00:15.020 It's just that it's not advancing. 01:00:15.280 --> 01:00:17.820 So because it's not advancing, it will appear in more samples. 01:00:18.310 --> 01:00:22.220 And then you will know that like that guy is like all the time being waiting for. 01:00:22.230 --> 01:00:23.460 So like, you know that it's happening. 01:00:24.420 --> 01:00:27.900 And then apart from that, it also has like dump mode when you can have like, 01:00:28.240 --> 01:00:28.360 you know, 01:00:28.570 --> 01:00:31.900 imagine that your application is frozen because you have a deadlock or perhaps, 01:00:32.060 --> 01:00:32.400 you know, 01:00:32.800 --> 01:00:36.580 you know, it's just waiting for IO that never arrives and you want to know where it's going. 01:00:37.040 --> 01:00:41.020 so you can use profiling.sampling dump 01:00:41.360 --> 01:00:43.200 and then you pass the PID of your application 01:00:43.670 --> 01:00:46.040 and then it will show you what's going on. 01:00:46.110 --> 01:00:47.100 It will show you the stack trace. 01:00:47.560 --> 01:00:48.540 Imagine the stack trace, 01:00:48.760 --> 01:00:50.500 like if an exception is thrown, 01:00:50.660 --> 01:00:51.820 but instead of throwing an exception, 01:00:51.890 --> 01:00:52.920 it will show you where it is. 01:00:53.300 --> 01:00:53.760 And of course, 01:00:53.980 --> 01:00:55.180 this works on everywhere. 01:00:55.480 --> 01:00:56.400 So it works on macOS, 01:00:56.590 --> 01:00:57.220 it works on Windows, 01:00:57.430 --> 01:00:58.160 it works on Linux. 01:00:58.820 --> 01:00:59.500 Same mechanism, 01:00:59.800 --> 01:01:00.360 same guarantees, 01:01:00.620 --> 01:01:01.040 same everything. 01:01:02.279 --> 01:01:03.140 Okay, that's awesome. 01:01:03.300 --> 01:01:03.380 Now, 01:01:03.590 --> 01:01:05.099 you threw out a bunch of stuff there 01:01:05.260 --> 01:01:07.140 And I think it's amazing how much is here, 01:01:07.320 --> 01:01:09.740 but let's just go back really quick and just, 01:01:10.180 --> 01:01:11.040 let's just focus on this. 01:01:11.200 --> 01:01:11.900 I think this is, 01:01:12.060 --> 01:01:12.920 this kind of game changing. 01:01:12.920 --> 01:01:13.600 Yeah, there is so much. 01:01:13.780 --> 01:01:15.400 There is, you need a second podcast, 01:01:15.640 --> 01:01:15.740 man. 01:01:15.820 --> 01:01:16.980 Like you need to invite us a second time. 01:01:17.000 --> 01:01:17.560 What do we even do? 01:01:17.680 --> 01:01:19.480 Like we got to just, we're going to go, we're going to go, 01:01:19.680 --> 01:01:19.960 keep going. 01:01:20.420 --> 01:01:21.600 So the attach, 01:01:22.120 --> 01:01:22.320 this is, 01:01:22.980 --> 01:01:24.120 there's something running in production. 01:01:24.600 --> 01:01:24.800 Let me, 01:01:25.440 --> 01:01:27.480 right now it's doing whatever it is that we're like, 01:01:27.640 --> 01:01:27.780 uh, 01:01:28.060 --> 01:01:28.640 what is this doing? 01:01:28.780 --> 01:01:29.100 What's going on? 01:01:29.180 --> 01:01:29.840 How do we fix this? 01:01:29.960 --> 01:01:30.940 Or just what's going on? 01:01:30.980 --> 01:01:32.200 you can attach to it, 01:01:32.740 --> 01:01:34.460 grab a pstat file, 01:01:34.900 --> 01:01:36.220 take that away and work on it, right? 01:01:36.590 --> 01:01:36.680 Without, 01:01:36.940 --> 01:01:39.740 I mean, it probably does affect the performance somehow, 01:01:40.439 --> 01:01:42.060 but not nearly the same. 01:01:42.660 --> 01:01:43.320 No, no, it's free. 01:01:43.600 --> 01:01:43.880 It's free. 01:01:43.880 --> 01:01:44.160 It's free. 01:01:44.650 --> 01:01:44.760 Okay. 01:01:45.000 --> 01:01:45.320 It's free. 01:01:45.920 --> 01:01:47.740 The key way it's free is because, 01:01:47.900 --> 01:01:47.980 well, 01:01:48.170 --> 01:01:48.500 okay, sorry. 01:01:48.600 --> 01:01:49.240 It's free by default. 01:01:49.560 --> 01:01:51.540 There is something that you can do. 01:01:51.880 --> 01:01:53.160 We can't go into detail in a second, 01:01:53.340 --> 01:01:57.900 but there's something that you can do to kind of like have more exact data 01:01:58.440 --> 01:01:59.780 that will slow it down a bit. 01:02:00.120 --> 01:02:00.400 But still, 01:02:00.740 --> 01:02:02.580 for most of the sampling rates, 01:02:02.740 --> 01:02:03.360 it's mostly free. 01:02:03.870 --> 01:02:04.440 So by default, 01:02:04.900 --> 01:02:07.280 Acheon will never stop your application. 01:02:07.530 --> 01:02:09.420 It will just always read what is going on. 01:02:09.540 --> 01:02:12.500 Of course, the trade-off is that you can do partial reads. 01:02:12.580 --> 01:02:14.360 So perhaps it's reading, 01:02:14.560 --> 01:02:17.420 a structure that's changing, so it gets some weird stuff. 01:02:17.620 --> 01:02:18.040 So Acheon, 01:02:18.540 --> 01:02:21.680 most of the time, will detect that that is happening and it will throw away the sample. 01:02:21.890 --> 01:02:23.180 So it will say, okay, this is a bad sample. 01:02:23.420 --> 01:02:24.560 And it will not read it. 01:02:24.700 --> 01:02:27.520 But of course, that means that you have less samples if that is happening a lot. 01:02:27.600 --> 01:02:29.840 That can happen, for example, you have a lot of generators, 01:02:30.590 --> 01:02:32.360 because generators tend to be a bit like, 01:02:32.700 --> 01:02:33.940 you know, you get partial reads. 01:02:34.270 --> 01:02:35.100 So sometimes it happens. 01:02:35.820 --> 01:02:36.300 But by default, 01:02:36.540 --> 01:02:38.060 it will only read from the application. 01:02:38.210 --> 01:02:38.700 And the application, 01:02:38.960 --> 01:02:41.000 first of all, the application doesn't know that this is happening. 01:02:41.440 --> 01:02:41.680 And, 01:02:42.050 --> 01:02:45.600 you know, unless you only have one CPU and then you are giving the CPU to both programs, 01:02:45.860 --> 01:02:47.600 which in today's computers, 01:02:48.150 --> 01:02:48.820 that will never happen. 01:02:49.820 --> 01:02:53.080 I don't know, you're running a Docker container with only one CPU core or whatever. 01:02:53.160 --> 01:02:53.480 but like, 01:02:53.660 --> 01:02:55.200 you know, you run the profile in another, 01:02:55.420 --> 01:02:57.580 you know, sidecar or whatever, like that would ever happen. 01:02:58.320 --> 01:02:59.140 So unless that's the case, 01:02:59.320 --> 01:03:01.500 like the application will never know and it will run at full speed. 01:03:02.400 --> 01:03:07.820 If you want to avoid this kind of partial read situation and you want to show that every read is consistent 01:03:08.180 --> 01:03:12.700 and increase the kind of like exactness and, you know, 01:03:12.880 --> 01:03:14.100 trustworthiness of the data, 01:03:14.640 --> 01:03:17.180 then there is this mode where you can pass does this blocking. 01:03:17.640 --> 01:03:19.780 And what does this blocking will do is that before reading, 01:03:19.940 --> 01:03:22.400 it will stop the application for a split millisecond, 01:03:22.720 --> 01:03:23.260 very quickly, 01:03:23.570 --> 01:03:25.400 it will do the read and it will restart it. 01:03:25.680 --> 01:03:25.800 So, 01:03:26.120 --> 01:03:28.440 of course, if you do the default sampling rate 01:03:28.640 --> 01:03:31.420 or even sampling rates up to 10,000 Hz, 01:03:31.840 --> 01:03:33.340 that is a very small amount of time. 01:03:33.560 --> 01:03:35.540 We are talking about slowdowns of 1% to 2%. 01:03:36.200 --> 01:03:38.180 So 1% to 2%, not two times. 01:03:38.260 --> 01:03:39.760 It's 1% to 2%. It's nothing. 01:03:40.260 --> 01:03:42.480 If you've done profiling before, it's much, 01:03:42.980 --> 01:03:43.600 much lower. 01:03:43.620 --> 01:03:43.680 It's like, 01:03:44.260 --> 01:03:44.480 whatever, 01:03:44.880 --> 01:03:45.020 man. 01:03:45.450 --> 01:03:46.280 What are you talking about? 01:03:48.380 --> 01:03:51.040 We would make Parthons 315 5% faster, 01:03:51.180 --> 01:03:52.180 and then that's fine, right? 01:03:52.420 --> 01:03:55.780 But like if you ramp the sample rate really high, 01:03:55.960 --> 01:03:58.780 like let's say 100,000 samples per second, 01:03:59.080 --> 01:04:02.800 then we are talking about like more, like 20% or 30% or 80%. 01:04:02.940 --> 01:04:05.980 I think that you do it at crazy rates, like 1 million samples, 01:04:06.540 --> 01:04:07.600 if your machine can do it, 01:04:07.660 --> 01:04:10.700 because now it depends on your machine to be able to do that. 01:04:11.100 --> 01:04:14.240 Then we are talking about two times lower or something like that. 01:04:14.560 --> 01:04:17.140 But like you see, like you need to like go really, really crazy. 01:04:17.640 --> 01:04:18.980 So in production, for example, 01:04:19.160 --> 01:04:23.680 you can sample perfectly at 100 or 1000 hertz and have a slow 01:04:23.780 --> 01:04:24.800 dance of one or two percent. 01:04:24.980 --> 01:04:26.160 And it's fine. It's acceptable. 01:04:27.280 --> 01:04:29.020 And this thing that we're talking 01:04:29.180 --> 01:04:29.280 about, 01:04:29.540 --> 01:04:30.820 this thing is so important. 01:04:31.080 --> 01:04:33.860 You don't understand how important this is. Like I'm free 01:04:33.860 --> 01:04:35.780 to say this is not a trade secret. 01:04:35.940 --> 01:04:38.820 Like my employer used this thing all the fucking time 01:04:39.420 --> 01:04:40.220 for making a lot of money. 01:04:40.500 --> 01:04:45.779 So you cannot imagine how critical it is to understand why your programs 01:04:45.800 --> 01:04:48.700 like are slow or how your programs can be optimized. 01:04:49.190 --> 01:04:51.820 Because a lot of the time you just need to see live traffic. 01:04:52.170 --> 01:04:53.720 Like if you have or like live data, 01:04:53.920 --> 01:04:53.980 right? 01:04:54.260 --> 01:04:58.720 If you never know how your application is really going to behave or what is slow 01:04:58.910 --> 01:05:00.540 until you measure the real deal. 01:05:00.730 --> 01:05:02.240 You cannot measure the fake deal. 01:05:02.410 --> 01:05:03.940 Like you need to measure the real deal. 01:05:04.360 --> 01:05:07.880 And for that, you need to attach because most of the times you don't have access to, 01:05:08.210 --> 01:05:10.460 oh, I just want to launch my application on the lab profiler. 01:05:10.620 --> 01:05:13.780 No, you need to like, you just have it there and it's doing something weird. 01:05:14.140 --> 01:05:15.600 And then you attach and then you look a bit. 01:05:15.740 --> 01:05:16.320 and then you detach, 01:05:16.540 --> 01:05:16.660 right? 01:05:16.820 --> 01:05:17.180 Even this, 01:05:17.680 --> 01:05:18.920 let's say for example, 01:05:19.720 --> 01:05:21.940 that blocking was 50% slowdown. 01:05:22.100 --> 01:05:23.680 You could do it for like a bunch of seconds 01:05:23.940 --> 01:05:25.220 and then detach and you know, 01:05:25.420 --> 01:05:26.020 here is all good. 01:05:26.300 --> 01:05:27.300 And then you got the information 01:05:27.760 --> 01:05:28.960 and then you can reason about it. 01:05:29.260 --> 01:05:29.740 And then you can manage - That 01:05:29.740 --> 01:05:31.180 information is so good, 01:05:31.340 --> 01:05:31.540 Pablo. 01:05:32.040 --> 01:05:32.360 The in 01:05:32.360 --> 01:05:34.300 -production information is so much different. 01:05:34.820 --> 01:05:34.880 It's 01:05:34.880 --> 01:05:35.940 key, it's key, it's key. 01:05:36.180 --> 01:05:38.200 Like for this 01:05:38.200 --> 01:05:39.480 year, I mean, this is kind of the segue 01:05:39.720 --> 01:05:40.140 and the secret 01:05:40.140 --> 01:05:40.520 and whatever, 01:05:40.760 --> 01:05:41.360 but like one 01:05:41.360 --> 01:05:43.980 of the things I've been working on here, 01:05:44.780 --> 01:05:48.720 had solar training for like this past year is precisely all about getting, 01:05:49.260 --> 01:05:49.880 well, not all about, 01:05:50.080 --> 01:05:50.480 but like, like, 01:05:50.660 --> 01:05:55.160 but a lot of my work here has been to ensure that getting that information and much 01:05:55.280 --> 01:05:57.380 more from light data is what you do. 01:05:57.900 --> 01:05:59.780 And we can go into why it's insane here. But like, 01:06:00.220 --> 01:06:00.820 part of the 01:06:00.940 --> 01:06:03.960 reason is because this information is gold, 01:06:04.280 --> 01:06:06.140 like, it's just the best thing that you can do, 01:06:06.320 --> 01:06:06.460 because 01:06:06.920 --> 01:06:08.820 it's telling you literally what you need to change. 01:06:09.120 --> 01:06:09.840 It's almost cheating. 01:06:10.120 --> 01:06:10.640 You know what I mean? 01:06:14.340 --> 01:06:14.740 You 01:06:14.760 --> 01:06:21.360 bad change this thing and then you go and like literally today today right we activated a mega 01:06:21.560 --> 01:06:25.720 version of this which is not tacky only something else but we activated a version of this at work 01:06:26.120 --> 01:06:30.740 in one of the machines that the data scientists used to like the supercomputers that we have here 01:06:30.740 --> 01:06:35.480 so here we have a lot of supercomputers and then like people do their their you know the experiments 01:06:35.760 --> 01:06:41.380 or things with trading and you know that's like i don't know it's war like it's huge machines 01:06:41.660 --> 01:06:43.820 like hundreds and hundreds of gigabytes of memory, 01:06:44.400 --> 01:06:45.380 like hundreds of cores. 01:06:45.880 --> 01:06:47.040 So it's like war, right? 01:06:47.310 --> 01:06:48.520 And then for the first time, 01:06:48.690 --> 01:06:50.460 because this is so ridiculously complicated 01:06:50.660 --> 01:06:53.860 and these machines are so big that like it's so different to do this thing at scale, 01:06:54.440 --> 01:06:57.180 we managed to have a version of this that is tracing all programs. 01:06:57.450 --> 01:06:59.020 So not just one, all of them, 01:06:59.240 --> 01:06:59.320 right? 01:06:59.620 --> 01:07:01.740 And using this live data, literally today, 01:07:02.180 --> 01:07:05.900 we saw one of these applications that was doing something slightly odd. 01:07:06.120 --> 01:07:06.300 It's like, 01:07:06.520 --> 01:07:10.240 why is this application taking like an entire core of the machine 01:07:10.280 --> 01:07:11.540 when it should be doing nothing. 01:07:12.040 --> 01:07:13.220 And I was like, hmm. 01:07:13.680 --> 01:07:17.380 And we look at that and it turns out that in some specific case 01:07:17.410 --> 01:07:20.260 that was happening a lot, but nobody thought that this was going to happen, 01:07:20.680 --> 01:07:23.980 one of the algorithms was underperforming slightly 01:07:24.200 --> 01:07:26.880 because it was accidentally quadratic in one way, right? 01:07:27.140 --> 01:07:28.820 Which is like, again, it's very difficult to see. 01:07:28.940 --> 01:07:31.760 If I show you the code, no person on the planet will see it. 01:07:31.920 --> 01:07:33.840 It is so ridiculously complicated to see. 01:07:34.120 --> 01:07:36.220 But we could see it as clear as day on the data. 01:07:36.440 --> 01:07:37.420 It was so obvious, 01:07:37.620 --> 01:07:38.160 so obvious, 01:07:38.320 --> 01:07:39.020 so obvious. 01:07:39.480 --> 01:07:41.300 And bam, we remove it and ba-da-boom, 01:07:41.580 --> 01:07:44.660 we have a core on every single machine of our supercomputer free now. 01:07:45.100 --> 01:07:45.740 That's awesome. 01:07:46.100 --> 01:07:46.720 This is ridiculous. 01:07:47.090 --> 01:07:50.680 You know how this is like a huge amount of money saved. 01:07:51.060 --> 01:07:53.800 So getting this data is the best thing that you can do. 01:07:54.160 --> 01:07:55.460 Listen, everyone listening to this, 01:07:55.680 --> 01:07:56.600 just grab this thing, 01:07:56.940 --> 01:08:00.260 put it to your application and your manager will be happy forever. 01:08:00.660 --> 01:08:03.040 Like it's just the easiest way to make your manager happy. 01:08:03.200 --> 01:08:07.460 Just go measure the shit and fix the thing and don't tell anyone the trick. 01:08:07.760 --> 01:08:08.540 And they're like, damn, 01:08:08.880 --> 01:08:09.440 you just fix it. 01:08:09.520 --> 01:08:10.900 You can do it to other teams. 01:08:11.440 --> 01:08:13.460 You're a performance wizard is what you've become. 01:08:13.640 --> 01:08:14.060 Look at you. 01:08:14.140 --> 01:08:14.420 Exactly, exactly. 01:08:14.780 --> 01:08:16.819 Just attach it to other people's programs. 01:08:17.380 --> 01:08:18.520 And then you see what's low. 01:08:18.600 --> 01:08:20.460 And suddenly you have a change in their code. 01:08:20.759 --> 01:08:21.540 And they say, wow, 01:08:21.740 --> 01:08:22.640 how did you do? 01:08:22.779 --> 01:08:23.060 And like, 01:08:23.460 --> 01:08:23.700 damn. 01:08:24.720 --> 01:08:25.100 Incredible. 01:08:25.339 --> 01:08:27.600 Just do it before everyone knows how to do it. 01:08:28.140 --> 01:08:28.960 What the secret is. 01:08:28.980 --> 01:08:30.720 No, like you knew from this podcast. 01:08:31.140 --> 01:08:32.359 You knew it because you're here. 01:08:32.759 --> 01:08:34.779 Don't worry, I'll delete the episode really soon. 01:08:35.060 --> 01:08:35.480 Oh, damn, 01:08:35.720 --> 01:08:35.779 damn. 01:08:36.720 --> 01:08:38.480 That way the secret will be safe. 01:08:38.900 --> 01:08:39.000 No. 01:08:39.180 --> 01:08:39.480 Yes. 01:08:41.060 --> 01:08:42.380 And this is just part of Python. 01:08:42.839 --> 01:08:43.440 It's just free stuff. 01:08:43.660 --> 01:08:44.700 I mean, the stuff that you're doing, 01:08:44.830 --> 01:08:47.020 the multi was obviously extending this, 01:08:47.160 --> 01:08:49.140 but this just comes with Python 3.15, 01:08:49.319 --> 01:08:49.380 right? 01:08:49.680 --> 01:08:51.259 Yes, this comes with Python 3.15. 01:08:51.540 --> 01:08:51.700 Sorry. 01:08:52.020 --> 01:08:54.259 I think it's really important to emphasize 01:08:55.140 --> 01:08:57.160 how, what a superpower this is, 01:08:57.290 --> 01:08:58.980 because I think in my experience, 01:08:59.520 --> 01:09:02.359 even if you have some tooling available 01:09:02.720 --> 01:09:04.580 for this sort of production profiling, 01:09:04.859 --> 01:09:06.680 it's not always the case that you have access 01:09:06.700 --> 01:09:08.700 that in the worst possible time 01:09:08.920 --> 01:09:10.420 when things are blowing up 01:09:10.470 --> 01:09:11.520 and you need something like that. 01:09:12.060 --> 01:09:12.799 It's the opposite. 01:09:13.529 --> 01:09:14.819 You don't have these tools ready 01:09:15.040 --> 01:09:16.759 when you realize that you need it, right? 01:09:17.020 --> 01:09:17.940 And what's amazing about this 01:09:17.970 --> 01:09:19.279 is that all you need 01:09:19.290 --> 01:09:20.440 is to have 3.15 installed 01:09:20.900 --> 01:09:22.240 and running in your application. 01:09:22.650 --> 01:09:24.299 And this opens up all sorts of possibilities. 01:09:24.720 --> 01:09:25.799 Like you can just go there 01:09:26.120 --> 01:09:26.480 and grab, 01:09:26.640 --> 01:09:27.020 you know, 01:09:27.100 --> 01:09:29.060 when things are happening, 01:09:29.299 --> 01:09:29.799 you can go there 01:09:30.540 --> 01:09:31.060 and sort of grab 01:09:31.779 --> 01:09:33.000 some sort of profile out of that. 01:09:33.580 --> 01:09:34.779 But I think it also opens up 01:09:34.779 --> 01:09:35.800 a lot of possibilities 01:09:35.799 --> 01:09:36.839 for the future. 01:09:37.680 --> 01:09:39.520 I mean, you can even implement 01:09:39.839 --> 01:09:41.220 continuous profiling with this thing 01:09:41.270 --> 01:09:42.060 if you wanted to, 01:09:42.440 --> 01:09:44.759 where you do this as a long-term thing 01:09:45.100 --> 01:09:47.500 on a lower sort of sampling frequency. 01:09:47.880 --> 01:09:48.779 But I think it's also important 01:09:48.830 --> 01:09:49.900 to emphasize here that 01:09:50.140 --> 01:09:52.420 Tachyon is not just a production profiler. 01:09:52.740 --> 01:09:54.160 So it has several modes 01:09:54.320 --> 01:09:55.700 where you attach to your process, 01:09:55.980 --> 01:09:56.600 to your live process, 01:09:56.820 --> 01:09:57.620 and that's super powerful 01:09:57.880 --> 01:09:59.260 and that's useful in many cases. 01:09:59.540 --> 01:10:00.820 But you can also load up 01:10:00.980 --> 01:10:01.980 your sort of application. 01:10:02.190 --> 01:10:03.200 It has a mode where 01:10:03.400 --> 01:10:04.800 you can just load up an application 01:10:04.800 --> 01:10:08.200 and then you start profiling as the program starts 01:10:08.640 --> 01:10:10.740 and it finishes at some duration 01:10:11.360 --> 01:10:12.380 and then you look at that data. 01:10:12.660 --> 01:10:14.800 So this is also very useful potentially 01:10:15.040 --> 01:10:16.160 in your development environment. 01:10:16.620 --> 01:10:18.780 Different reporters which are designed 01:10:19.120 --> 01:10:21.320 for development sort of processes, 01:10:21.640 --> 01:10:22.120 for example, 01:10:22.340 --> 01:10:23.080 you're looking for, 01:10:23.420 --> 01:10:27.320 you want to compare two different profiling snapshots. 01:10:27.700 --> 01:10:28.860 You know, like you make code changes 01:10:28.940 --> 01:10:31.960 and you want to compare how well those changes are, 01:10:32.320 --> 01:10:33.640 how much faster, how much slower 01:10:34.220 --> 01:10:35.180 are making your application. 01:10:35.680 --> 01:10:37.460 So I definitely wouldn't sort of 01:10:37.500 --> 01:10:39.520 lock it back in and like, you use this 01:10:39.720 --> 01:10:41.480 when you have performance issues in production. 01:10:42.580 --> 01:10:43.340 You can use it 01:10:43.660 --> 01:10:45.400 absolutely for any sort of 01:10:45.560 --> 01:10:46.340 profiling situation. 01:10:46.820 --> 01:10:46.920 Yeah, 01:10:47.080 --> 01:10:49.680 like a CLI sort of situation like you're talking about, maybe. 01:10:49.860 --> 01:10:50.280 Exactly. 01:10:50.440 --> 01:10:51.360 So you can use it 01:10:52.920 --> 01:10:53.200 actually 01:10:53.380 --> 01:10:54.821 we have the run here on the 01:10:55.580 --> 01:10:57.320 screen. So you can run a script or you can import 01:10:57.320 --> 01:10:59.500 a module with it and then you basically go from there. 01:10:59.800 --> 01:11:01.360 And there you can have, that can be 01:11:01.720 --> 01:11:03.620 your test suite or it can be a sample 01:11:03.920 --> 01:11:07.620 thing or it can be a CLI which you're looking at sort of improving the performance. 01:11:08.110 --> 01:11:08.500 By the way, 01:11:08.760 --> 01:11:11.040 since you were talking about quantum mechanics before. 01:11:11.900 --> 01:11:12.800 We're on the string theory now. 01:11:12.930 --> 01:11:13.340 No, just kidding. 01:11:13.370 --> 01:11:13.600 Go ahead. 01:11:13.760 --> 01:11:13.960 Yeah. 01:11:14.010 --> 01:11:14.240 No, no. 01:11:14.360 --> 01:11:14.620 Well, well, 01:11:14.780 --> 01:11:15.720 well, well, well, well, well, well. 01:11:16.580 --> 01:11:17.540 We're in relativity now. 01:11:17.710 --> 01:11:18.800 Do you know why it's called tachyon? 01:11:19.080 --> 01:11:20.460 I was wondering, but I haven't asked yet. 01:11:20.600 --> 01:11:20.960 No, why? 01:11:21.300 --> 01:11:21.660 I don't know. 01:11:22.340 --> 01:11:27.000 So a tachyon is the name of hypothetical particles that go faster than light. 01:11:27.560 --> 01:11:30.860 So like a photon is a particle that goes at the speed of light because, 01:11:31.140 --> 01:11:31.380 you know, it's 01:11:31.820 --> 01:11:33.340 the particles that made up light. 01:11:33.480 --> 01:11:33.660 So, 01:11:33.920 --> 01:11:34.940 you know, that's a photon. 01:11:35.820 --> 01:11:39.300 And, you know, special relativity doesn't say that this is not possible. 01:11:39.460 --> 01:11:39.860 It just says, 01:11:40.080 --> 01:11:44.820 like, if you're a photon, you cannot go faster than the speed of light or slower than the speed of light because you are light. 01:11:45.100 --> 01:11:45.660 And if you are, 01:11:45.960 --> 01:11:47.500 you know, you have mass, 01:11:47.700 --> 01:11:51.400 like a normal thing, you cannot go faster than the speed of light or at the speed of light. 01:11:51.580 --> 01:11:54.600 So the speed of light is like a limit, but you can also don't achieve it. 01:11:54.920 --> 01:11:59.340 But then special relativity also says, well, you could also be faster than the speed of light. 01:11:59.460 --> 01:12:00.920 And what happens if that is true, 01:12:01.120 --> 01:12:03.700 you cannot go at or lower than the speed of light. 01:12:03.940 --> 01:12:05.500 And of course, we have not seen such a thing. 01:12:05.760 --> 01:12:08.940 So that is very likely not true because among other things, 01:12:09.120 --> 01:12:10.920 tachyons will travel back in time, 01:12:11.520 --> 01:12:12.840 which is like perhaps a bit crazy. 01:12:12.980 --> 01:12:14.360 But in theory, 01:12:14.600 --> 01:12:18.100 you know, we have to talk about those possibilities and to investigate them. 01:12:18.130 --> 01:12:19.100 So we have to name them. 01:12:19.520 --> 01:12:22.440 And then the name for those particles is called tachyons, 01:12:23.280 --> 01:12:25.340 which is tachy, which is fast in Greek. 01:12:25.800 --> 01:12:28.420 So that's where it comes from. 01:12:28.760 --> 01:12:31.160 That's awesome. And it has its own logo as well. 01:12:31.490 --> 01:12:31.740 Oh yeah, 01:12:31.900 --> 01:12:32.960 Mike Jimenez did that. 01:12:33.390 --> 01:12:37.640 So Mike has been the person who has done some of the logos for the releases. 01:12:37.770 --> 01:12:39.740 So she did the 310 and 311. 01:12:40.180 --> 01:12:43.600 She's an ML researcher at World DeepMind. 01:12:44.120 --> 01:12:47.080 And she's also a very good artist. 01:12:47.480 --> 01:12:48.820 And she does a bunch of these logos. 01:12:49.020 --> 01:12:50.460 So she created the Tachyon logo. 01:12:50.680 --> 01:12:51.160 Very cool. 01:12:51.520 --> 01:12:53.520 I do want to say one more thing about the attach. 01:12:54.020 --> 01:12:56.440 Something that has always bothered me about profiling. 01:12:56.540 --> 01:12:59.340 and I've sort of solved it by writing code, 01:12:59.450 --> 01:13:01.220 like disable the profile or it start 01:13:01.460 --> 01:13:04.140 and then enable it for certain things I'm interested in. 01:13:04.480 --> 01:13:07.140 But there's just so much overhead about the startup 01:13:07.740 --> 01:13:09.120 of your app that's just noise, 01:13:09.410 --> 01:13:09.540 right? 01:13:09.660 --> 01:13:11.120 Like all the imports, 01:13:11.300 --> 01:13:11.660 all the other, 01:13:12.040 --> 01:13:13.080 you're like, I don't care about that. 01:13:13.130 --> 01:13:14.920 I just want to know once it's up and running, 01:13:15.420 --> 01:13:16.220 what does this take? 01:13:16.360 --> 01:13:16.840 You know what I mean? 01:13:17.420 --> 01:13:20.320 And for an app that's sort of up and running for a while, 01:13:20.720 --> 01:13:21.840 just attach to it, 01:13:22.030 --> 01:13:23.760 do the couple of things you're going to do 01:13:23.860 --> 01:13:25.440 that invoke that and then detach, 01:13:25.960 --> 01:13:27.560 gets a really focused view, I would imagine. 01:13:27.980 --> 01:13:28.460 Right. Yes, 01:13:28.660 --> 01:13:29.220 absolutely. 01:13:29.660 --> 01:13:31.580 So getting a little bit short on time, 01:13:32.060 --> 01:13:33.820 but one more thing I want to ask you about, 01:13:34.330 --> 01:13:35.540 because there's this profiling mode, 01:13:35.560 --> 01:13:36.600 which I don't fully understand. 01:13:37.140 --> 01:13:39.680 I know you guys can tell me we have wall clock mode, 01:13:39.980 --> 01:13:42.300 CPU mode, we have GIL mode, 01:13:42.310 --> 01:13:43.520 and we have exception mode. 01:13:43.570 --> 01:13:44.340 They all sound useful. 01:13:44.710 --> 01:13:45.420 But what are these? 01:13:45.570 --> 01:13:45.980 What are these? 01:13:46.610 --> 01:13:46.700 Yeah, 01:13:46.880 --> 01:13:47.240 yeah, yeah. 01:13:47.660 --> 01:13:47.840 So, 01:13:48.180 --> 01:13:48.440 you know, 01:13:48.740 --> 01:13:49.280 okay, 01:13:49.840 --> 01:13:50.220 interesting. 01:13:50.460 --> 01:13:52.500 So the idea here is that, 01:13:52.680 --> 01:13:52.840 you know, 01:13:53.280 --> 01:13:55.920 the profiler will look at your application 01:13:55.940 --> 01:13:57.840 time to time, I will say like, okay, what's going on? 01:13:58.060 --> 01:14:00.560 And it will basically take notes of what 01:14:00.600 --> 01:14:01.220 application is doing, 01:14:01.420 --> 01:14:03.040 right? So that's the basics. 01:14:03.520 --> 01:14:06.300 Okay, so these modes are basically filters or the 01:14:06.340 --> 01:14:06.580 samples. 01:14:06.980 --> 01:14:07.720 The reason is the following. 01:14:07.920 --> 01:14:10.580 Imagine that this is very common. Okay, I will tell you an 01:14:10.680 --> 01:14:11.900 actual case that happened here, right? 01:14:12.120 --> 01:14:14.840 So let's say you have two threads. 01:14:15.320 --> 01:14:16.020 Let's say one of those 01:14:16.120 --> 01:14:18.020 threads is waiting for the network. 01:14:18.280 --> 01:14:20.500 So it's blocking on select or listen or whatever, 01:14:20.680 --> 01:14:20.760 right? 01:14:21.360 --> 01:14:23.560 And then the other thread is doing, 01:14:24.000 --> 01:14:24.380 I don't know, 01:14:24.680 --> 01:14:25.860 calculations or whatever it is, 01:14:26.080 --> 01:14:26.140 right? 01:14:26.410 --> 01:14:28.180 And then you profile those threads. 01:14:28.350 --> 01:14:32.040 And then if you can look individually at every thread, but if you look at the whole thing 01:14:32.280 --> 01:14:35.280 together, it will tell you, okay, half of the time you're spending listening on the 01:14:35.360 --> 01:14:38.600 network and half of the time you're calculating pi, right? 01:14:38.980 --> 01:14:39.940 So in this case, 01:14:40.400 --> 01:14:41.280 the network is noise. 01:14:41.620 --> 01:14:44.480 I don't care about that because I cannot improve the network. 01:14:44.610 --> 01:14:45.080 You know what I mean? 01:14:45.120 --> 01:14:46.220 I cannot just make it fast. 01:14:46.320 --> 01:14:47.340 Or let's say I'm sleeping. 01:14:47.470 --> 01:14:48.760 I call sleep, sleep five. 01:14:49.100 --> 01:14:50.980 then the provider will correctly tell you, 01:14:51.320 --> 01:14:53.260 you are waiting five seconds on Sleep 5. 01:14:53.590 --> 01:14:55.380 But like, okay, but that's not very useful. 01:14:55.450 --> 01:14:55.960 You know what I mean? 01:14:56.020 --> 01:14:57.160 Like that, I cannot fix that, 01:14:57.400 --> 01:14:57.460 right? 01:14:57.600 --> 01:14:58.280 As part of the, 01:14:58.480 --> 01:14:59.720 and that can be written in a file 01:15:00.280 --> 01:15:00.600 or like, 01:15:00.760 --> 01:15:01.660 you know, IO in general. 01:15:02.140 --> 01:15:03.860 So, but that is through the steel, 01:15:04.100 --> 01:15:04.160 right? 01:15:04.220 --> 01:15:05.480 Because if you wait five seconds, 01:15:05.650 --> 01:15:06.840 your program will take five seconds. 01:15:06.960 --> 01:15:08.400 So if you want to know why is my program 01:15:08.900 --> 01:15:09.780 waiting five seconds, 01:15:10.180 --> 01:15:11.500 it's because it's sleeping for five seconds. 01:15:11.610 --> 01:15:12.620 So that's the correct answer. 01:15:12.880 --> 01:15:14.620 It's sometimes it's a good answer, sometimes it's not. 01:15:14.940 --> 01:15:17.760 So by default, it will tell you that's called wall clock. 01:15:18.000 --> 01:15:23.260 It's called wall clock because if you look at a clock in the wall, it will tell you that your application took five seconds, 01:15:23.500 --> 01:15:23.620 right? 01:15:23.880 --> 01:15:24.780 Even if it wasn't sleeping. 01:15:25.120 --> 01:15:25.580 But of course, 01:15:25.800 --> 01:15:27.900 sometimes we don't want to look at that. 01:15:27.960 --> 01:15:30.660 We want to know when my application is doing something, 01:15:31.080 --> 01:15:32.360 you know, because here it's just waiting, 01:15:32.620 --> 01:15:32.700 right? 01:15:32.760 --> 01:15:34.080 Waiting for data, 01:15:34.760 --> 01:15:36.020 waiting to read disk. 01:15:36.460 --> 01:15:37.720 Maybe your disk is slow. 01:15:38.060 --> 01:15:39.140 Maybe the network's slow. 01:15:39.300 --> 01:15:40.000 Maybe you're sleeping. 01:15:40.560 --> 01:15:42.960 So you want to know when your application is doing something. 01:15:43.300 --> 01:15:44.540 That is called CPU time. 01:15:44.760 --> 01:15:46.420 So it's not time that you will measure. 01:15:46.600 --> 01:15:49.780 like how much time is actually your application running in the CPU. 01:15:50.150 --> 01:15:51.800 So when you run it in mode CPU, 01:15:52.490 --> 01:15:55.600 you are filtering all those things that your application is waiting. 01:15:55.860 --> 01:16:00.480 So you're filtering all the samples when you're sleeping or waiting for network. 01:16:00.910 --> 01:16:04.180 And then you only get the samples when your application is doing something, 01:16:04.440 --> 01:16:06.000 which are the ones that you can actually improve, 01:16:06.280 --> 01:16:06.340 right? 01:16:06.440 --> 01:16:10.180 Because like now I can look at a better algorithm or I can look at better module 01:16:10.740 --> 01:16:10.960 or, 01:16:11.240 --> 01:16:11.340 you know, 01:16:11.500 --> 01:16:12.240 I'm parsing JSON. 01:16:12.920 --> 01:16:13.100 So, 01:16:13.500 --> 01:16:13.660 you know, 01:16:13.840 --> 01:16:15.900 that both are interesting because for instance, 01:16:16.060 --> 01:16:16.920 let's say you're parsing JSON, 01:16:17.080 --> 01:16:18.340 you need to read the JSON. 01:16:18.660 --> 01:16:20.220 And for that, you need to wait for your drive 01:16:20.330 --> 01:16:21.680 or the network to give you the JSON. 01:16:22.060 --> 01:16:23.480 That might be interesting to know 01:16:23.840 --> 01:16:26.080 that 30% of the whole time is you waiting 01:16:26.280 --> 01:16:28.400 for the JSON to be given to you. 01:16:28.740 --> 01:16:30.120 So that is all useful information. 01:16:30.740 --> 01:16:30.900 Now, 01:16:31.360 --> 01:16:32.960 once you know that, you cannot improve that 01:16:33.040 --> 01:16:33.900 because that is, you know, 01:16:34.300 --> 01:16:35.820 you tear or the file system, 01:16:36.360 --> 01:16:38.980 but you can know that you're parsing JSON and that's slow. 01:16:39.010 --> 01:16:41.620 So you can say, okay, I can use a better JSON parser, right? 01:16:41.640 --> 01:16:45.200 Right, or JSON or RISM rather than built in JSON 01:16:45.220 --> 01:16:45.820 and see how that works. 01:16:45.900 --> 01:16:46.000 Yeah. 01:16:46.480 --> 01:16:46.860 Exactly, exactly. 01:16:47.020 --> 01:16:48.320 So that is wall and CPU. 01:16:48.540 --> 01:16:50.560 And then we have like the most exotic ones, 01:16:50.740 --> 01:16:52.100 which is like GIL mode. 01:16:52.260 --> 01:16:54.500 So GIL mode tells you how much time 01:16:54.660 --> 01:16:56.060 are you spending on the GIL. 01:16:56.480 --> 01:16:57.440 And this is a bit of a different, 01:16:58.100 --> 01:16:59.920 is a bit of the same idea. 01:17:00.140 --> 01:17:02.960 The idea here is that if you have, 01:17:03.140 --> 01:17:03.400 for instance, 01:17:03.620 --> 01:17:05.240 some program that is spending time 01:17:05.480 --> 01:17:06.900 without the gill, like for example, 01:17:07.160 --> 01:17:07.440 NumPy, 01:17:07.600 --> 01:17:07.680 right? 01:17:08.360 --> 01:17:10.620 You cannot optimize NumPy because that's a SQL, 01:17:11.020 --> 01:17:11.120 right? 01:17:11.160 --> 01:17:12.860 Like you cannot even see what's going on there normally. 01:17:13.580 --> 01:17:14.460 Spoilers in a second. 01:17:14.800 --> 01:17:17.920 But like you cannot see inside that because no Python code. 01:17:18.360 --> 01:17:20.620 So what happens is that because you, you, 01:17:20.840 --> 01:17:20.900 you, 01:17:21.220 --> 01:17:22.200 that thing is running without the 01:17:22.320 --> 01:17:23.720 guild, you don't have access to it. 01:17:23.860 --> 01:17:24.360 So by default, 01:17:24.640 --> 01:17:27.420 it will tell you who is calling that code and it will associate 01:17:27.620 --> 01:17:29.520 all the time that you're spending in NumPy to a caller. 01:17:29.820 --> 01:17:29.960 Right? 01:17:30.060 --> 01:17:32.960 So let's say the function calling NumPy is called call NumPy. 01:17:33.280 --> 01:17:35.880 So it will tell you that call NumPy is 10 seconds. 01:17:36.200 --> 01:17:36.280 Right. 01:17:36.520 --> 01:17:39.520 But let's say that for 10 of those, for nine of those seconds, 01:17:40.080 --> 01:17:41.260 the guild is not 01:17:41.360 --> 01:17:41.500 held, 01:17:41.740 --> 01:17:43.800 of course that is on CPU, so it's doing actual work. 01:17:44.080 --> 01:17:47.080 But because the GIL is not held, some other thread could do something, 01:17:47.500 --> 01:17:47.580 right? 01:17:47.860 --> 01:17:52.780 So you may have a situation where you have many threads and then you want to know who 01:17:52.940 --> 01:17:57.560 is holding the GIL to know what is preventing other threads to run. 01:17:57.930 --> 01:18:02.320 Because if you filter all the samples by only the samples that happen with the gill, 01:18:02.540 --> 01:18:02.760 you 01:18:02.790 --> 01:18:04.620 know that that thing, as long as it's running, 01:18:04.940 --> 01:18:05.900 nothing else is running, 01:18:06.340 --> 01:18:06.460 right? 01:18:06.700 --> 01:18:07.680 So that's interesting. 01:18:08.060 --> 01:18:09.320 So that's another way to cut. 01:18:09.540 --> 01:18:10.160 You may say, well, 01:18:10.340 --> 01:18:11.580 that sounds very complicated. 01:18:11.920 --> 01:18:13.620 Well, it's more complicated than CPU or wall. 01:18:13.860 --> 01:18:15.200 is harder to understand. 01:18:15.640 --> 01:18:18.620 But if you are, you know, already very deep into the weeds 01:18:18.860 --> 01:18:20.440 and like you are very advanced, 01:18:20.580 --> 01:18:21.220 and this is a bit of, 01:18:21.920 --> 01:18:23.360 I want to say this very explicitly, 01:18:23.600 --> 01:18:25.360 like something that we try to do with Tachyon 01:18:25.820 --> 01:18:28.440 is that it tries to be useful 01:18:28.660 --> 01:18:31.000 to a very big range of developers. 01:18:31.360 --> 01:18:31.680 Tachyon, 01:18:31.880 --> 01:18:33.460 if you just want a quick answer, 01:18:33.860 --> 01:18:35.120 you just want to know what is going on, 01:18:35.320 --> 01:18:36.040 just tell me what is wrong. 01:18:36.360 --> 01:18:37.260 Like it will give you that. 01:18:37.360 --> 01:18:38.560 You don't need to know flame graphs. 01:18:38.900 --> 01:18:39.740 You don't need to know, 01:18:40.220 --> 01:18:41.060 like to read this thing. 01:18:41.060 --> 01:18:42.380 It will tell you like, bam, bam, bam. 01:18:42.460 --> 01:18:43.840 These are the five functions that are slower. 01:18:44.160 --> 01:18:44.280 Bam. 01:18:44.480 --> 01:18:45.080 And then you go and look, 01:18:45.320 --> 01:18:45.380 right? 01:18:45.660 --> 01:18:50.520 If you are a bit more advanced, you can start asking for frame graphs or heat maps or, 01:18:50.680 --> 01:18:50.760 you 01:18:50.760 --> 01:18:50.780 know, 01:18:50.960 --> 01:18:51.920 read a bit of these things. 01:18:52.340 --> 01:18:53.760 And if you're really an expert, 01:18:54.180 --> 01:18:56.680 there is like extremely complicated things that you can ask 01:18:56.780 --> 01:18:57.200 this profiler. 01:18:57.420 --> 01:18:57.840 Like, for instance, 01:18:58.320 --> 01:19:00.860 the exception mode is the same thing, but it's asking the question, 01:19:01.020 --> 01:19:02.860 how much time I spend handling exceptions? 01:19:03.400 --> 01:19:03.980 So like errors, 01:19:04.160 --> 01:19:05.840 how much error handling is costing me? 01:19:06.080 --> 01:19:07.120 Of course, you will say, well, 01:19:07.280 --> 01:19:08.380 that probably is nothing. 01:19:09.000 --> 01:19:09.100 But, 01:19:09.280 --> 01:19:09.440 you know, 01:19:09.620 --> 01:19:10.000 if you are already like... 01:19:10.000 --> 01:19:10.840 It can be a lot. 01:19:11.120 --> 01:19:11.240 Yeah, 01:19:11.420 --> 01:19:11.680 exactly. 01:19:11.740 --> 01:19:13.180 You don't know if you are fine tuning. 01:19:13.840 --> 01:19:16.540 And also exceptions are used for much more than exceptions. 01:19:16.610 --> 01:19:16.980 For instance, 01:19:17.200 --> 01:19:18.140 as you very well know, 01:19:18.580 --> 01:19:20.220 the iterator protocol raises, 01:19:20.440 --> 01:19:21.680 you know, a stop iteration 01:19:22.240 --> 01:19:23.320 as exception control. 01:19:23.450 --> 01:19:24.060 So that can be, 01:19:24.300 --> 01:19:24.440 you know. 01:19:24.800 --> 01:19:28.420 And then there is also opcode mode where you can look at individual opcodes. 01:19:28.650 --> 01:19:31.960 So there is very complicated questions that you can ask your program. 01:19:32.300 --> 01:19:32.940 And in particular, 01:19:33.460 --> 01:19:39.520 CPU wall exception and GIL are filters over the samples such that you 01:19:39.480 --> 01:19:43.460 can like know what is your application exactly doing and what is not doing. 01:19:43.620 --> 01:19:43.740 Yeah. 01:19:43.900 --> 01:19:44.340 Very interesting. 01:19:44.660 --> 01:19:48.760 I mean, it's super simple example of exceptions costing is I'm parsing a CRV, 01:19:48.980 --> 01:19:49.340 a CSV, 01:19:49.540 --> 01:19:49.820 and it's 01:19:49.920 --> 01:19:50.760 coming in as text. 01:19:51.040 --> 01:19:51.840 And I think it's a number. 01:19:51.920 --> 01:19:54.360 So we do try this thing. 01:19:54.860 --> 01:19:55.160 Excellent. 01:19:55.520 --> 01:19:56.300 Well, I guess it's, 01:19:56.460 --> 01:19:57.480 you know, set it to none, right? 01:19:58.000 --> 01:20:02.320 You could regex check that or some, you could do something else instead of an exception in 01:20:02.380 --> 01:20:02.880 a tight loop. 01:20:03.080 --> 01:20:03.380 Exactly. 01:20:03.380 --> 01:20:04.380 You get a big bonus. 01:20:04.940 --> 01:20:05.340 Absolutely. 01:20:05.520 --> 01:20:05.660 Absolutely. 01:20:06.000 --> 01:20:06.920 That's exactly it. 01:20:06.940 --> 01:20:07.040 Right. 01:20:07.480 --> 01:20:13.420 And there's some modes when you don't filter by it, like there's a secret mode that only runs when you use one of the outputs, 01:20:13.800 --> 01:20:14.280 which is Gecko. 01:20:15.000 --> 01:20:18.420 So we don't advertise the mode because it only makes sense to this mode, 01:20:18.780 --> 01:20:19.680 to this format, 01:20:19.840 --> 01:20:20.460 which is Gecko. 01:20:20.960 --> 01:20:22.040 But it's called All. 01:20:22.340 --> 01:20:25.160 And what All does is that it records all the samples, 01:20:25.460 --> 01:20:28.800 but it kind of like colors the sample with what was going on there. 01:20:29.160 --> 01:20:31.260 So if the sample is on CPU, it says CPU. 01:20:31.480 --> 01:20:31.940 If it's not. 01:20:32.240 --> 01:20:37.200 So in that way, you can see kind of like a bunch of lanes of what every thread was doing. 01:20:37.600 --> 01:20:41.520 So for example, if you have two threads and the threads are both calculating CPU, 01:20:41.830 --> 01:20:42.880 they will fight for the guild. 01:20:43.170 --> 01:20:45.060 So they will kind of move the guild back and forward. 01:20:45.470 --> 01:20:49.320 And then you can see in the timeline what thread has the guild and for how much. 01:20:49.680 --> 01:20:50.420 So that you can see, oh, 01:20:50.660 --> 01:20:50.840 you know, 01:20:51.080 --> 01:20:52.840 this thread ran for like 20 milliseconds. 01:20:53.060 --> 01:20:54.800 And then this other thread ran for like 10 milliseconds. 01:20:54.990 --> 01:20:56.760 And you can see how it's been bounced. 01:20:57.100 --> 01:20:59.720 So you can like calculate maximum throughput and things like that. 01:21:01.140 --> 01:21:03.420 Yeah, that contention is a big challenge of multi-threading. 01:21:04.080 --> 01:21:04.160 Exactly. 01:21:04.660 --> 01:21:06.640 You said that stuff doesn't scale linearly, 01:21:06.970 --> 01:21:08.640 like 10 threads don't give you 10x performance, 01:21:08.790 --> 01:21:10.160 and that's a good reason right there. 01:21:10.470 --> 01:21:10.560 Exactly, 01:21:10.780 --> 01:21:13.260 and this can give you a more or less good answer for that, 01:21:13.390 --> 01:21:14.000 perhaps not perfect, 01:21:14.200 --> 01:21:14.740 but... 01:21:14.890 --> 01:21:15.960 You guys, there is so much here. 01:21:16.290 --> 01:21:17.200 There is so much here. 01:21:17.760 --> 01:21:19.440 But we're also so much out of time, I think. 01:21:19.880 --> 01:21:21.120 I mean, people should check this out. 01:21:21.260 --> 01:21:23.120 There's really a lot that you can... 01:21:23.140 --> 01:21:24.600 I'm super impressed with this, 01:21:24.760 --> 01:21:25.760 Pablo and Laszlo. 01:21:25.820 --> 01:21:26.200 Well, thank you. 01:21:26.360 --> 01:21:26.860 Thank you very much. 01:21:27.520 --> 01:21:28.040 Thank you. 01:21:28.160 --> 01:21:31.980 There was a really good presentation on PyCon this year, 01:21:32.160 --> 01:21:33.680 which has a little bit more info. 01:21:34.300 --> 01:21:36.840 I also remember because you mentioned how much there is here. 01:21:36.850 --> 01:21:43.380 And I remember working on the slide and I was just panicking how we're going to get all this information into this half an hour talk. 01:21:43.540 --> 01:21:43.920 What's our talk? 01:21:44.900 --> 01:21:45.220 30 minutes? 01:21:45.370 --> 01:21:45.860 Oh, uh-oh. 01:21:46.740 --> 01:21:47.000 No, no, 01:21:47.160 --> 01:21:47.980 it's impossible. 01:21:49.700 --> 01:21:50.140 Yeah, 01:21:50.930 --> 01:21:51.440 that presentation. 01:21:51.630 --> 01:21:54.880 But I also recommend that for anyone who's interested. 01:21:55.010 --> 01:21:57.380 I think that has a pretty good overview. 01:21:57.900 --> 01:21:58.060 Yeah, 01:21:58.320 --> 01:22:00.120 I'll definitely link to your talk on YouTube. 01:22:00.680 --> 01:22:03.440 There is only one thing I want to add before we go, 01:22:03.780 --> 01:22:07.420 which I think is interesting and it's also a bit, you know, selfish, 01:22:07.920 --> 01:22:09.940 which is that, so Tachyon is really good. 01:22:10.640 --> 01:22:13.960 The only interesting challenge of Tachyon is that, 01:22:14.640 --> 01:22:17.780 you know, if your application spends a bunch of time in C code, 01:22:18.120 --> 01:22:21.580 like NumPy or, you know, Pandas or Polars or like nowadays, 01:22:21.880 --> 01:22:25.060 like all the machine learning stuff like CUDA or whatever it is. 01:22:25.580 --> 01:22:25.880 Unfortunately, 01:22:26.280 --> 01:22:26.360 again, 01:22:26.520 --> 01:22:28.560 like Tachyon is a Python profiler. 01:22:28.610 --> 01:22:29.820 So it only sees Python, right? 01:22:30.160 --> 01:22:32.540 It is who calls that C code, but not the C code. 01:22:32.820 --> 01:22:38.820 And of course, you can say as a user, you are not super interested in what is happening on the guts of NumPy. 01:22:39.040 --> 01:22:40.180 But sometimes you are, right? 01:22:40.280 --> 01:22:45.160 Because what is going on or why my application is low, sometimes you need to just go there. 01:22:45.320 --> 01:22:47.640 That's the unfortunate life we live in. 01:22:47.840 --> 01:22:51.280 And Tachyon cannot analyze C code because it's a Python profiler. 01:22:51.340 --> 01:22:52.160 It's not a C profiler. 01:22:52.580 --> 01:22:55.420 On the other side, a C profiler cannot analyze Python. 01:22:55.680 --> 01:22:57.980 So you can only see C code. 01:22:58.180 --> 01:23:00.000 So you will see the internals of CPython, 01:23:00.480 --> 01:23:01.780 but it will not have Python names. 01:23:01.820 --> 01:23:03.500 It will not tell you full bar batch. 01:23:03.540 --> 01:23:05.200 It will tell you by valuable frame default, 01:23:05.560 --> 01:23:06.280 and that's not useful. 01:23:06.700 --> 01:23:08.540 So that is one big challenge. 01:23:08.700 --> 01:23:12.080 And the other is that the moment you want to go into the world of C profilers, 01:23:12.520 --> 01:23:13.520 complexity explodes. 01:23:14.620 --> 01:23:17.160 It's just like, if this is hard, which is hard, 01:23:17.600 --> 01:23:18.700 when you want to do C profiles, 01:23:19.080 --> 01:23:20.680 it's like 10 times harder. 01:23:20.800 --> 01:23:22.860 And then when you want to do both at the same time, 01:23:23.040 --> 01:23:24.380 it's a million times harder. 01:23:24.760 --> 01:23:25.720 So unfortunately, 01:23:25.900 --> 01:23:28.680 we didn't want to add that complexity to the standard library because, 01:23:29.220 --> 01:23:32.020 you know, it's a lot of complexity for core developers to maintain. 01:23:32.320 --> 01:23:33.660 And it's also really, 01:23:33.940 --> 01:23:34.120 really, 01:23:34.420 --> 01:23:35.080 really hard. 01:23:35.400 --> 01:23:38.620 It's extremely hard and it requires a lot of care and, 01:23:38.780 --> 01:23:39.740 you know, bugs will happen. 01:23:40.020 --> 01:23:41.240 So we didn't want to like put. 01:23:41.480 --> 01:23:45.520 So one of the things I've been working on here at Hudson River Trading is a version of 01:23:45.640 --> 01:23:47.300 Tachyon is like the big brother of Tachyon, 01:23:47.480 --> 01:23:48.320 which is called Cronon, 01:23:48.840 --> 01:23:49.580 which is also physics 01:23:49.600 --> 01:23:55.500 related because like the in quantum mechanics if you quantize time the particle of time is called 01:23:55.500 --> 01:23:59.900 a chronon so you know because it's time because it's processed so we call it chron and this is a 01:24:00.000 --> 01:24:06.800 profiler we have been using for nine months already here for with extreme success extreme success like 01:24:06.800 --> 01:24:11.220 the whole company is using it all the time and this profiler does everything that tackian does 01:24:11.470 --> 01:24:18.480 but it's also multiple versions of python so you can profile 312 313 314 315 all that you want and 01:24:18.500 --> 01:24:20.500 also profile C code and kernel code. 01:24:20.680 --> 01:24:22.340 So you can see literally everything. 01:24:22.620 --> 01:24:23.300 Like you can see something 01:24:23.520 --> 01:24:23.780 like, 01:24:24.180 --> 01:24:25.720 oh, your application is calling, 01:24:26.020 --> 01:24:28.980 I don't know, calculating Python and that goes into NumPy 01:24:29.180 --> 01:24:30.440 and that's creating a bunch of arrays. 01:24:30.860 --> 01:24:33.720 But one of those arrays just trigger a page fall in the kernel 01:24:33.960 --> 01:24:37.880 and then the kernel is like trying to find memory for you. So you can see how much that takes. 01:24:38.820 --> 01:24:39.700 And it supports the streaming. 01:24:39.920 --> 01:24:43.480 So you can have like Grafana dashboard with all the profile and 01:24:43.500 --> 01:24:46.580 The Flame Graph is changing as your application is running, 01:24:46.780 --> 01:24:48.780 so you don't need to even wait for it. 01:24:49.050 --> 01:24:50.320 You don't need to attach even. 01:24:50.610 --> 01:24:53.760 You can just keep it 24-7 and see the data as it comes in. 01:24:54.820 --> 01:24:56.260 It supports all sorts of formats, 01:24:56.560 --> 01:24:57.420 all sorts of tools. 01:24:58.300 --> 01:25:00.500 It works on macOS as well and Linux. 01:25:02.050 --> 01:25:05.680 So we are going to make it open source soon in a month or two. 01:25:06.020 --> 01:25:08.820 Yes, and that's going to be a big deal, big deal. 01:25:09.120 --> 01:25:09.380 Cool. 01:25:09.480 --> 01:25:09.980 Secret sauce. 01:25:10.720 --> 01:25:11.740 Yeah, that sounds amazing. 01:25:12.100 --> 01:25:12.260 I mean, 01:25:12.500 --> 01:25:12.620 already, 01:25:12.820 --> 01:25:13.940 Takion is very amazing. 01:25:14.480 --> 01:25:16.840 Alas, it will be known as profiling dot sampling. 01:25:17.480 --> 01:25:18.280 Yes, yes, yes. 01:25:18.600 --> 01:25:19.280 We now know. 01:25:19.280 --> 01:25:20.100 Most people call it Takion. 01:25:20.240 --> 01:25:21.340 Most people call it Takion. 01:25:21.680 --> 01:25:22.340 Call it Takion. 01:25:22.480 --> 01:25:23.300 That's the real name. 01:25:23.440 --> 01:25:28.000 The other one is the, you know, the name that you need to use when you go to boring meetings 01:25:28.580 --> 01:25:28.860 with, like, 01:25:29.140 --> 01:25:30.920 suits and, like, you know, companies. 01:25:31.100 --> 01:25:31.440 Companies, oh, 01:25:31.680 --> 01:25:32.360 who are you Takion? 01:25:32.440 --> 01:25:33.900 No, no, I'm profiling dot sampling. 01:25:34.030 --> 01:25:34.100 Oh, 01:25:34.360 --> 01:25:34.600 okay, okay. 01:25:35.080 --> 01:25:35.480 But, like, no, 01:25:35.660 --> 01:25:35.780 no. 01:25:36.120 --> 01:25:36.760 When you go to the party. 01:25:36.870 --> 01:25:37.500 We trust this one, 01:25:37.670 --> 01:25:37.760 yeah. 01:25:37.980 --> 01:25:39.880 It's not one of the more weird physics things. 01:25:40.400 --> 01:25:44.000 Everyone should just call it Tachyon until it spreads and it just becomes the name. 01:25:44.300 --> 01:25:44.360 Yeah, 01:25:44.540 --> 01:25:44.580 yeah. 01:25:44.820 --> 01:25:47.380 No one will understand what you're talking about profiling or something. 01:25:47.620 --> 01:25:50.440 If you talk to your manager, you say, oh, I use profiler or something. 01:25:50.780 --> 01:25:51.040 Oh, 01:25:51.200 --> 01:25:51.660 okay. 01:25:51.940 --> 01:25:52.380 But like, no, 01:25:52.620 --> 01:25:56.900 with your friends, you say, I use Tachyon and I fucking did it. 01:25:57.020 --> 01:25:57.200 Oh, 01:25:57.400 --> 01:25:57.480 okay. 01:25:57.720 --> 01:25:58.180 That's right. 01:25:58.460 --> 01:25:59.340 Who is this Tachyon? 01:25:59.480 --> 01:26:00.240 Why does he solve it? 01:26:00.260 --> 01:26:01.000 Yeah, who is this guy? 01:26:01.140 --> 01:26:01.580 Who is this guy? 01:26:02.000 --> 01:26:02.080 Well, 01:26:02.400 --> 01:26:02.960 thank you guys, 01:26:03.140 --> 01:26:03.460 Pablo, 01:26:03.900 --> 01:26:04.200 Laszlo. 01:26:04.380 --> 01:26:04.920 Thanks for being on the show. 01:26:05.460 --> 01:26:07.340 Awesome work on getting this out to the world. 01:26:07.480 --> 01:26:09.160 This is going to be super cool for everyone. 01:26:09.420 --> 01:26:09.820 It's going to be great. 01:26:10.000 --> 01:26:11.280 Thank you so much for hiring us, 01:26:11.440 --> 01:26:13.780 and I hope people enjoy it. 01:26:13.920 --> 01:26:14.540 I'm sure they will. 01:26:14.940 --> 01:26:15.160 All right. 01:26:15.540 --> 01:26:15.820 Thanks a lot. 01:26:16.180 --> 01:26:16.440 Bye, guys. 01:26:16.440 --> 01:26:16.720 See you. 01:26:16.980 --> 01:26:17.300 Thank you. 01:26:18.220 --> 01:26:20.640 This has been another episode of Talk Python To Me. 01:26:21.020 --> 01:26:21.740 Thank you to our sponsors. 01:26:21.980 --> 01:26:23.240 Be sure to check out what they're offering. 01:26:23.440 --> 01:26:24.800 It really helps support the show. 01:26:25.600 --> 01:26:27.140 This episode is brought to you by Sentry. 01:26:27.460 --> 01:26:31.500 Sentry's new MCP server connects your error reports straight to your coding agent. 01:26:32.040 --> 01:26:33.060 Claude Code reads the errors, 01:26:33.500 --> 01:26:34.100 finds the bugs, 01:26:34.360 --> 01:26:34.960 and fixes them. 01:26:35.480 --> 01:26:38.480 Get started at talkpython.fm/sentry. 01:26:39.120 --> 01:26:41.120 If you or your team needs to learn Python, 01:26:41.310 --> 01:26:49.320 we have over 270 hours of beginner and advanced courses on topics ranging from complete beginners to async code, Flask, Django, 01:26:49.720 --> 01:26:49.880 HTML, 01:26:50.360 --> 01:26:51.400 and even LLMs. 01:26:51.720 --> 01:26:52.100 Best of all, 01:26:52.340 --> 01:26:54.000 there's no subscription in sight. 01:26:54.560 --> 01:26:56.240 Browse the catalog at talkpython.fm. 01:26:56.920 --> 01:27:00.300 And if you're not already subscribed to the show on your favorite podcast player, 01:27:00.500 --> 01:27:01.580 what are you waiting for? 01:27:02.220 --> 01:27:04.020 Just search for Python in your podcast player. 01:27:04.110 --> 01:27:05.000 We should be right at the top. 01:27:05.400 --> 01:27:06.840 If you enjoy that geeky rap song, 01:27:07.030 --> 01:27:08.260 you can download the full track. 01:27:08.500 --> 01:27:10.320 The link is actually in your podcast below or share notes. 01:27:11.220 --> 01:27:11.780 This is your host, 01:27:12.000 --> 01:27:12.460 Michael Kennedy. 01:27:12.790 --> 01:27:13.920 Thank you so much for listening. 01:27:14.150 --> 01:27:14.940 I really appreciate it. 01:27:15.360 --> 01:27:16.060 I'll see you next time. 01:27:34.160 --> 01:27:35.220 Get old. 01:27:36.740 --> 01:27:38.720 We tapped into that modern vibe. 01:27:39.160 --> 01:27:40.120 Overcame each storm. 01:27:40.940 --> 01:27:42.220 Talk Python To Me. 01:27:42.310 --> 01:27:43.600 I sync is the norm.