#405: Testing in Radio Astronomy with Python and pytest Transcript

00:00 So you know about dependencies and testing, right?

00:02 If you're going to talk to your database in your app, you have to decide how to approach that with your test.

00:07 There are a lot of solid options you might pick from, and they vary by goals.

00:12 Do you mock out the DB layer for isolation?

00:14 Or do you use a test DB to make it as real as possible?

00:18 Or do you even just punt and use the real DB for expediency?

00:22 What if your dependency was a huge array of radio telescopes and a rack of hundreds of bespoke servers. That's the challenge on deck for today, where we discuss testing radio astronomy with pytest and our guest, James Smith. He's a digital signal processing engineer at the South African Radio Astronomy Observatory and has some great stories and tips to share. This is Talk Python to Me, episode 405, recorded February 13th, 2023.

01:02 Welcome to Talk Python to Me, a weekly podcast on Python. This is your host, Michael Kennedy.

01:10 Follow me on Mastodon, where I'm @mkennedy and follow the podcast using @talkpython, both on fosstodon.org. Be careful with impersonating accounts on other instances, there are many. Keep up with the show and listen to over seven years of past episodes at talkpython.fm.

01:26 We've started streaming most of our episodes live on YouTube. Subscribe to our YouTube channel over at talkpython.fm/youtube to get notified about upcoming shows and be part of that episode.

01:39 This episode of Talk Python to Me is brought to you by Taipy.

01:42 They're here to take on the challenge of rapidly transforming a bare algorithm in Python into a full-fledged decision support system.

01:48 Check them out at talkpython.fm/taipy and it's brought to you by Sentry.

01:55 Don't let those errors go unnoticed.

01:57 Use Sentry.

01:58 Get started at talkpython.fm/sentry.

02:01 James, welcome to Talk Python to Me.

02:04 Normally, I talk Python to computers, Michael, but this will be a first.

02:08 You are a human, right?

02:10 You gotta ask chatGTP about that, we'll see.

02:12 There's a lot of people who are interested in talking about Python.

02:16 When I first put this podcast together, I thought, "Who's going to be the target audience?" I thought people were really into Python.

02:22 People make things like Flask and stuff.

02:24 It's kind of a big part of the world.

02:26 But there's a ton of people who are scientists, or just curious about programming, who listen as well.

02:32 It really surprised me how many people you can talk Python to, They seem to appreciate it.

02:37 So it's cool.

02:38 - Indeed.

02:39 - Yeah, but I also spend a fair amount of time talking Python to computers as well.

02:43 Sometimes more fun, sometimes more frustrating.

02:45 You never know.

02:47 - Yes, we're familiar with that.

02:49 - I'm sure, I'm sure.

02:50 Well, you have this extra angle of it's not just talking to pure software, right?

02:56 That lives in some internet API vacuum, but you have physical things, many, many physical things in radio astronomy and large arrays of telescopes and receivers and as we're gonna talk about, lots of different things.

03:09 And I think that's one of the really interesting aspects of this episode to talk to you is Python against real world things out there and real time as well.

03:19 - Perhaps sort of to jump back and put the listeners in a bit of context, I work at the South African Radio Astronomy Observatory.

03:25 Our sort of primary project is called MeerKAT.

03:28 It's a precursor of one of the world's biggest radio telescopes, which is planned.

03:33 So the project is called SKA, so it stands for Square Kilometer Array, which refers to the size of the collecting area of the telescope that will eventually be built.

03:42 It's in its beginning stages now.

03:44 And that has a lot of moving parts, right from the actual antennas that are pointing in into the sky and recording radio emissions from the universe somewhere, through down to the point where a scientist actually sits down with a data product in order to analyze to write his paper that he's hoping to publish and win his Nobel Prize.

04:04 And so everything from the beginning of the end there needs some software control.

04:09 And Python is the sort of weapon of choice for all of those applications.

04:13 - Yeah, it's pretty interesting to see what you're going to talk about.

04:17 You gave a couple of presentations at PyCon South Africa, is that right? - Zeday.

04:21 - Zeday. - That's right, yes.

04:23 Last year. - Yeah, fantastic.

04:24 And people, of course, link to those in the show notes and people can check them out.

04:28 a lot of cool graphics and stuff there.

04:31 The fact that you built this real-time aspect involving Python, which is not normal use perhaps, I think is really something that we're gonna have a good time talking about, as well as just how do you test something like a large receiver array rather than, you know, is the user logged in in the database, yes or no, and let's mock that out, right?

04:52 This is both, I think, very similar and yet very different, right?

04:55 - Yes, it can be.

04:57 And there's a lot that you can do, but there's also some interesting gotchas which we can talk about there.

05:04 So, I mean, I don't know where you wanna start.

05:06 - Let's get a real quick introduction with you before we get too far down in this topic.

05:10 And so tell people how you got into programming and astronomy and all that kind of stuff.

05:15 - Well, programming, well, both of those have become, both of those started out as hobbies and they've sort of become my job, which means I don't do them as a hobby anymore, which is always fun.

05:25 I was younger in school, I made the mistake during a summer holiday of telling my dad that I was bored. And so he handed me a book and this book was programming in C, I think. I forget exactly.

05:36 And being young and naive, I thought this was a great idea and started working my way through it.

05:41 So since then, I've been programming more or less continuously. It was sort of the mid to late 2000s that Python started to become more prominent and I picked that up and haven't looked back. Astronomy was also another hobby. In my student days, I spent many South African, cold South African nights looking at the stars. And then a job opportunity opened up at Sareo to do electronic engineering, which was the field that I trained in university.

06:08 That was about seven years ago. And so I've been combining these sort of programming and electronic engineering and my astronomy hobby, which has shifted now into radio instead of optical. But yeah, I basically come from that. Fantastic.

06:20 I remember getting a book in C, thinking, "Oh, I'm going to learn programming," and then read it like, "Okay, this..." When I was pretty young, I was like, "Okay, this is a little bit much to bite off at the moment," but came back to it as well.

06:32 But that's how it was, right?

06:33 We used to get books pre-internet, YouTube, and where you just fired up and say, "Teach me programming in the next half hour." - With listings of code and you have to type them in, and then you miss a semicolon somewhere and scratch your head for half a day.

06:46 (laughing)

06:47 - That's right.

06:48 - But I think that was a good preparation, though.

06:49 I mean, as great as Python is, it is somewhat high level, and I think it's helpful to have a sort of a, even if you don't often work at that level, to have an understanding of what's going on at a slightly lower level than what the computer is doing.

07:00 - It's valuable to have that experience, but I'm also, even though I did it professionally for a couple years, happy that I'm not, I don't have to continuously work at that level, right?

07:09 You can just be so more productive in a higher level language like Python, indeed.

07:13 - Totally. - Yeah.

07:14 So I think you introduced this a little bit, But tell people about SARAO, this project, how'd you get to that place?

07:21 Yeah.

07:22 Sarao, well, I got there by responding to a job advert and interviewing and what have you.

07:29 Sarao sort of started out as a response to the international kind of scientific project to build a square kilometer array telescope.

07:35 The idea is to make it the most sensitive radio telescope that has ever been built.

07:40 And when you're doing that kind of project, you want to build it in an ideal setting, like optical telescopes, if you want to do proper science, you want to do it far away from cities where there's no light pollution.

07:50 Similarly, radio telescopes, you want to do them where there's no radio interference.

07:54 So cell phones, TV signals, Wi-Fi, that kind of stuff.

07:57 The Southern Hemisphere also has a bit of an advantage in that we can see parts of the galaxy that the Northern Hemisphere can't.

08:03 And so, as this project progressed, various sites were identified and ultimately the decision was made to put part of the telescope in South Africa and part of it in Australia.

08:13 And Sarao is really the, just the organization around developing the South African part of that telescope.

08:19 That's where we are.

08:20 - It's a very cool project.

08:21 I think a square kilometer array of telescope, that's pretty impressive, right?

08:26 And also something that's much easier to do with radio than with optical, I would imagine.

08:31 - Yes, when it comes to telescopes, bigger is better, but there reaches a point of where building a bigger one becomes expensive and very difficult.

08:40 And so a trick that you can use in radio is called interferometry. So if you measure your radio waves at different points and you measure them in phase with each other or what they call coherently, then you can get away with making a lot of smaller telescopes to build up the same kind of area, which will give you the same effect as a bigger one, but much cheaper because you can do a lot of smaller, cheaper telescopes.

09:02 If that in a nutshell is really why these telescope arrays exist, it's very difficult to do that with optical telescopes because the wavelength is so short, the frequency is so high, that getting, getting the, it is possible, it can be done, but getting the signal coherent or in phase is very, very difficult.

09:18 - I see.

09:19 - That's why we do it in radio.

09:20 - Yeah, you don't have much time.

09:21 There's not much of a gap between how you've got to coordinate the signals across the radio, but it's even worse in optical, I see.

09:27 - Exactly, exactly.

09:28 - Nice.

09:29 So people who listened to the episode about imaging the black hole, we talked a lot about that.

09:34 So, you know, I don't necessarily need to, think we need to cover too much depth about it.

09:39 But one thing I think that people might find interesting is, I think, remember from your talk that you said, this is part, some of the work that you're doing is part of the deep space network for communicating with, is that right?

09:50 With NASA.

09:52 - South African radio astronomy has its origins in the deep space network.

09:55 So if you're trying to communicate with spacecraft that are outside of earth orbit, you need a very powerful, or something that looks a bit like a radio telescope in much, except that you're transmitting as well, because you need to be able to talk to the, know, Voyager or your Mars Rover that's out there in space. But very inconveniently for scientific purposes, the earth rotates. So for about a third of the day, if you have a telescope in California, for example, I think is where one of NASA's big ones are, you know, half of the day you can't really talk to your, to your spaceships. And NASA built a network of telescopes around the world and one was in Australia and one was here in South Africa. And some of the early Mars missions were actually used the telescope in South Africa for communication.

10:38 But that was, I think in the seventies, I forget my history exactly, but they got a better one in Spain.

10:43 And so the one in South Africa was kind of converted into a facility just for science.

10:48 And it's been operated by our national research foundation, which evolved into Sarao for the last sort of few years.

10:55 And that's what kind of gave us our leg up.

10:56 - Yeah, exactly.

10:57 It was already built and it was there.

10:59 And they're like, you know what?

11:00 Instead of just let it, put it on mothballs, why don't you guys do science with it, right?

11:04 That's cool.

11:04 And we've been using it quite well since then.

11:06 They've done, I wouldn't say groundbreaking science.

11:09 I don't know that any Nobel prizes have happened, but we have some great radio astronomers, originate from South Africa, that have used data from this telescope.

11:15 So, yeah.

11:16 - Excellent.

11:17 All right, let's talk about first the real-time aspect, this thing that you focus on called the correlator, right?

11:24 And maybe give us this side of the story, because I think this alone is pretty interesting, what you've built here with Python, you and your team.

11:32 So from a conceptual point of view, in the episode a few weeks ago, when you interviewed Dr. Sarah Asun, I don't know if I'm pronouncing her name correctly, the Event Horizon telescope worked by pointing different telescopes around the world at the same object. Or all radio, shall I say, all radio telescopes arrays, whether they're very long baseline, like what she used, or much shorter, have something called a correlator. And this is the thing that basically combines the signals from the individual telescopes in a way that is, that enables downstream users or scientists to actually make sense of this data. So in order for that to work, the data needs to be in phase, it needs to be coherent. And at very long baselines, they do this by having each station having their very own atomic clock to generate very precise reference signals.

12:20 - So as a reminder for people maybe didn't hear that episode is, you know, the earth - Earth is curved, as we all know, hopefully.

12:26 And the radio waves come in, and as it hits that sphere, they hit the different parts of the array.

12:32 The more spread out it is, even more exaggerated.

12:34 But it hits the different parts of the array at different times, so you have to offset those back in the signal to say, well, it came in at this angle, and it came in at this time, and the speed of light says this one is a nanosecond behind that one.

12:46 So you gotta figure out how to realign those so it looks like a flat surface that they hit all simultaneously to look like a single picture, right?

12:54 So that's basically the job of this thing.

12:57 - So partially, you can do the first bit with just with geometry.

13:00 We know what the rotation of the earth is.

13:03 We know where our different telescopes are.

13:05 And so we can calculate roughly what the time difference will be.

13:08 The job of the correlator is once you've got those signals and you've applied your sort of rough delay offset to each of those signals, the correlator will find the correlations between the incoming signals simply by multiplying the signals together.

13:20 Mathematically is not terribly complicated.

13:22 It's really just multiplication of each pair of antennas.

13:26 But it's with that, you may remember from your high school mathematics, the trigonometric identity.

13:31 When you multiply two sine waves together, you get a somewhat a difference product.

13:36 And it's a similar kind of concept, applying to not just to abstract sine waves, but to radio waves that are at various frequencies.

13:44 The main engineering challenge is just doing this fast enough for practical purposes.

13:49 So in the example of the Event Horizon Telescope with very long baseline interferometry, it's not possible to do it in real time because your individual elements are just too far apart.

13:59 You can't get all the data together.

14:00 - Not for math, but because the actual data quantity, getting them all back and forth across the world is too hard. - Exactly.

14:08 Exactly.

14:09 So there's specialized hardware that takes those signals and writes them basically just straight onto hard drives.

14:15 Then these hard drives are physically taken to a central location.

14:19 I think Sarah did talk about that in her episode.

14:22 - Yeah, I think it was Boston and maybe Cambridge and London, something.

14:27 - One was in Europe, yes.

14:28 - No, it was Max Planck Institute in Germany is what it was, I believe.

14:30 - That's right, yeah.

14:31 One was in the States, one was in Europe.

14:33 - This portion of "Talk Python to Me" is brought to you by Taipy.

14:38 Taipy is the next generation open source Python application builder.

14:42 With Taipy, you can turn data and AI algorithms into full web apps in no time.

14:47 Here's how it works.

14:48 You start with a bare algorithm written in Python.

14:51 You then use Taipy innovative toolset that enables Python developers to build interactive end-user applications quickly.

14:58 There's a visual designer to develop highly interactive GUIs ready for production, and for inbound data streams, you can program against the Taipy Core layer as well.

15:07 Taipy Core provides intelligent pipeline management, data caching, and scenario and cycle management facilities.

15:13 That's it, you'll have transformed a bare algorithm into a full-fledged decision support system for end-users.

15:19 Taipy is pure Python and open source, and you install it with a simple pip install taipy. For large organizations that need fine-grained control and authorization around their data, there is a paid Taipy Enterprise Edition, but the Taipy core and GUI described above is completely free to use. Learn more and get started by visiting talkpython.fm/taipy.

15:43 The link's in your show notes. Thank you to Taipy for sponsoring the show.

15:46 Yes, and then the same, well, similar equipment does the reverse process. It reads the data off and then it basically batch processes it. This happens in software, so you cross multiply all of the signals together. That's great if you have a limited number of telescopes, up to a few dozen or so, as with the Event Horizon Telescope. In the MeerKAT case, we've got 64 of them and we're building a few more. And as it becomes the full square kilometer array we're going to be talking about hundreds or possibly even thousands of individual telescope elements. So recording everything onto a hard drive at that data rate becomes impractical.

16:24 So the approach that we take is we use what we call a real-time correlator. It processes and reduces the data as a sort of first stage in real-time live before recording to the disk and as a part of that step we integrate for a little while. So if you think about it from point of view if you've done some photography at night time you want to do a a long exposure, so you open your camera shutter for a long time to get them all signal.

16:47 It's very similar to that.

16:48 So we would integrate over, you know, half a second or a second or, you know, eight seconds.

16:52 And that gives you a much reduced, there's still a lot of data that comes out of the other side, but it's much, much smaller than if you were recording straight onto hard drives from the telescope, the way that they do in Event Horizon Telescope.

17:05 - Right, that is interesting.

17:06 I guess it is just like taking a picture, just a different frequency, right?

17:10 But same basic idea?

17:11 - Exactly, exactly.

17:12 - Nice, and so there's a lot of data coming through here.

17:15 You talked about how Event Horizon couldn't even ship it around the world quick enough, or even though they were very, very far apart and pretty remote, but there's a lot of data here.

17:23 And so maybe give people a sense of the data center.

17:26 Excuse me, give people a sense of the data center.

17:28 And like, you've got this array of, I don't remember how many of these correlators, server, U1 slices you've got, but it's not just one corner, a PC in the corner, is it?

17:38 - No, no.

17:40 Although with the way that technology progresses, maybe one day, no. So our current generation Correlator uses an FPGA based computing platform. We call it Scarab. There's a bit of a tradition to name radio astronomy compute things after insects. So the previous generation one was called Roach. This one is called Scarab. They stand for something. Reconfigurable Open Architecture for Computing Hardware, I think was the Roach. And Scarab, the first three letters is ISKA. Yeah, there we go. There's the scarab. And what that is, is an FPGA is a field programmable gate array. So it's a little piece of reconfigurable silicon that has, it's like having a hardware accelerated signal processing. It's very fast and it's wired straight into Ethernet for interconnect. So in our data center in the Karoo in South Africa, we've got about 280 of these individual scarabs and each of them has a role to play in the signal processing pipeline. They talk to each other via Ethernet and they're controlled by a central master controller computer that runs Python. So it uses an asynchronous little routine to coordinate the activities of each of these scarabs. The processing parameters need to be updated from time to time. So for example, as the earth turns, the geometric delay between a given pair of telescopes will change slightly. And so that gets updated periodically in the scarab so that it can carry on processing the data at the full rate that the telescope is taking data in.

19:06 So to give you an idea of that, each telescope is producing data at about 35 gigabit per second.

19:12 - 35 gigabit a second?

19:13 - Yeah, times 64.

19:14 - Yeah, that's a lot of data.

19:18 - After the processing, the other end of the correlator, so we get about four or five gigabit per second, so that's reduced.

19:24 Due to kind of the long exposure effect, it's averaged over a few seconds, so it's four or five gigabits per second out the other end.

19:31 And then that gets stored on hard drives for processing and imaging so that scientists can do their science later on.

19:36 But that's the initial stage of the signal processing pipeline.

19:39 - Yeah, so you've got 64 of these running in concert.

19:43 What is uptime in DevOps look like for you all?

19:46 Is this continuously 24 hours a day recording or is it offline for a certain number of hours?

19:52 - Yes, it is.

19:53 And the limiting factor I'm happy to say is usually not the correlator.

19:57 We don't need all 280 scarabs to run.

20:01 We can run with about 190 of them with full science capability, so there is capacity for spares.

20:06 And when you have a system with this many moving parts, there is downtime.

20:10 You know, people actually need to go and change the oil in the motors on the telescope and routine maintenance kinds of things like that.

20:16 I don't remember our numbers exactly, but I believe it's above 75% of the time that the telescope is busy and doing science.

20:24 This is more recent.

20:25 It's only in the last sort of few years that the technology platform has become a bit more mature.

20:30 In the early stages, it was really still sort of engineering commissioning, but I think we're using above 75% of time now for actual science observations.

20:37 - That's pretty impressive for considering all the pieces involved.

20:40 And what are these scarabs?

20:42 Each one is basically assigned to an individual telescope, part of the array?

20:46 - Partially.

20:47 So the first one will do what is called channelization.

20:51 So it's much easier to operate on narrow signals of, you know, very close to a sine wave.

20:56 So the telescope will sample a little bit more than a gigahertz with the bandwidth and that will be chopped up into, you know, a few thousand channels.

21:04 Then those channels will get sort of dished out and another scarab later on will cross multiply the corresponding frequency channels from every single telescope together.

21:14 So this architecture is called FX.

21:16 So first frequency generalization, then cross-correlation, cross-correlation, pardon me, before you get your ultimate, well, product which gets stored in the archive and on which then further science process is done.

21:27 - People start to ask questions and draw graphs and things.

21:31 - Exactly.

21:31 - Yeah, so one of the parts that you employ to make this run fast is, I think you said you use CUDA, CUDA cores on NVIDIA GPUs or something like that.

21:41 Is that right?

21:41 Do I remember this correctly?

21:42 - Yes, that's right.

21:43 So the Scarab that you showed on the screen, that's our previous but current generation correlator is working on that platform.

21:51 It has its pros and cons.

21:53 It's kind of a bespoke piece of hardware.

21:54 It is reconfigurable, but there are not that many customers for this particular board.

21:59 The problem with doing your own hardware is that it takes a lot of time, particularly if you're an organization that you want to do science.

22:06 So for the next expansion of MeerKAT, to build more antennas, we need more processing capability to be able to handle more bandwidth. We've kind of figured out that where they are now, GPUs can do the work. GPUs have actually been powerful enough to do the work for a while now. The problem has actually been the memory bandwidth to actually get your data from your telescope onto your GPU to do the number crunching. So for Scarab, it was very easy. It has some 40 gigabit ethernet and that's wired straight into the FPGA. So there's no operating system taking its time, there's no PCI Express that needs to be negotiated.

22:41 The data just comes straight off the network and it can do its processing.

22:45 Previously computers were not fast enough to get the numbers on and off the card.

22:48 But since PCIE 4 has become a thing and sort of the GeForce RTX 3000 series cards, which we're using PCIE 4, we can do it on GPUs now.

22:59 So we haven't deployed one yet.

23:01 What I spoke about is the prototype that we've developed.

23:03 And that's part Python and part CUDA, right?

23:07 Yes, yes. So the CUDA is actually quite small. As I mentioned earlier, the processing is quite quite simple. So there's a stage for channelizing. So that uses a piece of math called a Fourier transform. And there's been years of research gone into that to make it very computationally efficient. It's very fast. The other part is simply multiplying lots of numbers together.

23:27 And that's something that GPUs are really, really good at. So we've made good use of technology advances driven by things like AI and machine learning, which relies on really large matrix multiplications. So we just piggyback off of that technology. And the bonus is we don't have to develop our own hardware anymore, which is nice.

23:44 And those things are only getting faster, right? If you could have done it on a 3070 or whatever, an NVIDIA 3070, then, you know, it's getting faster and going to get easier and easier as things go. You might need a small power generator to run one of those cards these days. They're kind of insane, but...

24:00 The 30 series are a little bit easier than the 40 series. To give you an idea, in our - Our prototype system, we're using 3070 TIs at the moment, and one of those GPUs does the work of four Scarabs.

24:10 - Wow.

24:10 - So it's the rate that technology moves forward.

24:13 To be fair, the Scarab is about six or seven years old.

24:16 It was revolutionary at the time, but commercial technology has moved forward a lot.

24:20 - Yeah, it sure has.

24:22 You talked, just a bit of a sidebar, you talked about one of the challenges being getting the data in and out of the GPUs from a bandwidth perspective.

24:30 Do things like systems on a chip in this Apple unified memory where the bandwidth, the memory of the CPU is the memory of the GPU.

24:39 Does that, have you guys considered playing with that?

24:41 Is that interesting?

24:42 - We have had a look.

24:44 That's, so Apple's model is something that's not really reached mainstream yet.

24:49 And Apple doesn't really sell a computer in a form factor that would be amenable to deployment in a data center environment.

24:54 - Yeah, you have to put a bunch of minis on their edge or something, yeah.

24:57 - Exactly, yes.

24:59 So it's something that we've got an eye on, but we haven't got a usable kind of hardware prototype at this point.

25:04 We're currently using AMD EPIC platforms.

25:06 They have lots of PCIe lanes and lots of memory channels.

25:10 So that lets you get data on and off, you know, from your network to your GPU very quickly.

25:15 Advances such as direct memory addressing and other such things, I forget all the terminology now.

25:22 But basically, the faster that you can get it off the, into the GPU's RAM so that the GPU can do its thing, the better.

25:27 Yeah, it's an interesting trade-off to consider because the GPUs, even in the new higher end Apple stuff, is still quite a bit slower than NVIDIA.

25:36 But the memory is instant, so there's, you know, where we have trade-off, cross the line.

25:41 It's gonna be interesting as this kind of architecture evolves.

25:43 - Yeah, we've definitely got eyes on that, and we'll see how it goes.

25:47 - All right, for doing the CUDA stuff, what's the Python side look like?

25:51 Is that, what library is she using and stuff?

25:53 - So the Python, that's got to kind of do, So the GPUs are very good at crunching numbers, but they're not very good at anything else.

26:01 So there's a lot of steps in that for a GPU to be able to do all of those calculations.

26:06 You have a high-speed network, 100 gigabit, 200 gigabit, that's very fast.

26:11 And so you need software to be able to run your network stack, receive data, and that comes into system RAM.

26:19 You need to be able to DMA it into the GPU's memory.

26:21 You need to be able to know when the work is finished so that you can copy the data back, packet that up in Ethernet packets again and send it out.

26:30 And so there's a lot of things that we use in Python to enable that.

26:34 The first and most sort of obvious one would be a library called PyCuda, which allows you to wrap up Cuda functions in a nice Python handle and interact with your GPU in a way that's more amenable to Python code.

26:48 There we go, that's the one.

26:49 - Yeah, very cool.

26:50 - So the other one, the thing that you've got to think about carefully then is coordinating your activities where your GPU is executing, CUDA calls the streams, if you're working in an OpenCL kind of abstraction, it calls it command queues.

27:04 So they work almost like threads on a computer.

27:07 So you need to have some sort of way of coordinating so that when you start one calculation, you want to make sure that the data is there, that it can work on it.

27:16 And similarly, you don't want to start copying it back before the calculation is finished.

27:21 So PyCUDA makes this quite easy.

27:23 you could use, I think, Nvidia calls it events, but they're basically semaphores and markers to help you to coordinate between your different threads for sending, processing, and receiving data from the GPU.

27:34 It's similarly getting it off the network and onto the network again.

27:37 - Yeah, PyCuda looks great.

27:39 Another thing you spoke about that was pretty interesting is your use of async and await.

27:44 Do all the network stuff, which I think is, it's clearly a good choice and fits right here, but I feel like a lot of people don't consider it fast enough or good enough?

27:53 Network, but not just that, also the GPU.

27:56 You have a few functions and they're running loops.

27:58 So the way that we've structured our code, and you can, we've got, I'll share a link with you, there's documentation in Read the Docs as well, explaining this, but we have one loop that just waits for traffic to come in off the network, but bundles it up, and this is kind of one of the trick things you need to do things in batches.

28:16 If you involve the Python interpreter, every time a packet hits the NIC, then it'll end up being very slow.

28:22 So we have a lower level library written in C++ that batches up a whole chunk in the order of about 100 or 200 megabytes worth of data.

28:29 And you use that await keyword to let you know when there's a chunk of data ready.

28:34 So when there's a chunk of data ready--

28:35 - I see, you wait until enough is buffered up and then you pull it back on the resume or whatever, okay.

28:41 - Exactly, so you mark that for upload, you put an event in the stream or the command queue and let that run.

28:49 And then in the next, you're using the await, so you're waiting for that event.

28:53 And when you see that, you know that the data is uploaded to the GPU.

28:56 You trigger the processing, whichever processing task needs to happen, and put another event and then pass it on to the next.

29:02 So the third loop waits for that final event.

29:05 When it sees it, it knows that the processing is done, and you can initiate the copy of the GPU RAM back to the system.

29:12 And then you can transmit that out on the network.

29:16 So those two things working in tandem.

29:18 - Yeah, I didn't expect the GPU to have this async interface, but it does make a lot of sense.

29:23 There's a lot of parallels in a GPU.

29:25 - I don't think that it's natively in PyCuda.

29:28 I think that's a wrapper that we've done around it.

29:30 It's been a little while since I've touched that particular code, but there's no reason that you can't do that at all.

29:35 And it's a process that we find has been very, very useful.

29:37 - Okay, but your assessment overall is that async and await, pythons, async.io has been a solid choice for this whole platform.

29:44 - It's been, I've seen this approach used in other places as well, even since before Async.io has been part of core Python, using things like Flask or Tornado.

29:53 So the approach, it's a very good approach.

29:55 It's very helpful.

29:56 Debugging Async stuff when it goes wrong is a little bit more tricky, but when it works, it works very, very well.

30:01 - Yeah, that's for sure.

30:02 You get the highs and bugs, keep it in the science space.

30:05 Okay, so-- - Exactly.

30:07 - Quick audience question, I think is gonna be interesting.

30:10 Before we move on to the testing of this whole system.

30:12 So out in the audience, Slavik asks, What do you think about simple distributed radio astronomy experiments for like citizen science?

30:20 Are they possible?

30:22 That's a very good question.

30:23 Yes, they are.

30:24 It's possible to do amateur radio astronomy using a relatively affordable SDR dongles.

30:29 SDR standing for Software Defined Radio.

30:32 It's been a little while since I've looked at this, I must confess, as I've mentioned earlier since I do this professionally, I don't bother with it that much anymore.

30:38 You have a square kilometer array you're building, you don't worry too much about it.

30:42 Exactly.

30:43 I've got a big telescope at work.

30:44 I don't need a small one at home.

30:45 You can do science, it's difficult in most practical cases because most people live in cities where there's a lot of interference.

30:52 But it definitely can be done.

30:54 One of the fun projects that I've seen has been using a satellite dish like what you would connect to satellite TV, but with a little bit of electronic knowledge and you don't need a degree, you can just be on a hobbyist level to do that.

31:05 You can build a square law receiver, so it's very much just is there signal or isn't there, you're not going to be decoding any TV streams.

31:13 you can measure, for example, the temperature of the sun using this and you can measure its angular dimensions by simply pointing it and then letting the sun drift through a few times. Yeah, so there are fun projects that can be done. As to say whether it can be useful as citizen science, in other words, useful from a scientific point of view over more than just something interesting to do, I'd have to go and have a look. I'm not 100% sure. It's very difficult to get meaningful results without very, very expensive equipment.

31:40 That's why we've got such expensive facilities being built.

31:45 This portion of Talk Python to Me is brought to you by Sentry.

31:48 How would you like to remove a little stress from your life?

31:51 Do you worry that users may be encountering errors, slowdowns, or crashes with your app right now?

31:57 Would you even know it until they sent you that support email?

32:00 How much better would it be to have the error or performance details immediately sent to you, including the call stack and values of local variables and the active user recorded in the report. With Sentry, this is not only possible, it's simple. In fact, we use Sentry on all the Talk Python web properties. We've actually fixed a bug triggered by a user and had the upgrade ready to roll out as we got the support email. That was a great email to write back. Hey, we already saw your error and have already rolled out the fix. Imagine their surprise.

32:30 Surprise and delight your users. Create your Sentry account at talkpython.fm/sentry, And if you sign up with the code Talk Python, all one word, it's good for two free months of Sentry's business plan, which will give you up to 20 times as many monthly events as well as other features.

32:49 Create better software, delight your users, and support the podcast.

32:53 Visit talkpython.fm/Sentry and use the coupon code Talk Python.

33:00 One thing you did point out, I can't remember which in the two talks it was, but you did say that you can go and rent at least optical telescopes remotely, right?

33:09 Yes, most optical telescopes are operated remotely, but a lot of ones that you can, that are commercially available as well, you know, head over to telescopes.com and you can buy ones that can be electronically controlled. And those you can do interesting, is it telescopes or telescope.com? Yeah, that's the one. You can buy ones that are electronically controlled.

33:28 Those can actually be useful for citizen science. There's a lot of, for example, astronomy organizations that do regular observations of variable stars or binary stars and they rely a lot on data submitted by amateurs. While any particular observation will not necessarily be that useful, but in aggregate thousands of these observations are very helpful for doing the science that they would be doing then. That could typically be run by university faculty and over, you know, they would aggregate these observations over long periods of time from different locations to… Yeah, that sounds like a cool research project.

33:59 To draw some scientific conclusions about what's happening with these stars.

34:02 - Very neat. - It's a good question though.

34:03 Yeah, it is indeed. Thanks.

34:04 So let's talk about the testing side.

34:08 From my sort of engineering background, one of the sort of key questions that you've got to ask as an engineer is, if you have specifications for a system to build, to design, at what point do you know that it's doing the thing that it's expected to do?

34:20 So you've got a certain set of user requirements.

34:22 The user needs to be able to have this level of sensitivity and this level of bandwidth and, you know, accuracy.

34:28 Less than this much noise or something, yeah?

34:30 Exactly, in terms of hard numbers.

34:32 So you need to have some sort of way to prove that the system that you've designed and built is going to meet those specifications.

34:38 And so testing is a fundamental part of that.

34:41 Sometimes you can prove by analysis, you know, just by using maths and showing, you know, this is what the system is designed to do.

34:47 But the first prize is if you can run a test and say, "Look, we've got a controlled set of inputs, we can measure the outputs, and we can determine from this that the system is doing what we said it is doing and how well it is." So, you know, what the noise level is or what the sensitivity is, whatever the case is.

35:02 And that's something that is often a little bit neglected in scientific projects, particularly astronomy is that's the one that I've got experience with. Other fields I've seen as well, often physicists and astronomers are very, very clever and they're trained in a lot of fields.

35:16 They tend to build their own instruments and hardware and write their own software.

35:19 It has a kind of amateurish aspect to it sometimes, not universally, but this is the tendency until until it's more or less doing what the researcher wants and then he carries on. So many a software project has been written by a PhD student, he's generated some results and you know published his paper, gotten his PhD. That's very difficult to scrutinize because the code is written by someone and the logic lives in someone's head. And so I'm kind of keen on this concept of testing because I think it adds a level of rigor and transparency to the scientific process because if I've got tests in place then anyone can come along and look at how I've tested my instrument and, you know, criticize or evaluate whether it actually does what I say it does when I publish my research.

36:03 So I think the concept is really important.

36:05 I do too.

36:06 And it's in open source, one of the important signals that people use when they go and look at a package they might consider like PyCuda or Pandas or whatever is, does this thing have tests as part of its design, right?

36:22 If I'm going to depend on somebody else's library, or if I'm a maintainer and I create a library and someone's going to send me a PR, how do I know whether their changes that they're suggesting to me broke something I didn't see coming, right?

36:33 And having unit tests or some kind of test typically driven with pytest for Python is a really important, not just supporting pillar of that project, but also a signal to others from the outside.

36:45 Like this thing, people care about this.

36:47 They've verified it.

36:48 They want to keep it strong.

36:49 - And it sounds to me like this might also be really important for science in the same way.

36:55 You talked about people coming to review your work or to try to reproduce it.

36:59 And if they see the test and they can run the test, it communicates something additional rather than just, well, I thought a lot about it and here's my graph.

37:08 - Exactly, yes.

37:09 We want to do groundbreaking science and we want to be able to say that when we make observations that what we're seeing is real science.

37:16 And it's not just some sort of anomaly that we found in the telescope signal chain.

37:20 And the presence of good tests helps with that because then we have rigorous testing that we can say, look, this is the performance characteristics of our system.

37:31 You know, it gives a little bit of extra confidence in your scientific results that way.

37:35 - Indeed, so how does this look different than like testing flask or something, right?

37:39 'Cause it's not exactly the same and you maybe are leaning on it a little bit more to communicate more information, not just pass or fail.

37:47 There are a few things. In one sense, we do use unit tests in the same way that a lot of open source projects do.

37:53 Just, you know, does each individual component of my module do the thing that I've designed it to do?

37:59 But if you go a little bit beyond that, the same sort of philosophy and application can be applied to systems as a whole.

38:06 If you have a complicated system with a lot of different parts talking to each other, it can be tricky to a change that you make might not break if you look at it in isolation, but do those interfaces still work?

38:19 If you're unit testing something, often you mock out whatever's at the end.

38:22 So you pretend that you'll get a real response from a website when you do an API call.

38:29 But if you've touched the code that actually handles that API call, then your unit test is not going to pick that up.

38:34 So we've got a relatively rigorous process for what one might call integration testing.

38:40 So an entire system gets set up in a simulated environment, but an input like it might get from a real telescope and the ability to watch their output.

38:51 And so we do that on a regular basis.

38:53 We spin up an entire correlator in a system in the lab, give it a deterministic input, and so we can measure the output and determine whether or not it meets the criteria.

39:04 - Yeah, cool.

39:05 Is this like a software emulator in Docker, or is this an actual one of your one you FPGA things that's dedicated to this integration testing?

39:16 - It can be both.

39:17 It can be both.

39:18 So we've used the same approach to testing the old Scarab correlator.

39:23 One of the Scarabs can be configured to emulate a telescope so it can produce a signal similar to what a telescope might be, except not having astronomical signals in the data.

39:34 it would have a deterministic signal such as, you know, a normal frequency tone or just white noise, depending on what kind of test we want to run on the correlator itself.

39:43 You don't replay the wow signal to it?

39:47 We could, but it's faster just to generate signals. You're not going to get, with a strongly deterministic signal, in other words, if you know exactly what you're putting in, you can see what you're getting out as well. The wow signal, not as useful for determining the noise level.

40:01 Yes, exactly.

40:02 More fun, not as useful.

40:03 So one of the things you did is you changed some of the output from pytest to be more representative of talking about how you've met your requirements and stuff.

40:16 Maybe tell folks about how that works.

40:18 Yeah, that was an interesting exercise.

40:20 Serao uses a process called systems engineering, which was developed in Bell Labs, I think in the 40s.

40:26 It was quite a while ago, and it's been strongly influenced by the sort of military processes.

40:31 And it's people in a software environment would recognize it as like a waterfall pattern where you'd have a big design up front.

40:38 So you'd have a lengthy analysis of what the requirements of your system are, and, you know, allocated different performance characteristics to different components.

40:48 And once you've done with the design, you need reams and reams of documentation to prove that your design meets the specifications as well.

40:55 Graphs, tables, pictures, numbers, all of these kinds of things.

40:59 pytest and other packages, there are other ones, but pytest is the one that we've used.

41:04 It has a much more simple output.

41:06 In other words, it'll give you lots of details if something goes wrong.

41:10 It'll give you a stack trace, it'll give you debug information of what the variables were at the time that there was a problem or that the assertion didn't meet.

41:17 But if everything passes...

41:18 It's all green.

41:19 Okay, we're good to go.

41:20 You got a dot.

41:21 Yeah, you know, like, okay, the sensitivity is better than X, but how much better?

41:27 And how does that vary with frequency?

41:29 kinds of things, what we'd like to know and what we'd like to be able to present to the various stakeholders that are interested in the performance of the telescope. We have a system. Okay. You need to zoom out a little bit there. And I, for those of you who are watching on YouTube, I do apologize for the, it's difficult to put code on a screen in a way that's not going to put your audience to sleep.

41:47 Yeah. And I just grabbed this off of your talk just so we had something to like kind of be concrete about. But you have this interesting reporting aspect that runs along with your test for sure.

41:55 Yes, so pytest has a plugin called Report Log and that if you sort of squint at it and look a little bit carefully, it allows you to kind of record metadata as you're going along. So things that are happening in your test. So there's several things happening at once here and maybe if I could just take a step back, this test that you've got up is a test of linearity. So what does that mean is that if I put in a signal x and I get an output y. So if I put in a signal 2x I should get 2y as the output so that they should be directly proportional to the other. And if there's, if they're not, then there's a non-linearity of some sort in the system. No system is perfectly linear so there will be some sort of margin for error. We don't have a specific spec on this particular linearity test. It's a quick one to run because it takes only a few minutes and it can give us a sort of a first eyeball of is, are things working as we expect them to do. pytest, if for those of you who are not familiar with it, the tests are written in the same syntax as Python functions, except where the inputs of the function are either parameters or fixtures. So parameters can be if there's a parameter space over which your test will run. So in our case, for example, different numbers of frequency channels, different numbers of antennas, this one isn't parameterized, it uses only a few fixtures. The fixtures are things to ensure that you have a test that is reproducible.

43:16 So in this case, the correlator fixture gives us a correlator.

43:20 It spins up in the test cluster that we've got in our lab, and it gives-- so the typehints there gives you the hint that we're getting a remote control, so that little correlator object there gives us the ability to communicate with it and control it.

43:32 The next one is a receiver, so that's--

43:36 you know, if a correlator is running, you need to be able to receive what it's outputting so that you can compare it against whatever the spec is.

43:43 And the third one is the PDF report.

43:45 So the name is a bit of a misnomer.

43:47 Initially, I...

43:48 So that little fixture is written using module called PyLaTex.

43:52 So shout out to the authors of that one.

43:54 It's very useful.

43:55 And the best way to generate...

43:57 You can generate PDFs directly from Python, but it's very low-level and it's difficult.

44:01 So we use LaTeX as a kind of intermediate step to do the typesetting and PDF generation for that.

44:07 So with that, you can focus on actually just getting the content and then use LaTeX to arrange things and typeset and make sure that there's, you know, the lines wrap over and what have you. Ultimately, the output will be a PDF with a report of the test as it's run. So this is the configuration that we ran, these are the steps that we followed, and these are our results. Version 2 of that uses an intermediate product. So instead of outputting a PDF straight away, we just serialize the data and put it into a little JSON dictionary.

44:35 The reason for that is sometimes in LaTeX, you got to tweak your formatting or something like and it would be nice to not have to rerun the entire test just to change the font or update the heading.

44:44 And so there's actually an intermediate step and there's another step just later that takes that JSON, pauses it and generates the PDF later.

44:51 - Right, well, you can also ask questions about it, right?

44:53 You can analyze the test JSON and maybe draw averages over time or here's how much has varied as we've evolved our system, right, not just pass/fail or what did this one look like?

45:06 So JSON's pretty good.

45:07 - Exactly, when you incorporate this with other CI/CD tools, you get GitHub Actions or Jenkins or whatever the case is, you can do this repeatedly and then you have records of over time, you know, how your system has evolved versus, you know, what changes you've made and it makes it much easier to reason about and, you know, understand the performance of your system and how it changes over time.

45:27 - I think that's really neat.

45:28 I don't see that being done very often at all.

45:30 And I think it's easy to look at this little bit of code that you put up in your presentation and say, oh, okay, well, that's pytest, and now here's a fixture and so on.

45:39 But like, if you think about what those fixtures are doing, one of those fixtures maybe is controlling one of those one-U FPGA machines to set it up, right?

45:48 And then the other one is in our, there's a lot of power and stuff going on with these little, you know, here's a variable we call the correlator.

45:55 - Exactly, yeah, so you're right.

45:58 There is a lot of code hidden behind here, but it's code that you're gonna run every time.

46:02 So it doesn't matter what test you're doing, you're gonna wanna spin up a correlator.

46:05 and whether it's an FPGA or a GPU, doesn't really matter.

46:09 You want to run through the same sort of steps so that your test is repeatable and that the result corresponds to the input that you give it.

46:16 And so the pytest fixtures are great for that because once they allow you to get the boring stuff done, once you're confident that you're doing it right, then you can just keep doing it.

46:24 You use the same correlator fixture for every test.

46:27 And if there's an update, if something changes, if there's a change to the hardware that you need or the protocol that communicate with them, then you can update it just in one place and all your tests will benefit from that update straight away.

46:38 Yeah, it's a very powerful aspect of pytest.

46:41 You've taken it to quite a level here.

46:43 One of the things that I kind of wanted to, when I did the talk last year, that I wanted to put forward is that, you know, these concepts of testing, you can apply them to real-world systems as well.

46:53 So, if you can talk to it, then you can use pytest to test it.

46:57 If you can talk to it over the network, a serial cable, or, you know, USB or something like that, You can test actual hardware, make it do things, you know, so that you can, you can qualify it.

47:07 It doesn't, it's not just restricted to, you know, little toy pieces of software, you know, your flask or your tornado or abstract conceptual things that live in the cloud somewhere.

47:16 It can actually translate to real world things.

47:18 Yeah.

47:19 People say when they talk about tests and stuff, they say, well, look, it is sometimes I say it is your documentation or it is the way you verify things are working.

47:28 But when you get to the engineering and the science level, and you're trying to verify physical things and stuff, it's cool to see how far you can push it to maybe even answer that question better, right?

47:38 Like with this reporting, for example, over time.

47:41 - I think it makes it more transparent as well.

47:44 In the open source community, we like to quote Linus' law a lot, we say with enough eyeballs, all bugs are shallow.

47:49 But if we're honest with ourselves, like not everyone is gonna go and read the code.

47:52 But on the other hand, if you can see a graph of, So this is the frequency response of the system.

47:58 That makes it much easier for interested parties to actually interrogate and say, look, you know, is this good or is it lacking in some aspects?

48:05 So from that point of view, I think it just increases the transparency and allows not only yourself, but the sort of the general, the wider scientific community to have more confidence in the performance of your telescope.

48:16 - Makes it more approachable too, I think.

48:18 - Exactly.

48:19 - Yeah.

48:20 People maybe wouldn't read a unit test and what the heck are they supposed to make out of it anyway?

48:24 know, they may well look at a graph that was generated by the unit test, right?

48:28 Exactly.

48:29 Yeah.

48:30 And this is, it's much easier for me to pass as well because, you know, I get to the office in the morning and, you know, the CI run is done overnight and I can see at a glance, you know, are things working the way that they should.

48:39 Yeah.

48:40 Cool.

48:41 What has been the reaction from other scientists and engineers when you talked to them about this?

48:47 How's this perceived more broadly?

48:48 It's been relatively positive.

48:50 Scientists want to know when they can get time on the telescope and then I have to tell them well, there's a committee that evaluates scientific proposals. We can't short circuit that unfortunately.

48:58 MeerKAT has gotten a lot of attention from, well, scientists who are interested in radio astronomy, particularly fields like pulsars and another of our sort of clients is possibly the wrong word, but is the project called Breakthrough Listen. If you've heard of SETI, it's the same crowd that are trying to search for extraterrestrial intelligence. Sometimes Sometimes they're interested in these kinds of things, sometimes not.

49:23 Often it's a much shorter question of like, "Is it working and can I get some of the data?" If the answer of those two is yes, and they start looking at the data, then you know, this performance figures and test results are sometimes more interesting to them.

49:34 But generally, it's quite well received.

49:36 - It seems like a cool idea to me.

49:38 Do you have, time on telescopes is really precious and rare, hard to come by.

49:44 The more powerful the telescope, I imagine the more contention there is for that time.

49:48 Is there a bunch of simulation and software and things that people can hand out to these folks that they can work with beforehand so they become more prepared?

49:58 Yes, and well, it mostly consists, it depends on what you're interested in.

50:04 MeerKAT has an interesting architecture in that there are some of the clients that have hardware right on site that are subscribing to data and processing it in real time and Breakthrough Listen is one of them.

50:16 You know, they're interested in very, very fast transient stuff.

50:19 Other scientists are more, they're not in as much of a hurry.

50:22 They're more interested in the end result, the output of the correlator.

50:25 That has been through several stages of pre-processing.

50:28 And it depends on what kind of science that they're doing.

50:30 And so, as with most telescopes, certainly, but scientific instruments in general, we keep all of the data from observations of past.

50:38 Typically there's an embargo on that.

50:39 So it could be a period of 12 to 24 months to allow the original originating scientist of the observation enough time to do his analysis and publish his papers.

50:48 But once that is lifted, then that data is available to basically anyone who wants to use it.

50:54 So all they will need to do is contact us and we can give them access to historical data sets and they can begin to test their algorithms on actual historical observations.

51:04 Yeah, that seems good.

51:05 Excellent.

51:05 All right.

51:06 Quick question.

51:07 Follow up question here before we kind of wrap things up.

51:09 James asks, what happens to the raw data?

51:12 Talking about coming out of the correlator.

51:13 I suppose received how much of it is captured and stored versus process down and then storing the result.

51:20 Okay.

51:21 So as I mentioned earlier, each telescope generates data at about 35 gigabits per second.

51:27 That is impractical to store.

51:30 We have logic on site that will buffer up a few seconds worth of data.

51:34 And then they have algorithms that are searching through it for interesting things.

51:38 And if there's something that they notice, there's a mechanism to dump that to disk.

51:41 So just a few seconds at a time.

51:43 And that happens a few times a year.

51:45 The generally the most interesting pieces are a little bit more downstream.

51:49 So 35 gigabits per second times 64 is a large number.

51:53 So that's north of two terabits per second that the antennas themselves generate.

51:59 The output of a 64 antenna correlator is about four gigabits per second.

52:03 You could ingest that on a Mac mini if you somehow had a USB for two--

52:08 - Just somewhere there to keep plugging and unplugging drives really quickly, yeah.

52:11 to Ethernet, yeah, you'd need to do it pretty quickly, but that is practical. That we have a storage cluster. So we have one on site in the desert in South Africa, it's relatively small only about five petabytes, but that acts as a cache. And then there is a fiber link to Cape Town where we have a data center with with a much larger archive that uses an object store. I believe it's Ceph, it was also an open source thing. And I'm speaking a little bit outside of my expertise now. But that data is retained in well, so far in perpetuity, we haven't gotten to the stage where we start deleting old data yet.

52:43 So we've got all of the observations that we've ever run.

52:46 And they range in size. It depends on the observation.

52:49 But sometimes they're a few gigabytes and sometimes they're sort of many terabytes in size.

52:54 Yeah, it's a lot of data. You wouldn't store the two terabit per second, though.

52:57 That's too much.

52:59 I don't know what we would do with that.

52:59 That's one of the reasons that the correlator exists, to sort of get that down to a manageable, useful level.

53:04 Yeah, indeed.

53:05 All right. Well, anything else you want to add that we haven't talked about briefly about this testing side of things?

53:11 Have we covered it pretty well?

53:12 - The one thing that we haven't kind of talked about is we talked a little bit about earlier about how critical the performance of this is.

53:18 And one of the things that testing has really enabled is for us to optimize the performance of the system.

53:24 And in a way that's, I think, not immediately obvious.

53:26 If you have a first naive implementation of your signal processing algorithm, that's easy to read, easy to reason about, it's not likely to be the fastest possible way to process the data.

53:36 But it's good to have that.

53:38 And then once you have that, then you can write a test for it.

53:40 So you can compare a known input to a known output and see if your maths is correct.

53:46 That way, when you iterate, perhaps we can change the memory access patterns or, you know, improve the coordination between the threads.

53:53 It's very easy when you do that to make a mistake and start messing up your results.

53:56 But when you have a unit test and you start messing up results, it can catch it straight away.

54:00 So having that testing in place allows us a little bit of, you know, room to experiment to really push this to the limits of what these GPUs are capable of doing in terms of increasing bandwidth or more antennas that we can process or all these kinds of things.

54:15 And you mentioned regressions earlier, that's an excellent thing.

54:19 So you're working on another part of a system and something breaks, having these tests in place will let you know, "Oh, we've messed something up.

54:26 Let's revert back and be a little bit more careful next time." So I think it's just a great concept.

54:30 In industry, in science, in academia, everyone would benefit from having more embedded tests in place.

54:37 You're not a panacea, you're not gonna anticipate all the possible failure modes in your tests, but it definitely helps catch many of the obvious ones.

54:45 - Yeah, good advice.

54:46 Testing science is tricky.

54:48 It's usually a stream of numbers in some way, it's not, yes, there was a user, no, it came back none, or there was an exception.

54:54 It's still shooting out a bunch of numbers.

54:57 Are they good?

54:58 Maybe they're better, I don't know, right?

55:00 But having systems in place to record them, to test them, to say if it matches this curve within some tolerance, it's still good.

55:07 - Yes.

55:08 - It's really important in those areas because it's so hard to look at it and know what the deal is.

55:13 - Exactly.

55:14 And I mean, so it is a team effort.

55:15 There's not necessarily any one person that's gonna have the experience to do all of these things.

55:21 So the person who is a very experienced systems engineer who will know, you know, the methods by which we can evaluate the performance is not necessarily gonna be the best coder.

55:29 So that's why we have a large team in order to pull all of these things in.

55:32 So yes.

55:33 So shout out to my colleagues at Serao.

55:36 We're doing good work.

55:37 Or at least I like to think we are.

55:38 - It sure seems like it.

55:39 Cool, all right, well, we're about out of time to talk more about this.

55:43 So let me ask you the final two questions before you get out of here.

55:45 If you're gonna write some Python code for this crazy big system you've built, what editor do you use?

55:52 - I'm a bit old school.

55:53 I basically just use Vim.

55:54 - Vim, right on.

55:55 - I feel most comfortable in the terminal, so no fancy GUIs for me.

55:59 - Yeah, there you go.

56:00 And then notable PyPI package.

56:02 Do you want to give a shout out to?

56:03 - Well, a couple of ones that I've already mentioned.

56:06 pytest is very important.

56:07 PyCuda has also been critical in our work so far.

56:12 So yeah, shout out to those two.

56:13 Thanks to the open source community for providing that for us.

56:16 - Yeah, it's really cool how much of these general open source projects are supporting science and other types of exploration.

56:23 - Cool. - Yeah.

56:24 Okay, final call to action.

56:25 People are interested in maybe adopting some of your practices.

56:29 Maybe they want to learn more about how you did this reporting in pytest, or even as someone in the audience asks, can they get access, how do they get access to the datasets potentially?

56:40 What do you tell them?

56:41 - The datasets, well, I hope you have a very big hard drive on your laptop.

56:45 If you go to serao.ac.za, there'll be a contact page.

56:48 And if you can send us a question and we'll make sure that it gets to the right person.

56:52 I'm not sure what the procedure is for access for people outside.

56:57 I can get to it when I want to, but then I'm inside the organization.

57:00 So start with an email.

57:02 - But if you ask the question on our contact page, yeah, and then we'll send you to the right place.

57:06 To learn more, I'll put the GitHub link to the software for the correlator.

57:12 It's all there and the documentation is relatively good.

57:15 If there's something that's missing, you're welcome to just raise an issue on the GitHub repo and then we'll see how we can help.

57:21 - Excellent.

57:22 Well, thanks for sharing your story and keep up the good work.

57:24 - Michael, yeah, thanks very much.

57:25 It's been great to be here.

57:27 - Yeah, it has been great to talk to you.

57:28 See you all later.

57:29 - Cheers.

57:30 This has been another episode of Talk Python to Me.

57:33 Thank you to our sponsors.

57:35 Be sure to check out what they're offering.

57:36 It really helps support the show.

57:38 TypeEye is here to take on the challenge of rapidly transforming a bare algorithm in Python into a full-fledged decision support system for end-users.

57:47 Get started with Taipy Core and GUI for free at talkpython.fm/taipy, T-A-I-P-Y.

57:54 Take some stress out of your life.

57:55 Get notified immediately about errors and performance issues in your web or mobile applications with Sentry.

58:02 Just visit talkpython.fm/sentry and get started for free.

58:06 And be sure to use the promo code talkpython, all one word.

58:10 Want to level up your Python?

58:12 We have one of the largest catalogs of Python video courses over at Talk Python.

58:16 Our content ranges from true beginners to deeply advanced topics like memory and async.

58:21 And best of all, there's not a subscription in sight.

58:23 Check it out for yourself at training.talkpython.fm.

58:26 Be sure to subscribe to the show, open your favorite podcast app, and search for Python.

58:31 We should be right at the top.

58:32 You can also find the iTunes feed at /iTunes, the Google Play feed at /play, and the Direct RSS feed at /rss on talkpython.fm.

58:42 We're live streaming most of our recordings these days.

58:45 If you want to be part of the show and have your comments featured on the air, be sure to subscribe to our YouTube channel at talkpython.fm/youtube.

58:53 This is your host, Michael Kennedy.

58:55 Thanks so much for listening.

58:56 I really appreciate it. Now get out there and write some Python code.

58:59 [MUSIC]