Durable Python Execution with Temporal

00:00 What if your code was crash-proof? That's the value prop for a framework called Temporal.

00:05 Temporal is a durable execution platform that enables developers to build

00:09 scalable applications without sacrificing productivity or reliability.

00:14 The Temporal server executes units of application logic called workflows in a resilient manner that

00:19 automatically handles intermittent failures and retries failed operations. We have Mason Egger

00:25 from Temporal on to dive into durable execution in Python. This is Talk Python To Me, episode 515,

00:32 recorded June 19th, 2025. Welcome to Talk Python To Me, a weekly podcast on Python.

00:54 This is your host, Michael Kennedy. Follow me on Mastodon where I'm @mkennedy and follow the

00:59 podcast using @talkpython, both accounts over at fosstodon.org and keep up with the show and

01:05 listen to over nine years of episodes at talkpython.fm. If you want to be part of our live episodes,

01:11 you can find the live streams over on YouTube. Subscribe to our YouTube channel over at

01:15 talkpython.fm/youtube and get notified about upcoming shows. This episode is sponsored by

01:21 Posit Connect from the makers of Shiny. Publish, share, and deploy all of your data projects that

01:27 you're creating using Python. Streamlit, Dash, Shiny, Bokeh, FastAPI, Flask, Quarto, Reports,

01:33 Dashboards, and APIs. Posit Connect supports all of them. Try Posit Connect for free by going to

01:39 talkpython.fm/posit, P-O-S-I-T. The PyBay Conference is returning to the UCSF Mission Bay

01:46 Conference Center in San Francisco, California on October 18th, 2025. Get your ticket and pick up a

01:53 free conference course bundle from Talk Python. Get started at talkpython.fm/PyBay.

02:00 Big announcement, we have a brand new course for you all data science types out there.

02:04 Just Enough Python for Data Scientists. Data scientists get things done in notebooks,

02:09 but production quality work needs more than ad hoc scripts. Just Enough Python for Data

02:14 scientist gives you the essential Python and software engineering habits to level up your

02:20 analysis without drowning in theory. In a few focused hours, you'll tighten up your core Python,

02:25 write clean and reasonable functions, organize code into importable modules, track work with

02:30 Git and GitHub, debug confidently, and make your results reproducible with pinned environments and

02:36 Docker. You'll also see how modern agentic AI tools can accelerate data exploration, bug detection,

02:43 refactoring, and documentation. The outcome is simple. You keep your notebook speed while gaining

02:48 the reliability, collaboration, and professionalism your projects deserve. Just visit talkpython.fm

02:55 and click on courses in the nav bar. The link to the course is also in your podcast player show

02:59 notes. And if you'd rather focus purely on building with LLMs, check out Vincent Warmerdam's

03:05 LLM building blocks for Python course we just recently released as well. Now on to that interview.

03:11 Mason, welcome to Talk Python To Me. Fantastic to have you here.

03:14 It's great to be here. Long time listener.

03:16 Oh, wonderful. I'm definitely a fan of stuff I've seen you doing online as well.

03:20 And it's super cool to get together here and, you know, share it with a couple of people.

03:24 Yeah, definitely excited.

03:26 Temporal. Wow, what a cool topic. Durable execution.

03:29 What a neat idea that I've seen in other places, but I've seen less of it in Python.

03:34 So I'm real excited to dive into this.

03:35 This is something I learned about recently.

03:37 We'll go into that in a bit, but there's a lot here.

03:41 Let's just leave it like this could be a two hour show and we'd still be going easy.

03:45 Definitely.

03:46 Yeah.

03:46 I spend a lot of my time educating people about durable execution.

03:49 that's, that's what my role is.

03:51 I'm a develop, one of the developer educators at temporal.

03:53 and definitely you could spend hours on this and we would, we could be going forever.

03:58 So yeah.

03:59 Yeah.

03:59 And we definitely could.

04:00 Well, before we do go on for hours, who are you?

04:03 It was Mason.

04:04 Oh yeah.

04:04 my name is Mason Egger.

04:06 I am a developer educator at temporal, as I mentioned.

04:09 And I also help run the PyTexas conference.

04:13 So PyTexas is one of the oldest, actually the oldest regional Python conference in the

04:18 world that we know of.

04:20 No one else has come to claim that spot yet.

04:22 We started in the late fall of 2007, like right after PyCon US would have started.

04:28 We'll be experiencing our 20th year this upcoming year, which is exciting.

04:31 And I also am the president of the PyTexas Foundation.

04:34 So PyTexas has its own 501c3 organization that is used to basically act as a sheltering organization for all of the Python meetups and events that go on within the state of Texas, which is really nice.

04:46 So I was elected president of that back in 2002.

04:50 I've been helping run and build the Python community there ever since.

04:53 And it's a lot of work, but it's a lot of fun.

04:55 And I really enjoy my community work.

04:57 Yeah, it sounds extremely fun, although also challenging.

05:00 You say for the entire state of Texas, Texas is a big place.

05:05 It's like countries as well, if you look at the size of Texas.

05:09 It really is.

05:09 Texas has some unique issues with it when it comes to running community stuff.

05:17 We founded our own virtual meetup for basically anyone in Texas, and anyone else can join,

05:22 obviously, too.

05:22 We don't say, oh, no, no Texans can join it.

05:25 But we did it because the amount of feedback we would get, it's like, oh, I live in, say, the DFW area, the Dallas-Fort Worth area, and I would love to go to the meetup, but I live two hours away from the meetup because of how large the Texas cities are.

05:37 Texas cities are sprawling metroplexes.

05:40 I grew up in the Houston area, and the joke when you grew up in Houston is you're always three hours away by car from every other part of Houston.

05:48 And that really is the case.

05:49 So it does have its own set of unique challenges, trying to coordinate four major cities across,

05:55 you know, technically two time zones.

05:57 If we can, if we include El Paso, the very, very edge, the very west part of Texas is

06:03 in the mountain time zone.

06:05 So it's an interesting bit of work.

06:07 And we've been doing a really good job of expanding and like adding more and more offerings

06:11 that we can offer to the community year over year.

06:13 And it's been a lot of fun and I really enjoy it.

06:16 It keeps me busy.

06:17 That's for sure.

06:17 Yeah, I could say.

06:19 is that why people drive so fast in Texas? They've got so far to go.

06:23 Exactly. Yes. I mean, the fastest speed limit in the United States is on the highway next to my

06:28 house. It's 85 miles an hour and it's on the toll road next to my house because you're right. Like

06:32 if you're trying to get from Austin to San Antonio, that's, I think it's about a hundred miles

06:37 down. And like, if you're going 55, you'll get there in, you know, two and a half hours,

06:41 but I can make it in an hour and a half hour and 15 minutes. If you just let me fly,

06:45 I had no idea that the speed limit were that high.

06:49 I knew they were high, but not that high in Texas.

06:51 Yep.

06:52 Wild, you know.

06:53 You guys in Germany, come on.

06:55 Yeah, it is the Southern Autobahn, basically.

06:59 Fantastic.

07:00 So you've had a bit of a tougher challenge, I suppose, or more challenges than a lot of people with this whole Pi Texas thing,

07:07 which I've always thought was super cool.

07:09 I'd love to go sometime.

07:10 It's far from Oregon, but still would be really lovely.

07:13 That said, you started working on this COVID time, right?

07:17 Which for conferences is the kiss of death.

07:19 You want to talk us through how that went?

07:21 Yeah, I could definitely talk about that for a while.

07:24 So I got involved in conference organizing, as most people do, by first speaking at the

07:29 conference.

07:30 So I spoke at the conference in 2019.

07:32 I had been a PyCon attendee prior to that.

07:35 My first PyCon attendance was the last Portland one, which I believe was 2017.

07:42 and then spoke at Pitexas in 2019 and volunteered to help for the 2020 conference.

07:47 And that was really interesting because, you know, we had planned to do it in person.

07:50 I think everybody had planned, was planning in person 2020, you know, because you, for

07:55 those that don't know, when you start planning a conference, you start planning about eight,

07:59 10 months prior.

08:00 So if you're a spring conference as Pitexas is, we were planning, you know, Pitexas 2020

08:05 in about summer of 2019.

08:07 No one knew what was coming.

08:09 We were so innocent then.

08:11 Yes.

08:12 We really were. So we were planning that. And then, you know, the events that we know of happened

08:18 and we kept chasing it. We kept pushing it back. I think a lot of conferences at that time kept

08:21 pushing back thinking, Oh, we'll be out of this in two weeks. We'll be out of this in three months.

08:26 We ended up going virtual. And then in 2021, we made the conscious decision to just not have a

08:30 conference. The virtual conference for us in 2020 wasn't overly successful. And we've, at that time,

08:36 we were feeling there was just a lot of virtual conference fatigue and we didn't have a really

08:39 good story on how to like make it work and dedicate the time. Everybody was also struggling

08:48 to handle the world at that time. So being able to put more resources into it was difficult.

08:53 So we pushed that off and then we came back in 2022. We made a very conscious decision about it.

08:58 Like we were like, we're going to come back in the safest way possible. PyCon US had announced

09:01 they were coming back. We decided we were going to come back. We had, we have a venue that has

09:05 custom air filtration. We instilled mask mandates and vaccine mandates and all of that stuff.

09:10 And we had 76 people return. But we knew that if we didn't come back, if we let this kind of like

09:16 continue on, that the likelihood of getting the community to come back, that the memory

09:20 is short, that if we didn't come back, we might lose it forever. And having run this for 18 years,

09:26 we were concerned about losing it at that point. I think that's totally valid. A lot of this

09:30 conference attending is it's habit. You form, you've gone a bunch of times. You really enjoy

09:36 it. Like, yeah, of course I'm going to go. You don't even question like, well, when is it? I'm

09:38 just going to sign up when it comes out. But you know, if you stop going, then I think that does

09:43 make a big change. So it definitely does. So in 22, we, we did it and then we kept coming back.

09:48 And then, you know, every year prior after that, we, we continue to grow. it took us three

09:53 years to get back to above, what, what I would consider pre-pandemic normals. and the,

09:59 the fun fact about that is, and the thing that I think that really kind of helped us out is like,

10:03 we didn't really start seeing the growth back in the conference attendance until us as the

10:08 foundation turned our inwards local or sorry, our vision local or our site to start focusing on the

10:15 local meetups, because the local meetup scene had not returned. And that was the vast majority of

10:19 our of our of our marketing was that we would send out these emails and stuff or the local meetups

10:24 would promote us. And these are huge meetups. I mean, the Python, the some of the Python meetups

10:29 are some of the oldest meetups that I know of. I mean, like they started in the early,

10:33 in the late aughts in the early teens. Yeah, I would say they probably were not even called

10:38 meetups when they started, right? The user groups or something. They predate meetup. Yeah,

10:43 meetup.com and all that. Yeah, I think the PyHouston meetup group is called,

10:47 like their tag on meetup.com is Python-14. So I'm assuming it was the 14th Python meetup when it was

10:55 created in the world on meetup.com. so, and you know, large groups, but these meetups had

11:01 gone dormant. So what I did as, as foundation president at that time was I was like, okay,

11:05 I'm going to spend all of my effort while my organizers are helping get the conference going.

11:09 I'm going to spend all of my effort finding organizers and rebuilding these meetups in

11:12 these areas. so I would connect with people that I knew in the area. I would reach out to

11:16 friends. I would put out all calls and after time of rebuilding the ecosystem, then we basically

11:22 everything came back to life. And that's kind of where a little, you know, the things that I have

11:25 learned is that if your, if your meetup ecosystem is not healthy, then your regional Python conference

11:30 ecosystem will not be healthy because it has to feed up into it. Yeah. Yeah. I think it's,

11:36 it's such a challenge to run like a conference or podcast or whatever to draw in new people.

11:43 I mean, you can do all the amazing mailing lists and announcements and everything,

11:47 but you're speaking to your existing people that know about you and finding new ones is really

11:51 interesting and this cultivating like user groups to sort of be the foundation. It's a very

11:56 interesting idea. Definitely. User groups work. I think we also started the PyTexas meetup,

12:01 which was a virtual meetup that we do monthly that we allow anyone to join. And we cultivated

12:05 amongst our network of friends and social media and word of mouth. And then we all know what

12:10 happened with the social media situation, which really did not help. I mean, there was a lot of

12:14 just a lot of things that have happened with like the dissolution of Twitter really did not help

12:20 the conference and the tech ecosystem. Basically, we've all fractured now. We're all in like six

12:24 different places. Like the move to Mastodon is not complete. Blue Sky had its moment, but I honestly

12:30 get less engagement on Blue Sky than anything else. LinkedIn has surprisingly been our most successful

12:36 social media platform. It seems like a lot of people have moved there for some reason or another.

12:41 But basically, it means that you just have to reapply your marketing strategies. And the fun

12:46 thing that I've had the benefit of is that as my work as a developer advocate, all the roles that

12:51 I have done, they tend to sit in marketing. Developer advocacy tends to either sit in

12:55 product or marketing. All the roles that I have taken sit in marketing. And I've had the benefit

12:59 of like, whenever this has started going weird, I can ask all of my marketing friends, hey, what

13:04 would you do in this situation? How would you approach this? So I've got to learn through

13:08 osmosis kind of how marketing stuff works and being able to apply that to conference organizing

13:12 and meetup organizing has actually been substantially beneficial to us.

13:16 Yeah.

13:17 Certainly, I think companies that are turned on enough to have developer advocates have

13:22 developer intelligent marketing teams, and that's a real resource.

13:25 Definitely.

13:26 Yeah.

13:26 It's been really useful to be able to get other people's opinions.

13:29 And then, you know, just ask other conference organizers, what are they doing?

13:32 I think that, you know, finding out what works and telling other people about it.

13:36 I mean, we, I haven't had a chance to write this year's blog post yet, unfortunately,

13:40 for this year's conference.

13:41 But whenever I became conference chair and then president, I was like, we're going to be as transparent as possible.

13:47 Every misstep we make, we're going to blog about it.

13:50 We talk about our budget.

13:51 I mean, we're also a 501c3, so that's kind of like part of our bylaws.

13:54 But at the same time, it's like this is everything that we do.

13:57 You can find it on all of our websites.

13:59 And this is what worked.

14:00 This is what didn't.

14:00 Because there's so many first-time conference organizers who may want to start a conference who don't know how to achieve these goals or whatever they're trying to do.

14:08 And we have 20 years of experience doing it.

14:11 And like it, we need to help each other out and make sure that we distill this, this knowledge

14:15 outward.

14:16 I don't have 20 years of experience.

14:17 So I'm only doing it for four, but you know, institutional knowledge.

14:20 There are, there are Google docs with like years of back data that I can go and look

14:24 at and be like, oh yeah, in 2009, we ordered way too many meals because we, you know, didn't

14:30 charge enough for, we didn't charge anything for tickets.

14:32 Like PipeSex used to be a free conference.

14:34 And basically when you don't charge anything for tickets, one of the lessons that we learned

14:37 is that people will just sign up for a ticket and then not show up.

14:40 And then your catering gets all kind of out of whack.

14:42 So even charging just a little bit of money, like five bucks, I think we charge like five dollars for our network event.

14:47 It's not because I need the five dollars.

14:48 I mean, I've told I've told my we spend, I think, thirty dollars per person on the food.

14:52 It's like it's to make sure that you have a little bit of like skin in the game

14:55 to make sure that you show up so I can get an accurate headcount for the for the for the catering.

15:00 So we don't blow the budget by ten thousand dollars.

15:02 Like I'm not envious of that.

15:04 I certainly it's easy to just check a checkbox.

15:07 Yeah, I'm interested in food.

15:07 Why not?

15:08 Yeah.

15:09 If I come.

15:10 Yeah.

15:10 Exactly.

15:13 This portion of Talk Python To Me is brought to you by the folks at Posit.

15:16 Posit has made a huge investment in the Python community lately.

15:20 Known originally for RStudio, they've been building out a suite of tools and services for Team Python.

15:26 Today, I want to focus on hosting your Python-based data science workloads.

15:31 This includes dashboards, reports, plots, interactive web apps, all the way to custom Flask and Django apps.

15:38 Their service is Posit Connect.

15:40 Posit Connect makes it easy for data scientists to share work built with Python code.

15:46 If you have a streamlet app, Dash, dashboard, Plotly interactive plot, a FastAPI service, or even a Quarto report,

15:53 just give Posit Connect the code it needs to maintain the asset and Connect automatically does the rest.

15:59 Connect will manage your APIs and serve your interactive apps for you.

16:03 And if you want, you can update your reports and dashboards on a scheduled basis.

16:07 That's right.

16:08 No need to explain to the stakeholders why that dashboard or plot stopped updating last week.

16:14 You get a focus on your data science and leveraging your skill set while Connect makes you look good,

16:19 keeping your code running and private.

16:21 With Connect, you get a private URL on your Connect server, ensuring that your asset is continuously available to your shareholders.

16:29 And you can control which users have access to the asset.

16:32 Let Posit Connect handle the delivery and DevOps involved in sharing your work.

16:37 You focus on what you do best.

16:39 So if you work on a data science team, you owe it to you and your org to check out Posit Connect.

16:45 Visit talkpython.fm/connect today and get a three-month free trial to see if it's a good fit.

16:51 That's talkpython.fm/connect.

16:54 The link is in your podcast player's show notes.

16:56 Thank you to Posit for supporting Talk Python To Me.

16:59 out in the audience.

17:01 Toon Army says, yay, PyTexas Foundation.

17:03 That's pretty awesome.

17:04 Yeah.

17:05 All right.

17:05 Well, let's talk.

17:07 Emporal.

17:08 Now, I came across this a few weeks ago from this, you know, I think this,

17:12 ironically, I think this might be the blue sky that I came across it on.

17:15 Interesting.

17:17 No.

17:18 Okay.

17:18 It was on X.

17:19 There's still apparently stuff that happens on there.

17:22 I miss Twitter.

17:23 I do too.

17:23 And, you know, people like post weird comments on, like reviews like oh you know they said they don't like twitter so they must be whatever like no just that used to

17:35 be so active and it's not so active like put put aside everything else it used to be like you could

17:39 have great conversation you still can but it's so much less than it used to it's an easy it's an easy

17:44 explanation the signal to noise ratio is completely messed up now yes it used to be such a much better

17:49 place because everybody was posting there and there wasn't as much noise and now the signal to noise

17:52 ratio makes it almost unintelligible to be able to find anything of any use and that's the number

17:57 one problem with it. And you pay for attention and there's six other places like you said. So anyway,

18:01 so I found this post from, we'll work it backwards, from Pietro. It says, people aren't talking enough

18:08 about how most of OpenAI's tech stack runs on Python, which I thought was a super cool post.

18:14 It comes from the Pragmatic Engineer newsletter and it talks about how most of the product's code

18:20 is written in Python, uses FastAPI, C for certain parts. And then, so all that stuff made, I'm like,

18:26 yep, yep. Temporal, use for asynchronous workflows. I'm like, what is Temporal? And then I think you

18:32 sent me a message and said, hey, I would love to talk to you about Temporal. And I looked at him,

18:36 yeah, this is super cool. So that's how I learned about you guys, which I thought was pretty neat.

18:42 And yeah, let's talk about it. What is Temporal? Yeah. So Temporal is essentially what we call a

18:51 durable execution platform. And it's durable execution is kind of a new term or a new like,

18:57 like field within the zeitgeist, as you would call it. And we're kind of going with like what

19:03 we're calling crash proof execution. And the way that we kind of talk about it now is like,

19:08 essentially, it handles the failure state of your of your applications and ensures that your code

19:13 will continue execution regardless of the failures that it encounters. So say, for example, you have

19:20 a application that's making a couple calls out to a microservice. Okay. And that microservice goes

19:25 temporarily down. You as the developer would have to traditionally write a whole bunch of logic

19:31 around handling that failure. Okay. So we have to detect that the failure has happened.

19:35 What type of error did we receive? Now, what do we do? Do we back off and retry? Do we decide that

19:41 this failure is non-retriable and we just don't do it? Like the difference between we're not

19:44 authenticated versus a 404, those are completely different failure handling modes. And then,

19:49 So there's all of this kind of logic that you have to build in to handle failure.

19:53 Temporal basically abstracts this away from you.

19:56 So whenever you have like a function call or a method call in Python, when implemented with Temporal, you automatically get like retries, for example, by default.

20:04 So you get this like declarative policy that happens as a default policy,

20:08 and it will automatically retry your application or your call until eventually it comes back online.

20:14 Because let's be honest, in most distributed systems like this, those are most of the time intermittent failures.

20:19 Like a microservice going offline, are you being rate limited?

20:22 Those are usually fixable with a little bit of time and some retries.

20:27 Now, there are other cases where they're not, but that's like the default policy.

20:30 That's the default use case or the default for retries alone.

20:34 And then there's a lot of other cases.

20:36 So Temporal maintains the application state of your application.

20:41 And so say you have like a Python application that has 10 steps in it and you get through

20:46 step five and then the python application were to crash for some reason out of memory uh maybe

20:51 your kubernetes pod got descheduled something like this and production happens happens all the time

20:55 this is as a former sre this was this was the life that i lived for years um yeah even hey

21:02 there's a security patch for the Linux server that's the yeah running this and we got to restart

21:07 it and like uh we have to reboot yeah and now yes it's not wrong but like stuff is going on exactly

21:14 So now when you have to deal with that, you're like, okay, so do we let everything get through and risk the attack surface?

21:19 Or then you have to make the calculus.

21:21 Do we just kill all the processes and then wait for, and then restart them?

21:26 What's the cost on the restart?

21:28 And then redoing all of that logic.

21:29 Did they make any rights to the database?

21:31 Do we have to worry about cleaning all that?

21:33 There's all of this conversation.

21:35 Since Temporal maintains that application state, that becomes a lot simpler.

21:39 Because as we, I think we kind of alluded to it, but I'll state it outright.

21:42 when that when you when that application crashes, and in a typical application, you have to start

21:47 back over from the beginning. And you have to replay every you basically re execute everything

21:51 you did unless you were have some like advanced sort of like event sourcing style model where you

21:56 are keeping track of it. Temporal does this by default. So it maintains the event history and

22:01 what's called an event history of your application. And what that does is every time a basically

22:06 creates checkpoints, every time a function executes, it stores the result of it. And if that application

22:10 were to crash, it will reschedule it onto another worker. That's what they're called in temporal.

22:15 And we'll get into that in a minute within your fleet, re reconstruct the state of that application

22:20 up into that point of failure, and then continue onward as if the failure had never happened.

22:24 And more often than not, this happens without the developer even being aware of it. Like no,

22:29 no alarms will go off unless you personally do them, because this is the design of the system.

22:33 It's to ensure that in spite of failure, your application will continue executing.

22:37 Yeah, that's awesome. And there's a bunch of neat ways in which that happens, which we'll talk about.

22:42 But I think one mental model people should maybe think about when they consider these is it centralizes a lot of the error handling and the retry code and all of that stuff.

22:54 So a lot of your application becomes simpler, not more complex.

22:58 Exactly. Yes. That's actually one of the main feedback things that we get from people is, you know, all of this.

23:04 There's so many mechanisms.

23:05 There's so many actually like patterns that we have now built to handle all of these kind of things.

23:11 Event sourcing, event-driven architecture, saga pattern, all of these different like CQRS,

23:15 like all of these different distributed systems patterns that exist to handle all of these things.

23:20 And Temporal basically abstracts away a lot of them and you get them basically for free

23:24 out of the box with the project.

23:25 And it does make your code a lot simpler.

23:27 It makes it a lot more contained so you can walk through it straight down.

23:32 Basically, as like almost as like it's a workflow, you know, the core primitive of temporal is called a workflow.

23:37 We don't tend to like to refer to ourself as a workflow engine, but some people will say that.

23:41 And I typically don't correct them on it.

23:44 If that's if that's what it takes for people to understand, that's what it is.

23:46 Fine. But durable execution is the actual phrase.

23:50 It's a meaty topic. It's a it's kind of hard for people to like kind of wrap their heads around it.

23:54 So I'm like, let's get you to understanding it and then we'll correct the little the little tidbits here and there.

23:59 Yeah, excellent.

24:00 So, you know, people are probably familiar with Hennick's stamina or with tenacity, which

24:07 is interesting, where you might put a decorator onto a function and say, hey, if this fails,

24:12 instead of just crashing, try it again.

24:15 Yeah.

24:15 Maybe a few times, maybe with exponential back off.

24:18 And this is kind of a single thread of execution type of form of durability a little bit, right?

24:25 Where it's like this function now becomes somewhat more reliable based on like maybe the service comes back or maybe your network comes back or whatever it happened to be that caused this particular example.

24:36 But with temporal, it's a lot broader, right?

24:40 Like every step, it saves the state.

24:45 If something goes wrong, it can basically resume.

24:48 I guess maybe this resumable idea is like one of the big difference, right?

24:52 It's like, let's take our reboot the server example.

24:56 Let's suppose we got some service that we work on that our app depends upon.

25:00 We had to reboot it.

25:01 Now what?

25:02 It happens.

25:03 Yeah.

25:03 So, I mean, yeah, you have a service that our app depends on.

25:06 We had to reboot it.

25:07 So if the service, so we're assuming our service is on in like an external machine and it's

25:13 calling out.

25:13 Yeah, for some reason.

25:14 Yeah.

25:14 Or maybe it's a Docker container and we rebuilt the Docker container.

25:18 It takes five seconds to start.

25:19 Something like that.

25:20 Yeah.

25:20 So our service is, you know, calling out to it.

25:22 it will basically retry until that comes back online.

25:24 Now, if we were to reboot our service, say our service was running and we rebooted the container that contained it,

25:30 it would basically, if we're in a Kubernetes system, the Kubernetes scheduler would probably reschedule it

25:36 onto another node within the pod.

25:39 And then it would reconstruct it.

25:41 And what happens is there's actually, the Temporal exists as a kind of orchestrator executor model, like a service worker model.

25:50 So the service maintains the history.

25:51 So what will happen is whenever that new service comes online, or whenever the new execution comes online,

25:57 we've rebooted it, it got rescheduled, it comes online, it will stream the old history from the service

26:02 and then reconstruct it, basically going step by step through it.

26:06 And so like the function A that was executed, the input and output values are stored in that event history.

26:13 So the output value, it's like, oh, we've successfully completed execution of function A,

26:17 store that in variable foo, and then continue onward.

26:20 And then we can continue reconstructing that service up until it gets to the point where it's like, okay, we have no more events in the history about what happened.

26:27 So now we know we've, you know, we have reconstructed successfully.

26:31 Now we continue forward and execute as if nothing had happened.

26:35 Yeah.

26:35 So in your non-durable execution code, you might have try, accept, do something else.

26:42 You might have stuff like stamina or tenacity where you're like, okay, we're going to try this again.

26:48 There's a lot of that kind of code that you write.

26:51 And in this world, you could just say like top to bottom, write the happy path, which is great.

26:57 And then what happens is temporal says, okay, I tried to run this, it failed.

27:01 So we'll reschedule it with a back off or something along those lines, right?

27:06 So you can basically not deal with a lot of this and get visibility into the errors and the state of flow

27:12 by letting the orchestrator manage that, right?

27:14 Yes.

27:15 And I'm covering like the top 10% of all the different features.

27:19 There are so many interesting other features that Temporal provides as an ecosystem.

27:25 So like one of the other really neat ones is that we provide a really simple way for

27:28 things to do a human in the loop interaction.

27:31 So you can very easily send in what's called a signal.

27:34 So basically sending in data into a running execution and then basically doing some processing

27:38 on it.

27:39 So you're waiting on a confirmed signal from someone like your application is doing something

27:43 you're waiting on to confirm.

27:44 you can send that in directly into the workflow.

27:47 And then that will basically be, that's persisted again within the events,

27:50 within the event history.

27:52 So if it crashes after that confirmation, that confirmation is stored as well.

27:57 So you have that, you have the ability to do long running schedules.

28:00 So there's some cron syntax, like what are called schedules in Temporal.

28:04 There's so many different features in Temporal that are just really neat

28:09 and can solve a lot of the problems that you're trying to do.

28:12 And scaling becomes super easy as well.

28:14 So like you want to scale, just add more workers to the fleet.

28:16 That's the easiest thing you can do is just add more of these workers and they basically

28:21 can be executed across all of your different fleets.

28:23 So the scaling story is super awesome as well.

28:28 This portion of Talk Python To Me is brought to you by PyBay.

28:31 PyBay is an annual conference gathering of Pythonistas put on by the Bay Area Python

28:36 Association.

28:37 This year is returning to the UCSF Mission Bay Conference Center in San Francisco, California.

28:43 It's a one-day conference on October 18th, 2025.

28:47 I've spoken there previously and had a great time attending.

28:50 And there's a bonus.

28:52 Talk Python is sponsoring the conference.

28:54 Every attendee gets a special conference bundle of paid courses for free as a conference gift.

29:00 Plus, we'll be giving away a complete set of training materials for a dev team of some lucky attendees.

29:06 So if you want to connect with the Python people of San Francisco and go home with courses from Talk Python,

29:12 check out PyBay.

29:13 Please use our link.

29:14 It's talkpython.fm/PyBay.

29:16 The link is your podcast player show notes.

29:19 Thanks to PyBay for supporting the show.

29:21 We'll get into it.

29:21 There's definitely some examples of large scale use cases, you know, high frequency use cases of these things.

29:28 But going back to the timing, you know, what if your user onboarding looks more like,

29:33 I guess you could be real simple.

29:34 You could say in order to create a user account, I have to first have them create and enter their username and email or their email and their

29:42 password. And then I'm going to send them an email and they got to take an action based on that,

29:46 right? That would be pretty common, like a long running type of thing you could consider. But

29:51 for some of these systems, it's like, and prove who you are and upload a document.

29:57 That's a picture of your ID and somebody will look at it and go, yeah, that looks real enough.

30:01 And they check a box or, you know, those kinds of much longer onboarding things,

30:06 Something like that could be modeled with temporal pretty easily.

30:08 It sounds like.

30:09 Yeah.

30:09 So long running workflows are the other, that, that was the feature I couldn't

30:12 remember, which was timers.

30:12 And I have no idea why that left my mind.

30:14 It's one of my favorite features.

30:15 long running workflows are one of the like amazing use cases of, of temporal.

30:20 So because everything is maintained in this, in that state and basically crashing

30:24 doesn't really matter because we can just reconstruct the state, you can have

30:27 workflows or you can have your executions that can last for days, weeks, years.

30:33 This is kind of what we know, we kind of known as the entity workflow pattern.

30:37 So essentially like a user who's going through an onboarding process, you know, like you just

30:41 said, I think the identity workflow process is actually one of our exact sample applications.

30:46 So, you know, like you're right, they lay sign up and they have to upload some forms of

30:49 ID.

30:50 Someone has to check it.

30:51 It has to go through maybe a background check process and all of that.

30:54 That's a long running workflow.

30:55 That could take days.

30:56 That could take weeks, depending on what kind of background check you're getting done.

31:00 Temporal can suspend its state for however long it wants, you know, and we can guarantee that it will come back online.

31:07 So the interesting thing, and whenever you ever, if you ever see Temporal at a booth at a conference, we were at PyCon this year, for example.

31:13 Our code sample on our booth has a really interesting statement that always catches people's eye, and it's a little mini workflow, and it's sending emails.

31:21 And what it does is it says sleep for 30 days.

31:24 And nobody in their right mind would actually write a sleep for 30 days in code and expect it to actually function.

31:30 100% works exactly the way you would expect it to in Temporal.

31:33 And we can guarantee that it works because of the way that Temporal is architected.

31:37 Those timers basically exist on the servers on this on this temporal service side.

31:41 And they just get basically your workflow just gets scheduled to resume after the timer has fired.

31:46 So you can guarantee that long running workflows will complete exactly the way that you expect them to.

31:52 So, yeah, long running workflows, amazing use case for Temporal.

31:55 Yeah, it sounds incredible.

31:56 Now, I do want to dive actually in.

31:57 It's super interesting how y'all made this happen in Python.

32:00 But I do want to just maybe talk about the scale.

32:03 Like if I were to run Temporal, now you guys have a cloud.

32:07 So I guess stepping just a bit back.

32:09 This is MIT licensed.

32:11 Yes.

32:11 But you also have pricing.

32:13 Yes.

32:14 Before we go into what I was going to talk about, let's talk about that difference there, that contrast.

32:18 Yeah.

32:19 What's the story?

32:20 So Temporal is 100% open source.

32:22 MIT licensed.

32:23 And so there's the temporal service and there's the temporal SDKs.

32:26 Every single one of our temporal SDKs are and the service and everything is MIT licensed

32:32 forever and always that our founders are big, big fans of that.

32:36 The only SDK that is not is the Java SDK, which is Apache 2.

32:40 And if you know anything about open source licensing, there is basically a dependency

32:44 further up the tree that was Apache 2.

32:47 And you're not allowed to downgrade licensing if you ever make a derivative.

32:50 So that's the only reason that one is.

32:52 but every one of our other licenses is MIT licensed.

32:55 So you could run Temporal open source and be fine.

32:58 The thing that we have found is that at scale, the service itself is challenging

33:03 and requires large SRE teams to run.

33:06 Because essentially what we're doing is we're offering distributed systems as a service.

33:10 We're offering reliability as a service.

33:12 Just because we have abstracted these problems away from someone does not mean that the price

33:17 does not have to be paid by someone.

33:18 And I'm talking about the metaphorical price, not the dollar price.

33:22 someone still has to ensure that that database stays up essentially and that your things are

33:27 still getting scheduled and that the network is up and all of these. So it's super complex to do

33:33 that. And you can run it. So you can run the temporal service locally. You can run the temporal

33:37 workers locally. Everything you run is still local. The pricing model for temporal is just the temporal

33:42 service part, which is run in temporal cloud. So there's a weird misnomer around cloud, which is

33:47 like cloud always assumes that we run everything for you. Temporal cloud is different. Temporal

33:51 cloud only runs the temporal service, your workers, where your code executes is always run,

33:57 at least by for now until who knows if another product will ever come out. I don't know.

34:02 Is run by you on you in your data center on your machines. So your code, your execution run by you,

34:07 that service, that brain, the orchestrator, that's the part you could pay temporal cloud for.

34:12 Right. That's the part that does the orchestration, the part that handles the

34:15 the failure and retries and that kind of stuff, right?

34:18 Yeah. Well, Ted, it's the part that actually the failure retries is handled by the state

34:22 machines that are built in the SDK. It's the part that basically maintains the event history. It

34:26 maintains the communication mechanisms and it is the orchestrator. Yeah. So, but.

34:32 So if we use the cloud, I could like reboot my local data center machine or even my local data

34:37 center. I mean, my version of the cloud, you know, wherever digital ocean or whatever. And when it

34:43 comes back it'll sort of resume along like you guys will see that it's back and then yeah start

34:49 running work on something like that that's what i was getting at technically yes so fun fun

34:53 fun fact about it temporal the again the architecture of this is so brilliant and we could

34:58 get so touched in the weeds about the temporal service does not actually know about any of the

35:02 workers that are running it's always a call out model so your your your machines would come back

35:07 online know that they have things they need to do they would basically start listening back

35:11 Everything that happens in Temporal listens across task queues.

35:14 So they would all come back online.

35:15 They would start listening again.

35:17 And then they would see that there's still work in the task queues, which the service

35:19 maintains.

35:20 But all the service does, it goes, oh, someone requested that I do something.

35:24 And then it puts it on the task queue.

35:25 And then the workers handle it.

35:26 The true magic of Temporal lies within the state machines that are built within the SDKs.

35:31 But they cannot function without that orchestrator service.

35:34 I remember when I first started working on the courses.

35:38 like my primary role is writing courses for Temporal.

35:42 And if you go on to our Temporal Learn site, you can find a lot of our, I think it's learn.temporal.io.

35:48 You can find a lot of our courses.

35:50 I wrote the Java and the Python ones.

35:53 I remember when I was first writing them, I kept asking the SDK engineers like,

35:56 well, how does the server do this?

35:57 How does the server do this?

35:58 And one of the engineers was like, Mason, you're giving the server too much credit.

36:00 It's not that smart.

36:02 It's really the state machines within the workers that are doing all the heavy lifting

36:06 and the service just maintains it.

36:08 but with the long way around, what the cloud service does is it maintains that database.

36:12 So the history, without that event history, that is the magic piece.

36:16 The event history is the single source of truth of all things that have happened,

36:19 and that's what the service provides and maintains.

36:22 So whether you use the cloud version of that or you self-host that, that is up to you.

36:26 Now, again, if you self-host it, you are now responsible for maintaining it.

36:28 You are now responsible for upgrading it, security patches, all of that.

36:33 And there are multiple case studies.

36:34 There are multiple YouTube videos on our channel of people who have gone the self-hosted route

36:39 and have found that the cloud route is easier once they've reached a certain scale.

36:44 So yeah, it's pretty neat.

36:45 - Yeah, yeah, cool.

36:46 More and more, that's the business side of what works for open source, right?

36:51 - Yeah.

36:51 - 'Cause there's something that you wanna do, and then there's another requirement

36:55 that you're probably not good at, and you guys would be really good at, right?

36:58 Like for example, making sure that Temporal itself is reliable and fast and keeps running

37:04 and that kind of thing, right?

37:05 - Exactly, yes.

37:06 And I think it does also come down to, and this is one of the interesting lessons that

37:11 I've learned throughout my career, is that like, what is your core business?

37:14 Is your core business to run and maintain a temporal service or is your core business

37:18 to provide a service to your customers?

37:20 And whenever the price of running your own services outweighs what you could have just

37:24 paid someone else to do it for, then at that point, you have to take a look at something

37:28 and go, maybe I should just pay the provider who has the expertise, who literally all they

37:33 do all day long.

37:34 Because usually someone's SRE team doesn't doesn't full time dedicate to learning the ins and outs of a single product.

37:39 Like I was an SRE.

37:40 I managed a cloud platform for VRBO for Virgo.

37:45 I knew there was like 12 different applications in that stack.

37:48 And the way that I learned about each one was the one that went down that day.

37:51 So you're constantly like figuring out with like you're learning about it as it's on fire.

37:55 It's a terrible way to learn.

37:58 And that's not a great way to run it to to like live your life.

38:01 i found right why i moved into developer education um exactly there's fewer uh midnight calls in

38:08 developer education that is the primary reason why i do this now is i i like there is a point when

38:16 the p when pager duty goes off like one too many times and you become a developer advocate and i'm

38:20 living proof of that you just know yeah you take it off a space style you just take it out put on

38:25 some uh gangster music just exactly right exactly if you don't know the reference you need to make

38:33 sure you put watching office space yeah the movie right i've driven by the list i've driven by the

38:38 area where it was filmed it was filmed out here in austin where i live was it incredible yes

38:42 incredible okay so why did i sort of start in that path what i want to talk about is what is

38:46 so you guys have a cloud version which obviously does stuff in massive scale you talk about a fleet

38:51 of these things. And by the way, at the execution level, we're probably mostly talking Go and Rust,

38:57 just so people know, it's kind of like fast, fast innards. But Rakesh asks, like, it seems to be

39:03 resource insensitive. Is there a lightweight option? Like what is a, what is a minimal temporal

39:08 rather than a, you know, what Uber does or what, you know, ChatGPT does? Because those things,

39:14 you just can't compare them, right? It's like, you're not Microsoft, you're not Google, you're

39:18 not LinkedIn, you're not Netflix. Don't try to be them most. Yeah, it depends on what you're

39:24 trying to accomplish. So I mean, like, I think when it comes to so there's, there's always there's

39:27 the service and then there's the worker fleet. So let's talk. I'll talk about the service first,

39:30 and then we'll talk about the workers. So the service comes in a handful of different flavored

39:34 offerings. There's a local development binary that is meant to be used for development. And

39:40 that's what I use on my local laptop all the time. It's also more than suitable for home lab stuff.

39:46 So if you're wanting to play with this in your home lab, use the use that single binary.

39:50 It can it's got an in memory data store, but you can have it persist to it to a SQLite database

39:55 will get you very, very far on home lab stuff. Non prod use cases. Totally fine.

40:00 Speaker 1: SQLite is underappreciated. People see it as like a toy thing or like I can use this while

40:06 I'm developing like a sample, but then I don't really like you can put millions of records in

40:10 SQLite. You can go a very long way. Speaker 1: Oh yeah. Yeah, it is. Yeah,

40:14 it is an amazing tool it's one of my favorite tools um and then there so in reality what the

40:19 way that the temporal service is actually built is it's actually just a single binary um but the

40:23 thing is there are the temporal service is a conglomerate when i say the service i mean like

40:27 all of the things together it's a conglomerate of multiple microservices that when you put them

40:31 together they they connect with they interact with each other and there's like four back-end services

40:36 no three services and a front-end service then there's like a web ui and that's the service but

40:40 but then you also have to build in a data storage.

40:44 So for when you get to that point, that's when you either need MySQL, Postgres, or Cassandra.

40:48 You could probably get away with SQLite on that one, but I've never actually tried it.

40:52 But we recommend MySQL, Postgres, or Cassandra.

40:55 And then you can add in other things like Elastic for visibility.

40:59 You can add Prometheus and Grafana for being able to see.

41:02 But this is when you're starting to scale up.

41:03 So if you were doing this on small, itty-bitty scale, you could probably deploy it onto, say,

41:08 a DigitalOcean droplet and be fine with a single binary.

41:11 We have tutorials on our learn site on exactly how to do this.

41:14 And then it scales.

41:15 So like there's a Docker Compose file.

41:17 So like once you want to see what all these microservices are doing, like in multiple areas,

41:21 like you can see how we've deployed all of them and you can play it.

41:24 There's multiple different Docker Compose files to be like, oh, I want it with this database

41:28 and these options.

41:29 And that allows you to kind of like tune it and tweak it to your liking.

41:32 And then once you get to prod scale, we have Helm charts that we support

41:36 and you can deploy it directly into Kubernetes.

41:38 Now, if you're self-hosting in production, Kubernetes tends to be what we see in the wild

41:42 as the most popular deployment.

41:45 Now, again, these are single binaries.

41:46 You can deploy it however you want.

41:48 So it has a, from development to prod, it has a very nice scaling story.

41:53 Or the other option is just go to Temporal Cloud.

41:55 The thing is Temporal Cloud has a minimum pricing structure for it, which we're constantly updating.

42:02 And that's really useful once you get to actual production use cases and you have people paying you

42:08 use that money to pay for the temporal cloud.

42:10 It's not a lot.

42:10 The temporal cloud pricing is super, I think it's really super interestingly well done

42:16 because it's based on what we call actions.

42:18 So like anything that basically causes a right to that durable database,

42:21 you get billed on like fractions of a penny.

42:24 So it's consumption based.

42:26 The main thing is making sure you have enough traffic to cover, basically there's a minimum support cost

42:31 that you have to pay for like a minimum bill requirement and then you get billed on the actions

42:35 and making sure you have enough traffic and workload to handle that.

42:38 I know, and then you get a lot of other cool features in cloud that, I have to make sure I say this carefully,

42:44 that you don't get on the open source version, but those are all deployment options.

42:49 So when it comes to feature to feature option, as right now, it's a one-to-one comparison.

42:54 What runs in open source is what is deployed in cloud.

42:58 Like, if I built all of my stuff on the open source stuff, I deployed to cloud on occasion to test it,

43:03 but I'm constantly building on the open source version And all of my stuff runs in cloud.

43:07 What you get when you go to cloud is you get those production features.

43:10 You get single sign-on.

43:11 You get role-based access control.

43:13 You get more advanced metrics and things of that nature.

43:17 You get all of these things that really matter at a large-scale production,

43:20 things that large enterprises care about.

43:23 And there are ways you can get those in the open source as well, but they come as first-class citizens in the cloud product.

43:31 Maybe you got to run your own single sign-on identity server or something like that yeah yeah um so over here if i go to the temporal github repo there's docker

43:44 which has got a docker compose think something yeah there's actually a docker repo repo yeah

43:51 is there okay so i could even i think it would be like temporal and then doc like in the search i

43:54 would just search or in the org i would search for docker compose yeah it might be there somewhere

43:59 uh there we go are you right yeah i'll put that in the links as well i'm a huge fan of docker

44:04 compose. I think it opens up a bunch of great options. And if you go in here, you can see that

44:08 there's like the doctor compose my sequel to Docker compose postgres, and then just a bare bones one,

44:15 right? So if you wanted to self host it, is this a pretty good way to go?

44:19 Yeah, I think it's a great place to start. I think it always depends on your scale.

44:22 Like if you if you are good at Docker compose, and you know, you think like, the thing with Docker

44:28 compose is like, and I believe this is all going to deploy it on a single service. So is like our

44:31 single server. So like we always talk about like, you know, depends on the level of reliability you

44:36 need of the service because if the database of the service goes away, then your reliability goes away.

44:41 Like, you know, but that's kind of like the truth here is with that database being the single source

44:45 of truth, if that database magically disappears, you don't have it. Now you could, you know, what

44:50 you could do like an RDS or like a digital ocean managed database and connect that in this.

44:55 Yeah, run just basically set a room, an off server database connection.

44:59 string. Yeah, exactly. Right. Of any form that's then, then it's matching the reliability of that

45:04 database. Yeah, exactly. Yeah. Yeah. So another thing, whenever I hear durable, and actually I see

45:09 a bit about this in the show notes as well, and you say retry, like if something goes wrong,

45:13 which we tried and it'll probably resume. Like sometimes it won't. Sometimes there's a poison,

45:18 what's called a poison message. Like it's cursed to always fail for whatever reason. Like it's

45:23 trying to get to a service. The domain is gone or it's not coming back. Right. Yep. One of those

45:28 sorts of things. Three months ago, it was there. Then you said sleep for three months and it woke

45:32 up and said, wait, it's like Planet of the Apes. You're like, yeah, what have they done? You wake

45:37 up, you're like, the world is not what I thought, right? How do you deal with that kind of stuff?

45:42 Yeah. So in the retry policy, so the default retry policy, what is, is it, its default is just retry

45:49 on a specific time limit forever until it's either canceled or it succeeds. Now, if you expect that

45:55 something could happen like that. Essentially what you would do is there's something what's

45:59 known or non-retriable error types. So certain HTTP error codes, you know, might, you might would be

46:05 like, Hey, you know, this is just not going to come back. 500 is fine, but maybe 404 is like

46:11 404 might never come back, but 500 very likely could or cannot connect. I don't know. Is that

46:17 404? I don't think so. I think it does. I don't remember. Yeah. I remember, but there's probably

46:20 a code that like i i can't connect to the server versus like i got there and it said not here yeah

46:26 so there's non-retriable error codes and i mean like what we do with the core temporal primitives

46:30 is more of like um i often tell people it's like yeah like things that can retry let them retry but

46:34 like say i do a divide by zero error no amount of retries is going to change the laws of physics

46:40 um you know it's like holding your breath until something changes you're just going to pass out

46:44 um same thing with these retries so you have to have what are called non-retriable errors and you

46:47 essentially say, hey, whenever you experience this error, you should just fail. And basically,

46:51 you would bubble it up and then you have to let the next layer of execution handle it.

46:55 So yeah, totally a good story for that. Okay. Yeah. But you kind of got to think a little bit

46:59 about it, right? Just yes. Yes. You still have to think it doesn't take away all the thinking for

47:03 when it comes to like, what potentially could go wrong, but at least like with all of like the weird,

47:08 like, you know, okay, this service might go down DNS, someone messed with DNS. It's always DNS.

47:13 I had I had an it's always DNS moment the other day. And I'm like, I need a sticker for my laptop

47:17 says this because it got me again.

47:19 I can't remember what took me out.

47:21 I suffered something with that as well.

47:23 And it wasn't even directly me, but it was something that I was subjected to.

47:27 Yeah.

47:27 It is always DNS.

47:28 It's always DNS.

47:29 Except for when it's the database.

47:31 Yeah.

47:32 Or except for when you accidentally deploy the walrus operator on Python 3.7 on the server.

47:37 That also is not good.

47:38 And it's not doing, which I took down Talk Python for 20 minutes.

47:42 I'm like, what?

47:43 Why won't it run?

47:43 It was running perfect.

47:45 Oh, this was like, you know, years ago when those things were like new, right?

47:48 Yeah.

47:49 Oh my goodness.

47:50 So one thing I want to talk about here, let me get back to, that's the right spot.

47:54 Let's talk about the programming execution model.

47:58 Let's see, is there a quick start?

48:01 Let's maybe, let's talk through the quick start.

48:02 And I think there's just some super interesting modern Python ideas that you got here, right?

48:09 So maybe talk us through like, how do we get started bringing this in?

48:13 Does it have to be a from scratch?

48:15 Or can I take some part of my application, like a particular API endpoint and go, this

48:19 thing needs, needs some help.

48:21 And Temporal can help it.

48:22 Let's just plug it in on this one bit.

48:24 That's actually how we recommend a lot of people get started with Temporal is like, we

48:28 don't, we never tell people like do like a whole ground up rewrite of like everything

48:31 you have.

48:32 We, I've often told people find something that, like find, find a service that annoys

48:38 you, that pages you because it's constantly going down because it's unreliable and maybe

48:43 do a like it's a small service and do a small rewrite of that in temporal and just see how your

48:47 life makes uh makes a difference so the way that you do it with temporal is like you have to use

48:51 the temporal sdks so the way that you build temporal applications is you build them using sdks

48:56 um typically with other workflow-esque engines or other durable execution engines um some of the

49:01 more modern durable execution engines have kind of followed suit with us but some of the earlier ones

49:05 didn't we're code-based um we're not dag based we're not yaml based we don't have our own structured

49:11 DSL, we are 100% code based. And there's a lot of advantages to that. So not XML based. No,

49:18 no XML. So yeah, so we build you build basically what's called a workflow. Workflow is you can kind

49:26 of think of a workflow is like your application. It's it's the it's the blueprint of your

49:29 entire application. And what we'd say about workflows is that they have to be deterministic,

49:34 like the code within a workflow must be deterministic, and you execute it line by line going

49:38 down. And then anything within your workflow that potentially could be non-deterministic or

49:43 could potentially fail. So calling out to a microservice is both of those. It's non-deterministic

49:49 because you don't know what you're going to get back. And it's potentially going to fail because

49:52 the service could be down. That goes in what's called an activity. An activity is that thing that

49:56 automatically gets the retries. Now you implement activities as functions or as methods in Python.

50:02 You can actually do them as both. And as you were saying in the readme, yeah, we use a lot of

50:08 Python tooling. So that's actually something that I think that our SDK engineers, we have an entire

50:13 team whose entire job is to build and maintain our SDKs. They're super proud of. It's one of the

50:17 things I love talking about. Temporal is not built off of what you would say like an open API spec

50:23 for a good reason. So it's built basically, I mean, Temporal itself is built in Go. So as you

50:28 can imagine, everything's built in protobufs. There is a spec on what we build off of, but open API

50:33 specs generate stuck stub client libraries. And like I've worked with a handful of them. They don't,

50:38 they're not very idiomatic to the language. Like it kind of looks like someone bolted a C library

50:42 on top of Python. Like it works, but it doesn't feel like Python. Our SDK engineers spend months

50:49 studying the programming language, learning out what is idiomatic of the language, what actually

50:53 makes sense. And then they build into it. So the interesting thing about this is like, you can see

50:58 here when we define our decorate, our workflows, we define them by decorating classes. And then we

51:03 the entry point by decorating it with an at workflow.run. So that's how we know where the

51:07 entry point is. We're using asyncio inside of this to do it. Our activities, when you want to

51:13 turn a method into an activity, you are a function in an activity, you just decorate it at activity.defin.

51:19 And now you have an activity. We've done a lot of those kinds of things. We're using context

51:25 managers very, very a lot or, you know, rigorously within the code base. The really interesting thing

51:32 is, and I can only talk a teensy bit about this because it's super complex, but there's a really

51:36 great talk track from it at the PyTexas conference from this year. We built a custom async event loop

51:41 for this. This runs in a durable asynchronous event loop. And the SDK engineer who built it

51:46 gave a talk at PyTexas this year, and it's on the PyTexas 2025 website. And I can provide a link to

51:51 that later. And it's really neat because essentially we had to build a custom event loop to handle

51:57 all of the way that we expect Temporal to work.

52:01 And I don't think that building a custom event loop is a very common thing for people to do.

52:05 So there's a lot of lessons learned there.

52:07 I think there should be more of it though.

52:08 I think it's a really interesting story.

52:11 Yeah, I'll put this in the show notes.

52:13 Yeah, that's Chad's talk.

52:14 It's amazing.

52:15 So like he talks about all the things that he had to do when he was building out a custom event loop

52:22 for Temporal's AsyncIO event loop.

52:24 Yeah, so for people for whom this is super new, this idea. You need places in your code to execute to say, okay, we can stop, maybe save some state.

52:33 And then you talked about like replaying behaviors and so on. So you can write code like await sleep

52:39 a day, you know, asyncio dot sleep one day. And that goes into this, as you said, durable asyncio

52:46 execution. So instead of just saying, well, we're going to let the thread do other stuff. It's like,

52:51 no, save it and then resume it. Right. That's pretty wild. It's pretty great actually. Yeah.

52:56 And I mean, those, those, and that's one of the great things about the workflows and like

53:00 that, those you'd have to definitely write as workflows, but yeah.

53:02 And it takes up no, no energy or it doesn't jam up a CPU thread by sleeping.

53:06 Like it's usually if you sleep something, that thread is kind of sitting there and it's

53:09 stuck.

53:09 this a hundred percent because it's all event sourced under the hood, essentially.

53:14 the event, basically the timer started event gets recorded into the service and then

53:19 it gets descheduled.

53:20 And then that worker, that, that executor can continue performing other tasks.

53:23 And then once that timer fires, the next task for it to continue execution gets put on the task queue.

53:28 The worker consumes it, knows, oh, I need to resume.

53:31 And it resumes.

53:31 And that can happen whenever.

53:33 That can happen, you know, a day from now, three days from now, three months from now.

53:36 It doesn't matter.

53:36 Eventually it gets put on the queue and it gets executed as if nothing happened.

53:39 Yeah, that's wild.

53:40 So interesting question out of the audience.

53:42 If people went with your cloud option, where does that run?

53:45 Is that AWS?

53:46 Is that DigitalOcean?

53:48 Currently it's in AWS and GCP.

53:51 Okay.

53:51 So those are the two clouds.

53:52 Pick different availability zones.

53:54 Like if I were in Virginia, US East one or whatever, I could say I want to use that one.

53:59 Exactly.

53:59 There's different availability zones and we even have multi-region failover.

54:03 So if you needed multi-region availability for super, super high availability,

54:07 which we do have definitely have large customers who need that level of availability.

54:11 And you can totally do that.

54:13 Cool.

54:13 And that doesn't mean you got to be in AWS.

54:15 No, no.

54:16 We have people who are running on their own private, they were running on their own infrastructure

54:20 on their on-prem and they're calling into it.

54:22 So that's the niftiest thing about temporal cloud is like the security model is super simple because temporal cloud or the temporal service.

54:30 It doesn't have to be temporal cloud.

54:31 It's the service, whether you self-host it or whether it's on the cloud.

54:34 It never connects to you.

54:36 The workers always do an outbound connection into the temporal service.

54:39 So if you're using temporal cloud, the only firewall rule you have to allow is a single outbound connection off of a single port.

54:45 That's it.

54:46 So it's really awesome.

54:47 Yeah, I'm impressed with a lot of this.

54:49 So let's sort of kind of get into that.

54:51 And I want to maybe close out with two topics, testing and what's next, or like sort of also maybe multi-language stuff

54:59 we should touch on as well.

55:00 But let's talk testing first.

55:02 So there's a couple of interesting challenges here.

55:05 Like first, if I write code like this and I want to have a unit test, no longer am I like, well, this is complicated.

55:11 I just have to use pytestAsyncIO.

55:12 It's like more than that, right?

55:14 So what's the testing story?

55:16 Yeah, so the cool thing about the testing story with this is it is technically still pytestAsyncIO.

55:21 because we're code-based, this is one of the things that I always have to harp, to, to remind people on my courses and everyone's always like, oh, that's so cool.

55:29 you, because we're code, you have to give up none of your tooling.

55:33 You don't have to get like, how do you like to package it?

55:35 Are you a poetry person?

55:36 Are you a B person?

55:36 Are you a Pippin person?

55:38 Whatever you want to use, use it for your testing.

55:40 Do you want to use a Po the poet?

55:42 Are you a pie test person?

55:44 Like, what are you using?

55:45 Use it.

55:45 It doesn't matter.

55:46 Temporal does provide, obviously because these are more complex workflows and stuff.

55:50 they require a little bit. Every single temporal SDK does provide a testing framework that is built

55:56 into it. And these integrate natively with all of your testing frameworks. So you can use, you use

56:02 pytest. I use pytest. Now you could use something else, but all of our courses, I think the temporal

56:07 102 course, yeah, I wrote that one, has a whole chapter on testing temporal workflows and activities

56:13 and mocking activities because you have to be able to call them independently because you,

56:16 Otherwise that's an integration test.

56:17 It's not a unit test.

56:18 It is important.

56:19 I always feel bad about mocking the code.

56:20 It doesn't seem nice, but you got to.

56:22 Yeah, it's super easy.

56:23 It's so nice.

56:25 The you you basically just read you basically redefine the activity and I just put it in the mocking story in Python is so nice.

56:32 But you just use our testing framework.

56:34 And what that does is it basically the testing framework will automatically

56:36 spin up a temporal service and a worker for you.

56:39 So because as you can imagine, like the execution part of this is like, you have to have a service running, you have to have a worker

56:44 and then you have to send like basically use a client to send a request to execute the workflow and it will be executed. The temporal testing service

56:51 does all of that for you. And it also has basically time skipping. So what you can do is, you know,

56:56 if I have to test, if I have to test something that sleeps for 30 days, I would actually prefer

56:59 it to not actually sleep for 30 days. So I can test it. Well, I see I take so long. Yeah. So

57:05 time skipping is in there and there's a lot of other really neat features in the testing. So

57:08 every single temporal Python or sorry, sorry, every single temporal SDK has a testing framework

57:14 built into it that really well enables testing.

57:17 And it all works natively with the tools that you are already used to using as a developer

57:22 of whatever language you're already using.

57:23 Yeah, that sounds great.

57:24 And you can say things like run this and wait for a week and then resume.

57:28 And you can just say, and now a week has resumed.

57:30 You know, a week has passed.

57:31 Now I see what's happened, right?

57:33 Like it'll just zip right ahead.

57:35 Yep.

57:35 Skips right ahead.

57:36 Doesn't even, doesn't even worry about it.

57:37 That's always a fun testing trick.

57:39 So the other one I want to talk about is when I'm over here, into the GitHub repo, I can scroll down.

57:45 If I look at the repositories, it's Java SDK, Python SDK, Go SDK,.NET SDK, Ruby SDK, et cetera, et cetera.

57:52 There's a bunch of interesting things here.

57:55 Like I can write part of my app in Java, part of the workflow in Java or the workflow items in Java,

58:01 the queues, and I can do part of it in Python or.NET.

58:04 And the other part I think is interesting is Python,.NET, Ruby, et cetera,

58:08 all share a common Rust base.

58:10 And so it's kind of like they all go in lockstep.

58:13 Yeah. Yeah. Yeah. So yeah, two great topics there. So the first one I'll start off with is the is the

58:19 polyglot stuff, because I think it's one of my favorite things. And we don't I don't I don't ever

58:23 get to talk about it enough. So I'm glad you asked. Underneath the hood, the way that all of this

58:27 communication happens is it's happening via essentially protobuffs across task cubes to the

58:32 temporal service back and forth. One of the things that you'll find if you dig into temporal is that

58:36 we require your input, the inputs and outputs of your functions to be serializable to basically

58:43 protobufs. Now, if you have something that's not serializable, we obviously, because it's code,

58:47 we provide you the way to extend the serializer. So as long as you can serialize it, again,

58:54 it's code, you can do whatever you want with it. But because of that, because everything speaks

58:58 protobuf, all of these languages can natively speak to each other. So you're right, I can write

59:03 workflows written in Python and call and have three different activities in that workflow,

59:08 one written in TypeScript, one written in Java and one written in.NET. And I can call them

59:12 seamlessly, like basically just by calling execute activity, giving it the name of the function.

59:17 And then I could actually still then pass in a data class of the data that I have as that

59:21 parameter, because it's still getting serialized down into a protobuf. It will get deserialized

59:25 into the other language, execute it and pass back data that I can resume. And then I could call it

59:30 technically from a client that's written in Go. So this enables Polyglot across all of these

59:37 languages. And it's amazing. So if you have legacy systems or you have stuff where like you really

59:41 need something to be written in a whole bunch of different languages, it just gives you this out

59:46 of the box for free. And I think it's one of the neatest features. And one of the other part that

59:50 it really does need about this is it's not just like the fact that it can call it, but it also

59:56 preserves the stack traces as well. So one of the other courses that I developed are Crafting the

01:00:00 handling strategy course. There's a demo in there where I am showing like basically that exact

01:00:07 workflow, a Go client that's calling a Java workflow that's calling a Python activity.

01:00:12 So three different languages. And then I intentionally throw an error in the Python

01:00:15 activity and I tell it, do not handle it, do not retry it, let it bubble up. And when I get back to

01:00:21 the Go client, I can see the different stack traces in the different languages all the way through.

01:00:25 So I get a go basically panic that contains a Java stack trace that contains a, that contains a Python stack trace.

01:00:32 And I can, I can, I can see all of this across the line.

01:00:34 So not, and, and also the thing, and just to do it for fun, because I like showing off, I have all of these workers running on different machines.

01:00:42 So I am, I am running on different, I am crossing process boundaries.

01:00:46 I'm crossing literally across the network IP boundaries, and then I'm crossing language boundaries and it happens seamlessly and you'd never know that it happened.

01:00:53 So the orchestration layer is the wildest thing ever.

01:00:57 And then you asked about the Rust SDK.

01:00:59 That's a fun one.

01:01:00 So that kind of goes back into the history a little bit of how Temporal was built.

01:01:04 And for a crash course in this within two minutes or less, essentially our founders started at AWS together and built out what would become,

01:01:15 they built the foundations for SQS and what would become simple workflow service.

01:01:19 Then one of the founders left, went to Azure and helped build the Azure Durable Task Framework at Microsoft.

01:01:24 They met up back together at Uber and built Uber's Cadence.

01:01:28 Cadence was then basically like battle tested for four years, open sourced, and then they got permission to fork it and build Temporal.

01:01:34 So Temporal, the company is six years old, but it was a four year old open source project prior.

01:01:39 So it's a 10 year old open source project, essentially.

01:01:41 But because of that, what happened at Cadence was I think they wrote the Go and the Java SDKs there.

01:01:46 So those are very uniquely themselves because they were written independently.

01:01:49 And then the PHP SDK is its own story.

01:01:52 Someone wrote that in the community because they really wanted it.

01:01:55 And it kind of follows its own rules.

01:01:57 But when they started building the temporal SDKs, TypeScript was the first one and then Python, if I remember correctly.

01:02:03 They wanted a common core, like basically re-implementing this.

01:02:06 Because in these SDKs, there are very complex state machines that maintain all of this state of what's going on.

01:02:13 And they did not want to keep re-implementing this every single time.

01:02:16 So they built a Rust core SDK, or it's not even an SDK. It's not an SDK. It's just the core. And

01:02:22 all of the, so the TypeScript, the.NET, the Python, and the Ruby SDKs all have a upper level SDK

01:02:28 that wraps this core Rust SDK and call into it. So they share a common theme. So there definitely

01:02:33 will sometimes be features or like things that happen in the Go or the Java SDK that you're like,

01:02:37 that's a little different because those are not based on Rust core. But yeah, that's how they all

01:02:42 call in. So like their PIO3 is basically being used here. We're calling into pot with PIO3 into

01:02:47 a rust binding. and that's like a Pydantic and others. Yeah, exactly. Yeah. So, and it's,

01:02:53 it's really cool. And that, that makes, adding new SDKs a lot easier because really,

01:02:57 and truly the hardest part of building the SDKs is, was like those state machines used to take

01:03:01 a long time. And once they got it figured out on the rust core side, it made adding new languages

01:03:05 easier. the Ruby SDK is in public preview and will be going generally available here soon. Um,

01:03:11 And there may be one or two more SDKs coming out within the future.

01:03:14 If you guessed really hard, you could figure out what it is.

01:03:17 There's an SDK.

01:03:17 There's a core that doesn't have an SDK.

01:03:22 There's no secret.

01:03:23 There's no secret about that.

01:03:25 Yeah, of course.

01:03:25 People have been begging for that for years and it's obvious.

01:03:28 So yeah, may involve crates.

01:03:30 Okay.

01:03:30 So what's, what is the, what's the future?

01:03:33 Like anything that's worth giving a shout out that's coming or that kind of stuff?

01:03:38 Yeah.

01:03:39 I mean, I think that like a lot of times people often ask me like, what is Temporal used for?

01:03:44 And I would say Temporal is used for anything that like you don't want your code to fail.

01:03:48 It like it's it's really interesting to help educate people and work on a product that really does affect nearly every single part of the application of the software development lifecycle and every single part of it, of, of, of the industry.

01:04:03 You know, I was used to working on other products that like, yeah, like I worked at a synthetic data company for a little bit.

01:04:08 And that had a very niche area.

01:04:09 And then I worked at DigitalOcean, which was cloud products, which is still awesome,

01:04:12 but like doesn't affect everything.

01:04:14 Temporal really does like, are you doing finance?

01:04:17 Temporal is great for long running transactions.

01:04:20 Are you doing food delivery?

01:04:21 Like, are you a fast food industry?

01:04:23 Are you doing groceries?

01:04:24 Are you doing, what are you doing?

01:04:26 Temporal can benefit from it.

01:04:27 So there's a lot of really cool things.

01:04:29 You can use it for anything.

01:04:30 And what we're seeing right now, specifically, and this kind of alludes back

01:04:33 to your open AI thing earlier, is that we're seeing a lot of AI companies

01:04:38 of value out of this because like when it becomes time to take your agents to production, there's

01:04:44 a handful of decent production stories out there, but it turns out these are AI agents in production.

01:04:50 This is microservices in production with a fancy label on top of it. These are just distributed

01:04:54 systems. Not only are they microservices, they're very slow, long running microservices,

01:05:00 which make it harder. Yeah. Yeah. That's exactly what Temporal's model is. Do you have a slow

01:05:05 running microservice that can sometimes fail. Great. We have a product that makes all those

01:05:09 problems go away. So, you know, like I'll, I'm working on a lot of content right now around

01:05:14 showing the benefit of temporal in AI. and we have, we have a handful of customers, who I

01:05:19 can't talk about, that are, using us for lots of different AI related things. but there,

01:05:25 I mean, you can look on our blog or anything. You can see tons of people that are using it.

01:05:28 and it's a really cool thing. So I would definitely say like, if you're, if you're trying

01:05:31 to take AI to production, you should be looking into temporal. it's not an AI tool, you know,

01:05:36 like we're not, we're not going to like, we're not going to do the thing that every company did is

01:05:39 we're not going to suddenly pivot and become an AI tool. because we're not, we just, yeah,

01:05:45 we solve everything. And AI is one of the great things we solve. So that's awesome. Yeah. You're

01:05:50 not going to vibe code with temporal. Maybe you vibe code, temporal code, but not with temporal.

01:05:55 No, I've, I've, I've actually vibe coded a handful of temporal things. And it's interesting because

01:05:59 like i'm super picky uh about what what people's temporal code looks like as because i've been

01:06:04 teaching people best practices for three years almost three years now and i i'm like no vibe

01:06:09 coding that's no claude that's wrong like no no no cursor that's wrong like you can't do that so

01:06:14 and the interesting thing about that is the way that i'm looking at it's like oh i need to make

01:06:18 more content about this because yes because the llms are not like it's actually funny every now

01:06:23 and then the llms spit out some of my content um and i can tell when it's my content because i know

01:06:27 So I write comments in a very particular way.

01:06:30 And I'm like, oh, okay.

01:06:32 So what that ends up telling me is, oh, I need to make more content around this

01:06:36 because we're still not vibe coding at 100% capacity.

01:06:39 Yeah.

01:06:40 Yeah.

01:06:40 Yeah.

01:06:41 That's a whole discussion we could go down.

01:06:44 You know, I was totally wrong when we started.

01:06:46 I said we could talk for two hours.

01:06:47 I think it's three to four.

01:06:49 Yeah.

01:06:49 We got so much more we could talk about, but there's only so much time we can dedicate to each episode.

01:06:55 So let's go ahead and call it.

01:06:57 I say, you know, thanks for being here.

01:06:59 The more I looked into this, this is a super interesting product.

01:07:02 And there's a lot of neat Python integrations like you program it with async and await

01:07:08 rather than some funky SDK bolt-on thing.

01:07:12 So people should definitely check it out.

01:07:13 Final call to action.

01:07:14 They're interested.

01:07:15 What do you tell them?

01:07:16 Check out, just check out the website.

01:07:18 Check out temporal.io or the learn site, learn.temporal.io.

01:07:21 It's a great place to get started.

01:07:23 You can install Temporal by just running brew install temporal on your Mac,

01:07:26 or there's commands for Windows and Linux as well.

01:07:29 Curl commands for that.

01:07:31 Or Docker compose up.

01:07:32 Or Docker compose up.

01:07:33 If you want to do that, totally can do that.

01:07:35 Just try it out, build a workflow.

01:07:37 And then what I tell people is try to break it.

01:07:39 Like start a workflow, kill the worker, bring it back online.

01:07:42 Like I think it's really magical when you first actually try to like actually break the software and stuff.

01:07:49 We've had multiple people that have taken jobs here who have said, I started playing with it.

01:07:53 I tried to break it.

01:07:54 And when I couldn't, I decided to apply for a job here.

01:07:57 So try to break it.

01:07:59 See what you can do.

01:08:00 And you'll be amazed by it.

01:08:02 Just a personal anecdote.

01:08:04 I remember when I applied here, I was reading through their docs.

01:08:06 And I told myself, I was like, if they can do half of the things they claim they can do in the docs, this is revolutionary.

01:08:12 I've never seen anything like this.

01:08:13 And it turns out we do all the things we say in our docs.

01:08:16 It's probably the most interesting tech product I've ever worked with in my career.

01:08:20 And I know that I will be working with it probably for the rest of my career.

01:08:23 It fascinates me and I love playing with it.

01:08:26 Like I build temporal applications at home for fun just because it's like, oh, look,

01:08:30 I don't have to worry about someone's API going down anymore.

01:08:33 Yay.

01:08:33 Yeah.

01:08:34 It's awesome.

01:08:35 So I hope you enjoy it as much as I do.

01:08:37 I'm pretty impressed.

01:08:39 All right.

01:08:39 Well, thanks for being on the show.

01:08:41 Thanks for coming on and sharing everything.

01:08:42 Yeah, it was great.

01:08:43 Great to talk with you.

01:08:44 Yeah, you as well.

01:08:45 Bye bye.

01:08:46 This has been another episode of Talk Python To Me.

01:08:50 Thank you to our sponsors.

01:08:51 Be sure to check out what they're offering.

01:08:52 It really helps support the show.

01:08:54 This episode is sponsored by Posit Connect from the makers of Shiny.

01:08:58 Publish, share, and deploy all of your data projects that you're creating using Python.

01:09:03 Streamlit, Dash, Shiny, Bokeh, FastAPI, Flask, Quarto, Reports, Dashboards, and APIs.

01:09:10 Posit Connect supports all of them.

01:09:12 Try Posit Connect for free by going to talkpython.fm/posit, P-O-S-I-T.

01:09:18 The PyBay Conference is returning to the UCSF Mission Bay Conference Center in San Francisco, California on October 18th, 2025.

01:09:26 Get your ticket and pick up a free conference course bundle from Talk Python.

01:09:31 Get started at talkpython.fm/pybay.

01:09:34 Want to level up your Python?

01:09:35 We have one of the largest catalogs of Python video courses over at Talk Python.

01:09:39 Our content ranges from true beginners to deeply advanced topics like memory and async.

01:09:44 And best of all, there's not a subscription in sight.

01:09:47 Check it out for yourself at training.talkpython.fm.

01:09:50 Be sure to subscribe to the show, open your favorite podcast app, and search for Python.

01:09:55 We should be right at the top.

01:09:56 You can also find the iTunes feed at /itunes, the Google Play feed at /play,

01:10:01 and the direct RSS feed at /rss on talkpython.fm.

01:10:06 We're live streaming most of our recordings these days.

01:10:08 If you want to be part of the show and have your comments featured on the air,

01:10:12 be sure to subscribe to our YouTube channel at talkpython.fm/youtube.

01:10:17 This is your host, Michael Kennedy. Thanks so much for listening. I really appreciate it.

01:10:21 Now get out there and write some Python code.