#317: Python at the US Federal Election Commission Transcript

00:00 When you think of government software development and projects, do fast apps and modern tech stacks jumped to mind? Probably not. So you'll be delighted to hear from our guest, Laura Buford. She's the tech lead at the US Federal Election Commission . She and her team have built a very modern tech stack running modern flask web apps with API's powered by SQLAlchemy and Flask Restful. This app and its API are available as open source on GitHub, and they deploy it with continuous delivery, right out to cloud.gov. There are lots of lessons to learn for governmental agencies around the world as well as private organizations, small and large. This is talk Python to me, Episode 317, recorded may 19 2021.

00:52 Welcome to talk Python to me, a weekly podcast on Python, the language, the libraries, the ecosystem, and the personalities. This is your host, Michael Kennedy, follow me on Twitter, where I'm @mkennedy, and keep up with the show and listen to past episodes at talkpython.fm and follow the show on Twitter via @talkpython. This episode is brought to you by Square and US over at Talk Python Training. Please check out what we're offering during our segments. It really helps support the show. Laura, welcome to talk Python to me. Thank you very much, Michael. Happy to be here. Yeah, I'm so happy to have you here as well. You're doing a whole bunch of interesting things around open source and the US government. And I find government to be so interesting. Because there's a lot of data. There's a lot of technology. There's many users. All right, we got it. 300 million users in the US are so in some sense. And yet a lot of times the technology there feel so old and ancient. And yet what you're building and some of the change that you're leading is, is very modern, right? You could put this up beside a lot of the startups in Silicon Valley, and it would look amazing still, so well done. Well, thank you so much for your kind words. Yeah, absolutely. Now before we dive into all the cool Python things, let's just start with your story. How'd you get into programming and Python? I have always been interested in computers. I love to tinker with my computers at home, like couldn't say that I had a natural aptitude for software development. I took classes in high school and college, I thought I wanted to be a computer science major and really struggled with writing software. It was a very exciting time politically at the time. So I decided want to be a political science and history major. And my mom really encouraged me to finish my minor which was great advice, Mom, thank you, I finished my computer science minor. When when I first graduated, I really wanted to get involved in campaigns. It was 2007 2008. And I found a I'm from DC, I came back to DC, I found a local political software company that was hiring. And I had originally interviewed for administrative assistant position and the founder recommended I work in tech support. I was really surprised because I felt like I didn't know what I was doing. But it was a really great organization. It taught me everything I needed to know and really loved. That's cool. Did you have one of those experiences where you're like, oh, there's no way I can help out with this tech support stuff. And then you get there, you're like, I'm actually able to help quite a bit. Yeah, you learn some skills. So we had one major system that we supported, and then a couple other smaller software. And I never really learned how to use myself. So you learned some tricks like, so walk me through the exact steps you've taken so far, you know, and then you sort of follow along with them. And you're like, have you tried clicking this, this looks promising. Let's try clicking on this and see what happens. So how funny. Yeah, tech support was a great first job out of college, I learned a lot helped me with my creative problem solving skills. And I really learned the software well to the point where, after a few years of tech support, I would send the developers the line of code where the bug occurred and say, you know, here's the bug that the user reported. So they asked me to come work on the development side. And I did that for a little while. And then I was still struggling with my confidence. It wasn't really a great fit. And I couldn't quite figure out why. I didn't have a lot of support. This is before a lot of the meetup culture on this was 2012. There are a lot of now I found there a lot of communities for helping people get into tech, but DC Yeah, the communities have really grown a lot last five years or so it's a lot a lot nicer in that absolutely . Yeah, and DC wasn't really a tech hub. It wasn't really known for being much of a tech hub. So I had a lot of self doubt. And I wasn't sure that being a developer was right for me. So but I while I was working at this company, it was a company that helped campaigns file their campaign finance disclosures, the FEC. So political campaigns in the in the US need to disclose money that they've raised and spent towards their election. This is if they've privately done it or through they get government support or something like that, right? You have options FEC provides free filing software and it's everything you might imagine.

05:00 Free government software might look like and we're working on that. But they're also third party companies like, like equivalent of like a Turbo Tax for filing campaign finance disclosures. And so that's what that company did. And I just fell in love with all the rules in the campaign finance, it's sort of like super nerdy, like anyone who gets excited about filing their taxes and all the different scenarios, it's very similar amount of rules for who can give how much under what circumstances there are situations where individuals can gift goods and services to campaigns, but there are restrictions on those types of gifts. So I really just enjoyed that. World campaign finance. Yeah, even though I had been a political science major in college actually hadn't learned that much about the FEC, and campaign finance laws and regulations. And it was kind of before campaign finance was a hotbed issue, might can sometimes be a hotbed issue now. But this was not something we covered a lot in class. It's not something that was getting covered a lot in the media. So those three work that I learned about FEC disclosure rules and campaign fundraising, whereas before had really learned more about voting. So fundraising is what allows the running of the campaign to take take place. Yeah, so much, at least in the US is about running ads, and hiring people to go out and knock on doors and all sorts of stuff like that, right? So even though really, you'd hope it's kind of some sort of meritocracy and you debate and then you choose the idea there's money's involved. Yes, it has become a hot, hot topic and a lot of ways, but I think at the time I was learning about it was the money was often thought to be an indicator of support. So if you could raise a lot of money from a lot of different people, it was an indicator that you had widespread support, and that those folks were even if they're just giving you $20, towards the campaign, they would help you support, like have office space, and like you said, print ads and write some sort of if you're going to give money, you're likely you would vote for them, right, giving money as a bigger signal than actually voting. So it's an early signal, I guess. Absolutely. So I wasn't sure being a software developer was right for me. So I went to go work for the FEC in a less technical role, where I worked to help the FEC with more of the enforcement side of things, making sure that the campaign finance reports were filled out properly, if people have questions about how to comply with the law, if there's any disparities or anything that was unclear on the reports, I would help clear, like, send correspondence on the public record, asking the regulated community that, you know, the campaigns and the packs and parties to further clarify their reports. And there was an element of tech support in that role. I also did tech support for the free filing software. And anyone who's done tech support, sometimes it can get pretty, like answering the same question over and over again, no matter how much you care it can can sort of wear on you. So I was looking for a challenge. I'm so happy to have your question. I'm really here to support you for the 100th time. Yes. And this is the first time they're asking, you know, so you have to? Of course, they it's not there, they don't know. Right? And there are a lot of very confusing rules. And yeah, some of the best advice I got for my supervisor was you don't have to defend the agency, you know, so if someone says, This is really confusing, you can say it is confusing. Let me see if I can help you work through it instead of, you know, trying to defend the regulations for. And so after about five years of doing that job, I wanted to challenge I wanted to change. And there was an opening on the website, Project team@ftc.gov. And my mentor at the time had recommended that I come work on this project was really cool project. And I can talk about why it's cool. And I said, I haven't written a line of code in five years, you know, there's no way I can work there. And I met with the team. And they were willing to if I was willing to work hard to get up to speed, they were willing to work with me since I had a lot of knowledge about the area and past experience. So basically did a crash course in Python, with a lot more support this time around, there are all these organizations in DC for under represented genders and tech and just new developers. And so really did a deep dive and kind of was able to talk myself onto the team. And it was drinking from the firehose, and really just, yeah, really had fun, though, it was a super fun. That's cool. You know, one of the things that I hear in your story here, that sounds a little bit different, between the two experiences, you're going to college and getting your minor and CS versus, you know, being really passionate about the politics in elections, and then learning code to sort of advance that mission. And those are really different in that, you know, so often in computer science, it's like, well, here's some random algorithm, we're going to study the algorithm, then you can reproduce the algorithm and how boring and dry and separate from like what you're interested in, is that right? Whereas this is like, well, I've got to go through this technology to use it as like a way to really achieve the goals I'm after, and I think those are just such different experiences. Would you agree? Absolutely. I distinctly remember one of my projects in college was to build a calendar.

10:00 And I was like, I have a calendar.

10:04 You have to remember, like, the paperwork and I got a digital and

10:07 yeah. And so it just wasn't very passionate about some of the projects. And they're good projects for teaching the concepts, but not necessarily the can be difficult to find. And I think there's been a re-evaluation of how certain ways of writing software is taught a little more beginner friendly. If you have a goal, you have something you want to build, and you know why you want to work on it? Well, then it's super exciting. But if it's just dry, I think we lose a lot of people who don't have some sort of personal drive that's going to get them through that is not necessarily part of the lessons are part of the official experience, you know, absolutely. And the thing I wanted to add to that is that I didn't learn at the time I was in school, I studied computer science, and I've sort of figured out that software development and computer science are can be somewhat different disciplines. And yes, I never learned how to really write software in college. And what I've learned the hard way is to really start small, get the smallest piece working, change the smallest part of it. And can you make that even smaller, maybe just hard code something for now. Whereas before I would write in college, I would write top to bottom, write the whole thing out and then just explode about a semicolon buried on line 463. Exactly. So those learning those skills has helped me to a lot more fun. A lot less miserable. Yeah. Fantastic. So I think you've given us quite a bit of background already about the FEC, Federal Election Commission. But I think two things are interesting to talk about real quickly here. One is this whole agency was born, I guess, a tiny bit before I was actually but it was born out of a financial scandal in 1972. Right, yes. After the Watergate scandal. Yeah, I bet people have heard of that one. I hope so.

11:50 I would hope so do. The other thing is, you know, you look at the FEC. And he said, it's not a huge organization, the government, but how many people work there, we have about 300 full time employees. That's federal employees. So they work for the US government. And for on fec.gov. We have a team of about, you know, we have a cross functional team. But we have about nine developers and the user experience designer who work on the technical side of the site. Yeah, cool. It's pretty small group of developers probably all work pretty closely. I'm sure it's a fun place to be. It really is. Yeah. And before we move on too far from the opening, Kim van wyck, who was one of the Ask me anything hosts for me, Hey, Kim says I strongly suspect that also helps you remember a software topic six months later, if you did something you found interesting. while learning? Absolutely, I would definitely say so. Alright, so as we get into the code side of things, maybe we could talk a little bit about the type of data to give people a sense of what we're working with here. So one of the things you can do is you can go to fec.gov. And you go to all data, and there's all kinds of stuff, you can learn about raising money spending money, candidates, I could go here, I could select my state, Oregon, I could go pick my location around Portland, and pull that up. And it'll do things like I could go up and say learn about the house, I guess it will show this person and this party raised $64,000 and had $110,000 just disbursements, how much cash they have on hand right now. So apparently $711,000, which is kind of insane. And then you can go see the sources for this data. Tell us about the kind of data and how people use it there. Yeah, that's a great our electoral profile pages, I'm really proud of those, we can see who's running in which in each election, yeah, it's beautiful. And you get like a colorful map of the areas, you just click on it, it pulls it right up. It's really neat. Thank you so much. And as you were demoing it live here, it was looking snappy, which just really warmed my heart.

13:44 So when we think about filing our taxes with the IRS, in the US, at least we have to disclose you know how much money we made and various other basically the money we made in our taxes. It's all summary level data. campaigns need to file the not only the summary level data, but also the transaction level data. So from every contributor who's given aggregate of $200, or more in a given election, they have to disclose their name and address employer occupation and the details of that transaction. And so it's a large volume of data, especially at the presidential level, or, or national level. Yes, yes, presidential level and like hundreds of millions of records, I think we're on track to have a total of 1 billion records by the end of the 2022 elections. It's just a lot of data. So it's not just how much they've raised. It's the individual contributions that go into that. So from here, if you wanted to click on the name of the candidate you're looking at for your district, it'll take you to the candidates profile page. It looks like this is a house candidate, and they run every two years. So there's a browse receipts button so you can actually see the individual contributions that made up and it's 2022. There hasn't been a lot of data yet for 2022 because it's early in the elections.

15:00 cycle Oh my gosh. So I'm in here looking at this and it says, Here's Diane snow, who had donated how much you know, $50. Like these are individual contributions and all sorts of stuff, right? This is crazy, the crazy detail. Yes. And we've had detailed we've we it's part of our mission to make detailed contributor data and not just contributor but expenditure data, how that the committee's are spending their money, since back to the late 70s. So we do have all the historical data here to explore. You could export this to excel spreadsheet, or techniques, a CSV, but at the top of the page, you could filter for, like all the money that's been received for 2022. So you could eliminate your there, the breadcrumbs there with the little X's on them, you could, you know, just maybe that one candidate across all time or all the money for 2022. And so it is for all the data is again, like about coming up on, we had about 500 million transactional transactions last in the 2020 election, that equals about four terabytes of data. And the expectation is it is filterable. It's searchable, it's exportable, and it user expectations that you'll be able to get the information quickly. So we spend a lot of time focusing on performance improvement for the site, which is powered by our campaign finance Data API under the hood.

16:18 This portion of talk python to me is brought to you by Square. Payment acceptance can be one of the most painful parts of building a web app for a business. When implementing Checkout, you want it to be simple to build secure and slick to use. "Squares new web payment SDK" raises the bar in the payment acceptance developer experience and provides a best in class interface for merchants and buyers. With it, you can build a customized branded payment experience and never miss a sale. deliver a highly responsive payments flow across web and mobile that integrates with credit cards and debit cards, digital wallets like Apple Pay and Google ACH bank payments and even gift cards. For more complex transactions. Follow up actions by the customer can include completing a payment authentication step, filling in a credit line application form or doing background risk checks on the buyers device. And developers don't even need to know if the payment method requires validation. Square hides the complexity from the seller and guides the buyer through the necessary steps. Getting started with a new web payment SDK is easy. Simply include the web payment SDK, JavaScript blog and element on the page where you want the payment form to appear. And then attach hooks for your custom behavior. Learn more about integrating with squares web payments SDK at talkpython.fm/square, or just click the link in your podcast player show notes. That's talk python.fm/square.

17:42 There's a whole bunch of API's. And you can even check out the code on GitHub, which is super interesting. This is the kind of data that people get. And I think it's it's pretty good. So one of the thing that's interesting to me is if I'm over here on 'FEC.GOV', this looks like a pretty well designed website. It looks nice, you can jump around. But then as you were showing me those different locations, it looks pretty data driven. Like there's a decent amount of logic behind it. But you said that this is built on wagtail. Is that right? So there are when you go to ftc.gov. It's mainly to applications, okay, the content management system we use is built with wagtail. So they're kind of three different buckets on the site. There's the campaign finance data, which we were just discussing, right, which is mostly powered by the API, we use some Ajax calls just basically JavaScript and data tables to display the data that the API provides. And this application that we're looking at is the Django based CMS application, right? And so for people who don't know, maybe just tell them really, what's the elevator pitch on wagtail? What is this thing? Oh, wagtail is a really great admin for the Django framework. So Django has a built in admin framework that so just to kind of give some context, the disclosure, the help for candidates and committees section of our site has information for a regulated community. So they want to find out, can I take this money from the source? Or what are the limits on contributions, they can go to help for candidates and committees and find the answers to their questions. And so that's stored in a in a in the Django database? Well, it's stored in a Postgres database that our Django application connects to. And Django as a web application framework has a built in admin. So you know, if you're working on this, if you've worked on like the Django tutorial or anything like that, you can go slash admin and use the built in Django admin to manage your content. So if you wanted to modify an article or you know you're working on a blog, or whatever, that's how you do it. wagtail is an open source tool that is that has more user friendly functionality on top of the existing Django admin little closer to like WordPress ish, kind. Yeah, it's got some great features. You can do custom templates. I don't work as I work more on the data sides, but we have a content team that uses wagtail to manage the content on the site. It seems like it's very customizable and just has more features than the

20:00 Django admin that I'm sure you can build yourself. But this is a free open source project that allows you to have a more robust admin interface. Yeah, a lot of times, that's a big platform decision, people have to make like, Oh, well, we want to have this admin thing that people let people go in. And just developers don't have to add the informational pages and just maintain the basic data of the site. So I can either build something in Django, and then it's kind of on me, or we got to go through the admin backend in a clunky way. Or I've got to build something or we go WordPress, and then but now you're out of Python, and you're in this other Joomla, or WordPress world that you're not really able to extend and it just becomes almost a static site. And so things like wagtail are cool, because they let you stay in your world if you're a Python developer, but then also have a nicer experience on top of it for the users. Yeah, wagtail it's been seems like it's been great for us. Yeah. Very cool. Very cool. Okay, so we've got that. And then we've also got the open FEC API is what I'm trying to say here, which is RESTful API around all this data that we've been talking about, right? Tell us about this. Sure. So if anyone hasn't worked with API's before, the API is just the source for all the campaign finance data. So one of my friends put it in a cute way we drink our own champagne. So any of the data you see on fec.gov is coming from the API. So you don't need to know how to use an API, you never need to touch the API. If you don't want to you can you go to fec.gov and get all of the data that you that you want out of this. That's really cool. I've heard the term dog feeding, eat, you're

21:33 drinking your own champagne sounds way better? Same thing, right? Way more unpleasant? I imagine. Yeah, it's absolutely. When we were interacting with the homepage, the old data section, that was JavaScript talking to this as well, just so you're basically the first client of this API. Correct? exactly correct. Okay. However, if the FEC went fec.gov doesn't have the information you need, or you want to build your own tools on top of the API, we do make that available to you directly. So you can bypass fec.gov and go directly to the source to get the data. How old is this API? How long has it been around, we realized around 2014, that the legacy systems we were running, were going to have a really big problem with our exponentially growing data. So in 2015, we partnered with 18 F, which is a government organization within consulting for government, by government. And we wanted to build an open source cloud based application that had an API. So we actually built the we launched the API before we launched fec.gov. So it's it's been, we've been drinking our own champagne since around 2015. It's been a party for 16 years.

22:41 So I've heard of 18 F . It sounds like these are some really tech savvy folks that can drop in and help the other developers on other organizations or divisions and the government say, look, you want to, you want to modernize what you're doing. Let us come spend six months or a year and we'll help you get going and then hand it off to you. I was hugely fortunate to have overlapped with 18F when I joined the project, because I again, joined the project hadn't done a crash course in learning Python, but not having had a lot of other skills that were needed to be successful in the project. So I was able to have really, really excellent mentors that helped me learn in a lot of government agencies. This was our like our first cloud based application, I think, hopefully, I'll get a chance to talk about 'cloud.gov', which is the platform that we run. Yeah, absolutely. But this is all new to us. And so 18F was really wonderful. They taught us a lot. They taught us how to build software, an agile way how to practice open source, and not just right, not just share your code, but work in the open. So if you go to our code repositories, you can see what I'm currently working on all of our issues, all of our tickets are public, and you can see that the priorities and how things are going. And now there's my change, or I was tweaking our workers for an event. And actually, I'm thinking of moving to a different mechanism for our asynchronous requests. And you can follow along with my research and if you have opinions about workers versus threads for G unicorn, I think, I don't know. I call it g unicorn. I say G unicorn. Some people say gunicorn to me, I don't know it's like a unicorn. But

24:15 if you have opinions on this, like please chime in all those issues are public. I've done some research on pros and cons of the different configuration. But yeah, 18F led the way on how to work in agile way and open way. And we were super lucky to be able to work with Yeah, this is really neat. So I'm sitting here looking@github.com/fec.gov/openfec and this is the GitHub repository for the API. Right? Correct. And start some forked off like well, we'll dump to it every now and then, like you said, this is where you all work. Yes, this is where we deploy from, we use circle ci, they have a really great open source tear. So when we merged the develop branch, it deploys to conduct the develop space. So this is our source of truth for the code nice and looks like you're using sort of feature branch style

25:00 PR development which is feels modern and fresh to me. Another thing that's interesting is if I come down here we've got our, my requirements for my JSON here and we've got flask, we've got flask restful and flask SQLAlchemy. This looks like a fun tech stack. I think you are. Yeah, I've enjoyed it. I gave a talk at flask con, which I think was the first time they ran it. I guess about a year ago, that goes into a little bit more technical detail about how we use flask for the API. Yeah, super interesting. One thing I'm noticing here, there's a flask.

25:30 1.1.1 , which is great. Last week, just last week, maybe five days ago, 2came out. They also have new async plans going on? Are you all thinking about that already? Or are you just still putting out features for now with our data doubling every two years, and we are down a couple staff members? We're sort of in the hopefully hang on until next year and more money plays if it's not broken, but we do have a regular dependency checking for security issues, or do you kind of if it's not, if there's an issue, we do update it? Basically, we're not in the place to take that on quite yet. But I did what I did hear the news, and I'm excited to see what that might that might help us accomplish. Yeah, absolutely. You've mentioned code.gov. I think that's a pretty interesting place. Let's talk about a couple of other things. Let's talk about this communities of practice @digital.gov. So he talked about working on the open. So let's talk about some of the things going on there that I think are pretty interesting. The story of these community of practices. So I'm here I can see that there's like an AI section. And there's a blockchain section and their cloud and infrastructure, which is getting close to cloud. gov stories and DevOps. What is this place? Yeah, so one of the things that surprised me about coming to government is how so FEC is an independent regulatory agency, which means it's not part of the one of the big cabinet agencies. And so kind of surprised me, we're all sort of doing similar things. And we sort of struggle to talk to each other about, you know, what's working, what's not working, it's not always clear who you might talk to just at other agencies about ideas and suggestions they have were pretty small potatoes compared to a lot of the bigger agencies, and we have a lot to learn from them. But I don't know how I'd feel about just sort of cold emailing someone I stopped on LinkedIn or something like that, it can be hard to connect to other people working on very similar problems, because you could hear that maybe the people in I don't know, IRS or something, they're doing something cool. And you because it's kind of related to what you're doing. And in some sense, it'd be neat to see how that might work for you all, but even don't know those people, right? Like the government is huge. And it's like 100,000, or more that are huge, huge company that many people I've never met, they're not close to each other. Right. So this is a way to set up some of those communities around ideas. Is that right? Absolutely. Yeah, it's really wonderful. And we've talked to people at the CFPB. They use I know, they use wagtail. And we've sort of traded ideas about what's working well, and what's some of the challenges we're facing with the communities of practice, or a kind of a formal way of bringing people together. And I've really enjoyed the DevOps community of practice. And they have a monthly short half hour Tech Talk, where they invite people to talk about the types of things they're working on, and successes and challenges. Yeah, super neat. Then another one was this digital services playbook. So I think the digital services is something I just learned about a little while ago, three years ago, I think I had, oh, gosh, I'm sorry, I don't remember the guests name. But I had one of the developers on from the US digital service, which I thought was a really interesting, just a really interesting organization that I hadn't heard about, tell people what that is real quick. And then what is this digital services playbook? I think this seems pretty neat. Sure. So I'm not affiliated with the US digital service in any way. However, I have met people who work there. And I've always been super impressed by them. And they'd have a similar mission to at Neff, there are people out there who've written blog posts about the difference, but they're through the Executive Office of the President. And they do have a similar mission of helping modernize government, government technologies. I think one of the biggest challenges and everyone I've ever spoken to who works in government is contracting. So if you were lucky enough to have staff, a federal employees that you can have some more flexibility around how you do the work. But if you are there are some advantages and disadvantages to contracting. But it's it can be very difficult to get right. And so one of the resources I often point my acquaintances towards that are trying to modernize their technology is the digital US digital services playbook. And it goes through how to approach your modernizing your technology, it talks about best practices for contracting and kind of gives you a roadmap. And another resource I'd like to touch on in a second. There's a statistic that I think like 80% of all government, custom build software projects in government fail. And I think that contracting is a really hard thing to get right. It's sort of like imagine outsourcing for the private sector. It has risks to outsource because it just has trade offs and the the amount of management and oversight you might need to have for a contractor.

30:00 Or outsourced project might differ from if it's in house. So this is a really great the digital services playbook is a really great resource for other federal employees or federal people working on federal projects. And then also just anyone looking to modernize their, their systems might find this helpful as well. Yeah, they've got 13 apparently, they're not superstitious, which is cool.

30:25 Got a 13 Digital Service plays. So things like that are pretty, I guess, pretty standard, but also good, you know, I understand what people need, make it simple and intuitive. But then stuff that maybe people within government don't necessarily, they wouldn't necessarily think about, like default to open or choose a modern technology stack, and things like that, you know, default to open for example, right? You might think, well, this is data about our people. And it's super sensitive. That's one reason to keep it hidden. I mean, obviously, what you're given away supposed to be public in FEC. And then another is, you know, for example, like, if you share your source code, maybe people who want to do bad things with it will come in, you know, use that to find some way to do something that they shouldn't, right. We've seen a lot of security in open source, as well, right? By having having multiple eyes on it. Absolutely. And I think us building in the open from the beginning is the best way to successfully have an open source project, because retro actively open sourcing that has been built using the assumption that you'll have protection throughout the station, can be really risky. And unfortunately, if you're building a closed source system, it can make it a little easier to have some bad practices like hard coding credentials, I think there can be some misunderstandings about how open source works, like no one can come in and just change the code for fec.gov. They need to submit a request which is reviewed by one of our team members and gone through a security review as part of that process. So it kind of open source sort of encourages best practices as you might think that you're you've the strongest network security that ever exists. If something phishing is, I think one of the biggest weaknesses that organizations face. And so I think that's sometimes called zero trust.

32:10 assume they're gonna get to the code either way, so you might as well, we use best practices for securing your code. And the best way to do that is from the very beginning, assume that that you're going to have eyes on it.

32:22 Talk Python to me partially supported by our training courses. Flask is one of if not the most popular Python web frameworks and developers are adopting it for the smallest in many of the largest Python based websites and API's. If you want to learn flask, we built a fantastic course called "Building data driven web apps with flask and SQLAlchemy". In this course, we build a "PyPI.org clone" from scratch using flask and SQLAlchemy, you'll learn many of the major ingredients needed to build most web apps. If this sounds amazing, just visit 'talkpython.fm/flask' or email us at sales@talk python.fm.

33:00 Before we move off this digital services Kim on live stream says David Holmes was your guest. Thank you, Kim. for that. Oh, yes, I remember that now. And says, as a non US citizen, he's from South Africa. Do organizations like the FEC use the US digital services? Or is it all in house? It sounds like it's you kind of separate from them. We partnered with 18F back in 2015. Again, they're sort of like consulting for government by government. They rolled off the 18F rolled off our project in 2018 ish. And we haven't partnered with us Digital Service, I think they choose their projects that can be influenced by the Executive Office of the President. So we haven't worked with them. But I've met, we do have an exchange of ideas for we're lucky to go to PyCon , for example, and meet other people who work there. And we can talk about what's working well, and so that is super valuable to be able to have that sort of relationship. But yeah, fantastic. One other resource that I think you were hinting at is this de risking guide from 18F, is that right? Yes, I'm a huge fan of this. They have a federal and non federal version. So if your state, that's a whole nother thing in the US that we have to deal with, where we have beings, the federal government and the state government, and just like all the federal agencies are out there replicating similar efforts, all the states are replicating similar efforts with their own budgets and their own challenges. And in some ways, if we're looking at things like security, we have a disadvantage, because we're splitting up our resources in many into many slices. So I love that 18F also built a non federal guide to help with the state and local governments as well. Yeah, there's a I think this also applies to large organizations that have maybe been around a while but you know, here's a stat from government says only 13% of large government IT projects succeed, and implementing custom software projects can be extraordinarily costly and risky in a government setting. And here's the big thing that waterfall software development remains the standard at all levels.

35:00 So in budgeting and management and rollout, all those things, which is very different from here's my PR, let's push that with CI to cloud.gov. Right? I absolutely feel like someone said recently, government contracting is really designed to buy tanks. And building software isn't really very, doesn't have a lot in common buying a tank. And so yes, it is difficult to get right. And this guide has some really practical recommendations for how to manage the process. And one of my biggest things I'd like to highlight is the quality assurance surveillance plan or cost that's mentioned in this guide. You can build it into your contract as a way to measure whether the quality of the software is acceptable. It can help you as the person who's managing the contract and help your contractors understand what acceptable levels of quality are. So it looks like the cost is highlighted there at the top and one of the menus. Oh, yeah, there you go. And they have a sample class, which is really cool. And so if you scroll down a little bit there in this page, you can see some example quality measurement ideas. So using an automated tool to check for accessibility and security vulnerabilities, you should be able to deploy with a single command following a wasp best practices, there are tools that can be used to check for vulnerabilities doing user research and every sprint. These are just really great recommendations. Oh, yeah. Honey, there's little. How do you do this measurement of a set of like, what is the acceptable quality level? What How do you measure it? So on? Yeah, this is really neat. I like it huge. I'm a huge fan. Yeah, cool. Now let me pull up one thing that I thought was pretty interesting here, I'm seeing this. I'm seeing this all over the different sites that where I'm seeing on your fec.gov, and so on, if I click here, it says, right at the top in official website of the US government, how do you know? And it says, well, it's .gov. And then it talks about this little lock. And if I look at the lock, it says this is all done through Let's Encrypt, which Let's Encrypt is awesome. You know, it's a way to basically automate getting SSL certificates. But it feels like very modern and, and cool and hip. And to see the government rolling that out, as that was kind of a surprise. And I feel like this ties a little bit back to our next topic, which is cloud.gov. Absolutely, yes. The first thing I want to point out is the banner you mentioned at the top of the page is part of the US web development standards webt development design system. It's an initiative to bring the look and feel of government websites in line with each other. So it can help build trust, and users can be assured that they're they are in the right place. If you're if this is the first time you've heard of it, they have some really great resources, USWDS and I might be getting an acronym like the what it stands for slightly wrong, but they have components you can use on your site. It's all open source, web design system. They go, thank you. And so you can see on their website, they have a official website of the US government, here's how you know, at the top of their page, just like we do not even have versioning, like version 2.11.2. Yeah, this is a really great resource as well. It's all open source. So even if you aren't in government, if you wanted to borrow any design is a design system. And people who know more about design systems would know more about what that means. But we do use parts of that in our site. But you mentioned, Let's Encrypt as part of how we manage our certificates. fec.gov is run on a cloud.gov. And cloud. gov is a platform as a service that's available to government agencies. And so when you build a cloud based system, you have the option of building your own cloud infrastructure like AWS, or Azure, Google Cloud, from scratch, or you could use a platform as a service. And there are pros and cons trade offs of each approach. But we find as a small agency, cloud misconfiguration is one of the leading causes of security vulnerable, like security breaches. Yeah, how often do we hear about here is a open source server running in the cloud without without authentication, or here is 10 million records of sensitive data in a non secured public s3 bucket because it was a pain to deal with security went to integrate that with the app or whatever, right? These are real problems. Absolutely. And the cloud itself is very secure. But you need to make sure you're configuring it properly, take advantage of those security features. And so that's not easy to do, or I haven't found it easy to do there. I would say like they give you enough. Well, I won't use that phrase phrase ology, but it's very powerful and difficult to get right. Yeah, I feel like AWS and Azure, those types of places. They're very powerful but you go there and it's just you know, deer in the headlights so many choices, just even at the what area Do you want to go into and then you'd factor well what are the security implications of changing this switch down in some level? It's crazy. Yeah, yeah, I won't even sign up for a free tier account cuz I'm convinced knowing me someone's going to get access in mind Bitcoin. I'm gonna have a $10,000

40:00 Like they they make you put your credit card in there? And I'm not Yeah.

40:04 So it takes a lot of expertise to build your own infrastructure in the cloud in a secure way, our agency being smaller, we could, we could go ahead and do that. But cloud.gov has a lot of advantages. And so what it does is it sits on top of AWS as a management layer. And so it manages cloud security. It manages authentication, you know, who has access to what it makes it super easy to deploy, really easy to keep our infrastructure as code, you can use infrastructure as code in AWS. But I still haven't really figured out how to do that. It seems wolf again, so many Yeah, haven't, I find it no matter how many trainings I take, I still find it difficult to work with so cloud.gov has a lot of advantages over building your own infrastructure has fixed pricing again, like if you mess something up and running a machine for too long, and you get a surprise, $10,000 bill, that doesn't happen in cloud.gov, because they have cost engineers and they have fixed pricing. in government, if you want to build a technical system, you have to have an it's called an ATO an authority to operate. It's all this paperwork that you need to do in order to prove that your system is secure enough, by using cloud.gov. You can inherit from their ATO and kind of save yourself a lot of paperwork, just put the checkbox I'm using cloud. gov, essentially. I mean, you mentioned we use, Let's Encrypt, and my first reaction was cool.

41:28 They managed a lot for us and make it super easy to deploy. We have scalability through cloud. gov. And we can focus more of our time and energy on the data and application side of development, which in and of itself is pretty time consuming for us. Yeah, maybe you could talk about two things. Maybe you could talk about what is your deployment look like? I think using AWS at the moment, is that right? And then you're using Postgres, maybe just talk about how you're running all that the API's and stuff out there. Sure. So application architecture, we run our application tier on Cloud. gov, and our database tier is actually we have our own managed AWS, Aurora, Postgres that we run. Because while cloud.gov does offer database services, our database is so massive that we wanted and our we have really experienced DBAs, who wanted more granular control over the settings on the database. Yeah, you talked about how you were in a place where if you want to scale horizontally by adding more read servers, read replicas, that was a redeploy for app and a big challenge and moving to Aurora made that automatic behind the scenes, right? Yes, I did. And I can probably put this in the show notes. I did a talk on how we use Aurora for scale database scalability. So AWS, RDS, Postgres just kind of plain, I think it's like relational database service management, basically managed database service, when you have read replicas. So at some point, we need to run like three to four read replicas equivalent. So you wouldn't like the difference there. But we need to run on numerous machines at the same time on the database side, in order to meet our demand, I suspected this data is very read heavy, the way it's been used. Well, it is very read heavy, but we also have the data is always changing. So we have some really big filers. That's true if if you're doing like a big campaign people are every little contribution is streaming in, right, yeah. And there's some big packs that will have like 30 million records, 40 million records need to insert all in one night. So So we do have a lot of it's pretty insert heavy as well. But in the past, when you use already, AWS RDS, each read replica had its own connection string, we are using the 12 factor framework for application development, which is a few is that that's published online that basically considered best practices for cloud based application development. And it recommends using your attached services connecting to them through connection strings. So basically, you should be able to swap out your database for one database to be able to swap out to another. So this is backing services. Number four there treat backing services attached resources. So our application connect to the database through environment variable connection string, and we're using plain RDS before each replica had its own connection string. So in order to scale, we could have set up auto scaling on that database side. But we would have had to come up with a fairly fancy way of consuming that new connection string, adding it to the environment variable, the application and then redeploying the application to reflect that that new connection string, so basically wasn't functionally, it wasn't very feasible to us, for us to have database auto scaling. And after doing some research, we decided to move to AWS, Aurora, Postgres, which has one connection string that you connect to, and it handles auto scaling on its end. So it doesn't call them replicas, it calls them instances but they're functionally replicas. So we connect to one connection string and the could have two instances running and then if it

45:00 has high CPU, it'll spin up some more machines to handle the database side. Cool. So you can set up like auto scaling rules and things along those lines. Yes. Okay, so that's the machine under the hood for the API. Yeah. And during one of your talks, was that presentation that I linked to? Is that one given to the community of practice group? Is that where that was, is? Yeah, that's all coming together. I see. So I'll link to that as well. But during that talk, you showed some need so see allies for deployment and configuring scale and things like that it was really cool. You over there, say, Hey, I just want five more web Brennan's boom, and you have them right. That's, that was quite neat how that works. Yeah. cloud.gov is based off of Cloud Foundry, which is an open source platform management tool. And so it can be a little bit difficult to run this on your own. But there are managed services that run Cloud Foundry. And one of the big ideas, as I understand it, behind Cloud Foundry is making the developer experience really easy. So you can deploy with a single command CF push, so we use on cloud.gov, all the Cloud Foundry commands. And there you just with one command, you can scale up your the number of instances you're running your application. And it's practically immediate. I mean, I would say like under under three seconds to quadruple your mission number of machines. Yeah, you made an interesting comparison to that about your previous model with physical servers. Yeah, so our legacy system was run on physical servers. And it was really difficult to predict how much we were going to need we'd sometimes we'd overbuy, sometimes we would under buy, it could take three months to get a new server setup. And now it takes three seconds to get a new instance running. In addition, when we moved to Cloud.gov , we saved about $1.2 million a year in infrastructure costs, which was really big for us as a small agency. That's a huge thing. And I believe one of the numbers you threw out there previously is 1.4 million was around the infrastructure cost before so to go from 1.4 to 200,000. That's a huge difference. It's not like, well, it's 100 million, and it went to 99 million, it was much, much better, right? percentage wise, that's a great point. Yeah. And we were able to again, like minded, the thing I love about cloud.gov is not taking on that risk ourselves, both the security risk, and also just the the risk of miss configuring something and something going down it for a small agency like ours, it's really huge to be able to depend on them for that. You don't have to have a team of network and DevOps engineers just to keep it going. I suspect, like that's the cloud. gov site, and they just deal with it. Yeah, I think when I talked to them, if we had taken all of this in house, the amount of services that they offer, we would have needed to hire like six or seven more people, and we have a team of nine. So it was just not cost effective for us to bring that the infrastructure and house at this time. Alright, so let me ask you a question. I've been thinking about this, as we've been talking about all these things, seeing all the cool stuff that you're doing, seeing how 18F came in and helped and all of those. Now, I don't necessarily expect you to have an answer to this. But I just want to hear your thoughts. So the rollout of the Affordable Care Act in all of the drama around the websites and the challenge of scaling that and stuff? Do you think it would be different? I think it would be smoother, if like this world existed in a more polished way. And we tried to roll it out again, in the same way. I think that there are a lot of challenges in developing and delivering digital services. I think a lot of the expectations for digital service delivery are high, you know, it's like, well, Google can find this for me, and and if you compare the operating budget between my agency and Google, it's like, yeah, Google just talked about rolling out like their own supercomputer for this other thing, right? That's like, a different set of capabilities or whatever. Totally, totally, I would say it. To me, it all boils down to as boring as it sounds, it boils down to contracting. It really depends on how the contract is written and how its managed. And I don't know very much about it. But I do know that if you don't get it, right, it's very difficult to course correct. So absolutely, we can do better. But it's there are still a lot of challenges to preventing those sorts of issues. Yeah, of course. All right. Well, I think that's probably all the time, we have to dive deep in this. But this has been a super interesting look at what's happening over there. And, you know, honestly, it's heartening to see how much the government is advancing, using modern tech and really putting these systems in place. So well done. Thank you so much. It was a pleasure to speak with you today. Yeah, you bet. Now, before you get out of here, I got the two final questions, though. If you're going to work on the API, for example, write some flask and Python code. What editor do you use? I use sublime. And VS Code does package management better, I think but I just can't think myself move and notable PyPI package. You know, maybe something cool that used during the project that you're like, oh, people should really know about this. It doesn't get enough press. Sure. I'm a maintainer for flask, SQLAlchemy, so it's a really great package community has been super welcoming, and it's the only one I've been able to really get my hands on. So if you're looking for a project to work on all the flask projects are great to contribute to

50:00 Oh, fantastic. Tell people real quick. Why would you choose Flask SQLAlchemy over just SQLAlchemy? I think it depends on like getting started. It's pretty friendly to get started. And you can maybe start with that and then re evaluate what sort of custom settings you might need. Yeah. Cool. All right. couple comments from the live stream. Antonio says no, don't go. I really enjoyed the chat. And then Brandon says, super nursing subject. Thank you. Yeah, thanks. Thanks both for being here. final call to action, you know, people broadly, or maybe even people with in governments around the world who want to do better work with modern tech stack know, what do you tell them? Check out fec.gov the link to our GitHub is in the footer. We look at all the feedback as public anything you can get out of our open source project, take a look at who's running for office and would just love it if you got involved in our project. Very cool. Well, like I said, well done. Very Nice work. It seems like when I was working with the site, it was super snappy and beautiful Impaler so Excellent. Thanks for being here, Laura. I'll pass it along to my team. Thanks so much. Yeah, you bet. Thanks. Okay, bye.

51:03 This has been another episode of talk Python to me. Our guests on this episode was Laura Beauford has been brought to you by Square and US over at Talk Python Training. With Square, your web app can easily take payments seamlessly accept debit and credit cards as well as digital wallet payments. Get started building your own online payment form in three steps with "Squares Python SDK" at 'talkpython.fm/square'.

51:29 Want to level up your Python we have one of the largest catalogs of Python video courses over at talk Python. Our content ranges from true beginners to deeply advanced topics like memory and async. And best of all, there's not a subscription insight. Check it out for yourself at training.talk python.fm Be sure to subscribe to the show, open your favorite podcast app and search for Python. We should be right at the top. You can also find the iTunes feed at /itunes, the Google Play feed at /play and the direct RSS feed at /rss on talkpython.fm. We're live streaming most of our recordings these days. If you want to be part of the show and have your comments featured on the air, be sure to subscribe to our YouTube channel at

51:29 talkpython.fm/youtube. This is your host Michael Kennedy. Thanks so much for listening. I really appreciate it. Now get out there and write some Python code