When you think of government software development and projects, do fast apps and modern tech stacks
jump to mind? Probably not. So you'll be delighted to hear from our guest, Laura Buford. She's the
tech lead at the U.S. Federal Election Commission. She and her team have built a very modern tech
stack running modern Flask web apps with APIs powered by SQLAlchemy and Flask RESTful.
This app and its API are available as open source on GitHub, and they deploy it with continuous
delivery right out to cloud.gov. There are lots of lessons to learn for governmental agencies around
the world, as well as private organizations, small and large. This is Talk Python to Me, episode 317,
recorded May 19th, 2021.
Welcome to Talk Python to Me, a weekly podcast on Python, the language, the libraries, the
ecosystem, and the personalities. This is your host, Michael Kennedy. Follow me on Twitter where I'm
at mkennedy and keep up with the show and listen to past episodes at talkpython.fm and follow the
show on Twitter via at Talk Python. This episode is brought to you by Square and us over at Talk
Python Training. Please check out what we're offering during our segments. It really helps
support the show. Laura, welcome to Talk Python to Me.
Thank you very much, Michael. Happy to be here.
Yeah, I'm so happy to have you here as well. You're doing a whole bunch of interesting things
around open source and the U.S. government. And I find government to be so interesting because
there's a lot of data, there's a lot of technology, there's many users, right? We've got 300 million
users in the U.S. or so, in some sense. And yet, a lot of times, the technology there feels so
old and ancient. And yet, what you're building and some of the change that you're leading is
very modern, right? You could put this up beside a lot of the startups in Silicon Valley and it would
look amazing still. So, well done.
Well, thank you so much for your kind words.
Yeah, absolutely. Now, before we dive into all the cool Python things, let's just start with your
story. How'd you get into programming in Python?
I have always been interested in computers. I loved to tinker with the computers at home,
but I couldn't say that I had a natural aptitude for software development. I took classes in high
school and college. I thought I wanted to be a computer science major and really struggled with
writing software. It was a very exciting time politically at the time, so I decided I wanted
to be a political science and history major. And my mom really encouraged me to finish my minor,
which was great advice, mom. Thank you. I finished my computer science minor.
And when I first graduated, I really wanted to get involved in campaigns. It was 2007, 2008. And I found
a, I'm from DC. I came back to DC. I found a local political software company that was hiring.
And I had originally interviewed for an administrative assistant position and the founder recommended I work
in tech support. I was really surprised because I felt like I didn't know what I was doing,
but it was a really great organization. It taught me everything I need to know and really loved.
That's cool. Did you have one of those experiences where you're like, oh, there's no way I can help out with this
tech support stuff. And then you can get there and you're like, I'm actually able to help quite a bit.
Yeah. You learn some skills. So we had one major system that we supported and then a couple other smaller
software. And I never really learned how to use myself. So you learn some tricks like,
so walk me through the exact steps you've taken so far, you know, and then you sort of follow along
with them and you're like, have you tried clicking this? This looks promising. Let's try clicking on
this and see what happens. So how funny. Yeah. Tech support was a great first job out of college. I
learned a lot, helped me with my creative problem solving skills. And I really learned the software
well to the point where after a few years of tech support, I would send the developers the line of code
where the bug occurred and say, you know, here's the bug that the user reported. So they asked me to
come work on the development side. And I did that for a little while. And then I was still struggling
with my confidence. It wasn't really a great fit. And I couldn't quite figure out why I didn't have a lot
of support. This is before a lot of the meetup culture. This was 2012. There are a lot. Now I found
there are a lot of communities for helping people get into tech. But DC. Yeah, the communities have really
grown a lot in the last five years or so. It's a lot, a lot nicer in that. Absolutely. Yeah. And DC
wasn't really a tech hub. It wasn't really known for being much of a tech hub. So I had a lot of self
doubt. And I wasn't sure that being a developer was right for me. So but I while I was working at this
company, it was a company that helped campaigns file their campaign finance disclosures, the FEC. So
political campaigns in the in the US need to disclose money that they've raised and spent towards
their election. This is if they privately done it or through the government support or something
like that, right? You have options. FEC provides free filing software. And it's everything you might
imagine free government software might look like and we're working on that. But there are also third
party companies like a like equivalent of like a turbo tax for filing campaign finance disclosures.
And so that's what that company did. And I just fell in love with all the rules and the campaign
finance. It's sort of like super nerdy, like anyone who gets excited about filing their taxes and all the
different scenarios. It's very similar amount of rules for who can give how much under what circumstances.
There are situations where individuals can gift goods and services to campaigns, but there are
restrictions on those types of gifts. So I really just enjoyed that world campaign finance. Yeah.
Even though I had been a political science major in college, I actually hadn't learned that much about
the FEC and campaign finance laws and regulations. And it was kind of before campaign finance was a
hotbed issue. It can sometimes be a hotbed issue now, but this was not something we covered a lot in
class. It's not something that was getting covered a lot in the media. So it was through work that I
learned about FEC disclosure rules and campaign fundraising, whereas before I had really learned more about
voting. So the fundraising is what allows the running of the campaign to take place.
Yeah. So much, at least in the U S is about running ads and hiring people to go out and knock on doors
and all sorts of stuff like that. Right. So even though really you'd hope it's kind of some sort of
meritocracy and you debate and then you choose the ideas, there's monies involved.
Yes. It has become a hot, hot bed topic in a lot of ways, but I think at the time I was learning about it,
it was the money was often thought to be an indicator of support. So if you could raise a lot
of money from a lot of different people, it was an indicator that you had widespread support and that
those folks were, even if they were just giving you $20 towards the campaign, they would help you
support, like have office space. And like you said, print ads.
Right. It's some sort of, if you're going to give money, you're likely you would vote for them,
right? Giving money is a bigger signal than actually voting. So it's an early signal, I guess.
Absolutely. So I wasn't sure being a software developer was right for me. So I went to go
work for the FEC in a less technical role where I worked to help the FEC with more of the enforcement
side of things, making sure that the campaign finance reports were filled out properly, that if
people had questions about how to comply with the law, if there's any disparities or anything that was
unclear on the reports, I would help clear, like send correspondence on the public record, asking the
regulated community that, you know, the campaigns and the PACs and parties to, you know, further
clarify their reports. And there was an element of tech support in that role. I also did tech support
for the free filing software and anyone who's done tech support, sometimes I can get pretty,
like answering the same question over and over again, no matter how much you care, it can,
can sort of wear on you. So I was looking for a challenge.
I'm so happy to have your question. I'm really here to support you for the 100th time.
Yes. And this is the first time they're asking, you know, so you have to, you know,
Yeah, of course they, it's not there. They don't know.
Right. And there are a lot of very confusing rules. And some of the best advice I got from my
supervisor was you don't have to defend the agency. You know, so if someone says, this is really
confusing, you can say, it is confusing. Let me see if I can help you work through it instead of,
you know, trying to defend the regulations for, for, and so after about five years of doing that job,
I wanted a challenge, I wanted to change. And there was an opening on the website project team
for fec.gov. And my mentor at the time had recommended that I come work on this project.
It was a really cool project. And I can talk about why it's cool. And I said, Oh, I haven't written a
line of code in five years. You know, there's no way I can work there. And I met with the team and
they were willing to, if I was willing to work hard to get up to speed, they were willing to work with me
since I had a lot of knowledge about the area and past experience. So basically did a crash course
in Python. And with a lot more support this time around, there are all these organizations in DC for
underrepresented genders and tech and just new developers. And so really did a deep dive and
kind of was able to talk myself onto the team and, and was drinking from the fire hose and really just,
yeah, really had fun though. It was super fun.
That's cool. You know, one of the things that I hear in your story here, that sounds a little bit
different between the two experiences, you're going to college and getting your minor in CS versus,
you know, being really passionate about the politics and elections and then learning code to sort of
advance that mission. Those are really different in that, you know, so often in computer science,
it's like, well, here's some random algorithm. We're going to study the algorithm. Then you can
reproduce the algorithm and how boring and dry and separated from like what you're interested in.
is that right. Whereas this is like, well, I've got to go through this technology to use it as a,
like a way to really achieve the goals I'm after. And I think those are just such different experiences.
Would you agree?
Absolutely. I distinctly remember one of my projects in college was to build a calendar.
And I was like, I have a calendar. You have to remember like,
got a paper one, I got a digital one.
Yeah. And so I just wasn't very passionate about some of the projects and they're good projects for
teaching the concepts, but not necessarily that can be difficult to find. And I think there's been
a reevaluation of how certain ways of writing software is taught a little more beginner friendly.
If you have a goal and you have something you want to build and you know why you want to work on it,
well then it's super exciting. But if it's just dry, I think we lose a lot of people who don't have some
sort of personal drive that's going to get them through that is not necessarily part of the lessons or part of the
official experience, you know?
Absolutely. And the thing I wanted to add to that is that I didn't learn at the time I was in,
in school, I was studying computer science and I've sort of figured out that software development and computer science
can be somewhat different disciplines. And I never learned how to really write software in college.
And what I've learned the hard way is to really start small, get the smallest piece working,
change the smallest part of it. And can you make that even smaller? Maybe just hard code something for now.
Whereas before I would write in college, I would write top to bottom, write the whole thing out,
and then it would just explode about a semicolon buried on line 463.
So those learning those skills has helped me made it a lot more fun, a lot less miserable.
Yeah, fantastic. So I think you've given us quite a bit of background already about the FEC,
FEC Federal Election Commission. But I think two things are interesting to talk about real quickly here.
One is this whole agency was born, I guess, just a tiny bit before I was actually, but it was born out of a financial
scandal in 1972, right?
Yes, after the Watergate scandal.
Yeah, I bet people have heard of that one.
I hope so.
Yeah, I would hope so too. The other thing is, you know, you look at the FEC and you said it's not a
huge organization of the government, but how many people work there?
We have about 300 full-time employees. That's federal employees. So they work for the US government.
And for on FEC.gov, we have a team of about, you know, we have a cross-functional team, but we have about nine developers and a user experience designer who work on the technical side of the site.
Yeah, cool. It's a pretty small group of developers. Probably all work pretty closely. I'm sure it's a fun place to be.
It really is.
Yeah. And before we move on too far from the opening, Kim Van Wyk, who was one of the Ask Me Anything hosts for me? Hey, Kim.
Says, I strongly suspect that also helps you remember a software topic six months later if you did something you found interesting while learning.
Absolutely.
Yeah, I would definitely say so.
Absolutely.
All right. So as we get into the code side of things, maybe we could talk a little bit about the type of data to give people a sense of what we're working with here.
So one of the things you can do is you can go to FEC.gov and you can go to all data and there's all kinds of stuff you can learn about raising money, spending money, candidates.
I could go over here. I could select my state, Oregon. I could go pick my location around Portland and pull that up.
And it'll do things like, I could go up and say, learn about the house, I guess.
And it'll show this person in this party raised $64,000 and had $110,000 disbursements, how much cash they have on hand right now.
So apparently $711,000, which is kind of insane.
And then you can go see the sources for this data. Tell us about the kind of data and how people use it there.
Yeah, that's a great, our election profile pages. I'm really proud of those.
We can see who's running in each election.
Yeah, it's beautiful. And you get like a colorful map of the areas.
You just click on it and it pulls it right up. It's really neat.
Thank you so much. And as you were demoing it live here, it was looking snappy, which just really warmed my heart.
It's very funny.
So when we think about filing our taxes with the IRS, in the U.S. at least,
we have to disclose how much money we made and various other, basically the money we made in our taxes.
It's all summary level data.
Campaigns need to file the, not only the summary level data, but also the transaction level data.
So from every contributor who's given aggregate of $200 or more in a given election,
they have to disclose their name and address, employer occupation, and the details of that transaction.
And so it's a large volume of data.
Especially at the presidential level or national level.
Yes. Yes. Presidential level, I mean, like hundreds of millions of records.
I think we're on track to have a total of 1 billion records by the end of the 2022 elections.
It's just a lot of data. So it's not just how much they've raised. It's the individual contributions that go into that.
So from here, if you wanted to click on the name of the candidate you're looking at for your district,
it'll take you to the candidate's profile page. It looks like this is a House candidate and they run every two years.
So there's a browse receipts button so you can actually see the individual contributions that made up.
And it's 2022. There hasn't been a lot of data yet for 2022 because it's early in the election cycle.
Oh my gosh. So I'm in here looking at this and it says, here is Diane Snow, who donated how much?
You know, $50. These are individual contributions and all sorts of stuff, right? This is crazy detail.
Yes. And we've had detailed, we, as part of our mission to make detailed contributor data and not just contributor,
but expenditure data, how the committees are spending their money since back to the late 70s.
So we do have all the historical data here to explore. You could export this to Excel spreadsheet or technically it's a CSV.
But at the top of the page, you could filter for like all the money that's been received for 2022.
So you could eliminate your, there are the breadcrumbs there with the little X's on them.
You could, you know, just maybe that one candidate across all time or all the money for 2022.
And so it is for all the data is, it is again, like about coming up on, we had around 500 million transactional transactions last in the 2020 election.
That equals about four terabytes of data.
And the expectation is it is filterable. It's searchable. It's exportable.
I think user expectation is that you'll be able to get the information quickly.
So we've spent a lot of time focusing on performance improvement for the site, which is powered by our campaign finance data API under the hood.
This portion of Talk Python to Me is brought to you by Square.
Payment acceptance can be one of the most painful parts of building a web app for a business.
When implementing checkout, you want it to be simple to build, secure, and slick to use.
Square's new web payment SDK raises the bar in the payment acceptance developer experience
and provides a best-in-class interface for merchants and buyers.
With it, you can build a customized branded payment experience and never miss a sale.
Deliver a highly responsive payments flow across web and mobile that integrates with credit cards and debit cards,
digital wallets like Apple Pay and Google, ACH bank payments, and even gift cards.
For more complex transactions, follow-up actions by the customer can include completing a payment authentication step,
filling in a credit line application form, or doing background risk checks on the buyer's device.
And developers don't even need to know if the payment method requires validation.
Square hides the complexity from the seller and guides the buyer through the necessary steps.
Getting started with a new web payment SDK is easy.
Simply include the web payment SDK JavaScript,
flag an element on the page where you want the payment form to appear,
and then attach hooks for your custom behavior.
Learn more about integrating with Square's web payments SDK at talkpython.fm/square,
or just click the link in your podcast player's show notes.
That's talkpython.fm/square.
There's a whole bunch of APIs, and you can even check out the code on GitHub, which is super interesting.
This is the kind of data that people can get, and I think it's pretty good.
So one of the things that's interesting to me is if I'm over here on fec.gov,
this looks like a pretty well-designed website.
It looks nice.
You can jump around.
But then as you were showing me those different locations,
it looks pretty data-driven.
There's a decent amount of logic behind it.
You said that this is built on Wagtail.
Is that right?
So when you go to fec.gov, it's mainly two applications.
The content management system we use is built with Wagtail.
So there are kind of three different buckets on the site.
There's the campaign finance data, which we were just discussing,
which is mostly powered by the API.
We use some AJAX calls, just basically JavaScript and data tables
to display the data that the API provides.
And this application that we're looking at is the Django-based CMS application.
Right.
And so for people who don't know, maybe just tell them really,
what's the elevator pitch on Wagtail?
What is this thing?
Wagtail is a really great admin for the Django framework.
So Django has a built-in admin framework.
So just to kind of give some context,
the disclosure, the help for candidates and committees section of our site
has information for our regulated community.
So if they want to find out, can I take this money from this source?
Or what are the limits on contributions?
They can go to help for candidates and committees and find the answers to their questions.
And so that's stored in the Django database.
Well, it's stored in a Postgres database that our Django application connects to.
And Django as a web application framework has a built-in admin.
So, you know, if you're working on this, if you've worked on like the Django tutorial or anything like that,
you can go slash admin and use the built-in Django admin to manage your content.
So if you wanted to modify an article or, you know, you're working on a blog or whatever, that's how you do it.
Wagtail is an open source tool that is, that has more user-friendly functionality on top of the existing Django admin.
A little closer to like WordPress-ish kind of?
Yeah, it's got some great features.
You can do custom templates.
I don't work as, I work more on the data sides, but we have a content team that uses Wagtail to manage the content on the site.
It seems like it's very customizable and just has more features than the Django admin that I'm sure you could build yourself.
But this is a free open source project that allows you to have a more robust admin interface.
Yeah, a lot of times that's a big platform decision people have to make.
Like, oh, well, we want to have this admin thing that people will let people go in and just developers don't have to add the informational pages
and just maintain the basic data of the site.
So I could either build something in Django and then it's kind of on me or we got to go through the admin backend in a clunky way
or I've got to build something or we go WordPress.
But now you're out of Python and you're in this other Joomla WordPress world that you're not really able to extend
and it just becomes almost a static site.
And so things like Wagtail are cool because they let you stay in your world if you're a Python developer,
but then also have a nicer experience on top of it for the users.
Yeah, Wagtail seems like it's been great for us.
Yeah, very cool, very cool.
Okay, so we've got that.
And then we've also got the OpenFEC API is what I'm trying to say here, which is RESTful API around all this data that we've been talking about, right?
Tell us about this.
Sure.
So if anyone hasn't worked with APIs before, the API is just the source for all the campaign finance data.
So one of my friends put it in a cute way, we drink our own champagne.
So any of the data you see on FEC.gov is coming from the API.
So you don't need to know how to use an API.
You never need to touch the API.
If you don't want to, you can go to FEC.gov and get all of the data that you want out of this.
That's really cool.
I've heard the term dog food, eat your own dog food.
Drinking your own champagne sounds way better.
It probably means the same thing, right?
Way more pleasant, I imagine.
Yes, absolutely.
When we were interacting with the homepage, the all data section, that was JavaScript talking to this as well.
Just so you're basically the first client of this API.
Correct.
Exactly correct.
Okay.
However, if the FEC.gov doesn't have the information you need or you want to build your own tools on top of the API, we do make that available to you directly.
So you can bypass FEC.gov and go directly to the source to get the data.
How old is this API?
How long has it been around?
We realized around 2014 that the legacy systems we were running were going to have a really big problem with our exponentially growing data.
So in 2015, we partnered with 18F, which is a government organization within consulting for government by government.
And we wanted to build an open source cloud-based application that had an API.
So we actually launched the API before we launched FEC.gov.
So we've been drinking our own campaigns since around 2015.
It's been a party for six years.
Awesome.
So I've heard of 18F, and it sounds like these are some really tech-savvy folks that can drop in and help the other developers on other organizations or divisions in the government and say,
look, you want to modernize what you're doing.
Let us come spend six months or a year, and we'll help you get going and then hand it off to you.
Yeah, I was hugely, hugely fortunate to have overlapped with 18F when I joined the project because I, again, joined the project, having done a crash course in learning Python, but not having had a lot of other skills that were needed to be successful in the project.
So I was able to have really, really excellent mentors that helped me learn.
In a lot of government agencies, this was our first cloud-based application.
I think hopefully I'll get a chance to talk about cloud.gov, which is the platform that we run.
Yeah, yeah, absolutely.
But this is all new to us.
And so 18F was really wonderful.
They taught us a lot.
They taught us how to build software in an agile way, how to practice open source, and not just share your code, but work in the open.
So if you go to our code repositories, you can see what I'm currently working on.
All of our issues, all of our tickets are public, and you can see the priorities and how things are going.
And, oh, there's my change, where I was tweaking our workers for a G event.
And actually, I'm thinking of moving to a different mechanism for our asynchronous requests.
And you can follow along with my research.
And if you have opinions about workers versus threads for G-Unicorn, I think, I don't know how, I call it G-Unicorn.
I say G-Unicorn, some people say G-Unicorn.
To me, I don't know, it's like a unicorn with a G.
So if you have opinions on this, please chime in.
All those issues are public.
I've done some research on pros and cons of the different configuration.
But, yeah, 18F kind of led the way on how to work in an agile way, in an open way.
And we were super lucky to be able to work with it.
Yeah, this is really neat.
So I'm sitting here looking at github.com/fecgov slash openfec.
And this is the GitHub repository for the API, right?
Correct.
And it's not some forked off, like we'll dump to it every now and then.
Like you said, this is where you all work.
Yes, this is where we deploy from.
We use CircleCI.
They have a really great open source tier.
So when we merge the develop branch, it deploys to CloudDive the develop space.
So this is our source of truth for the code.
Nice.
And it looks like you're using sort of feature branch style PR development, which feels modern
and fresh to me.
Another thing that's interesting is if I come down here, we've got our requirements.json here.
And we've got Flask.
We've got Flask RESTful and Flask SQLAlchemy.
This looks like a fun tech stack.
Thank you.
Yeah, I've enjoyed it.
I gave a talk at FlaskCon, which I think was the first time they ran it, I guess about a
year ago, that goes into a little bit more technical detail about how we use Flask for the API.
Yeah.
Super interesting.
One thing I'm noticing here, there's Flask 1, 1.1.1, which is great.
Last week, I guess last week, maybe five days ago, two came out.
They also have new async plans going on.
Are you all thinking about that already or are you just still putting out features for now?
With our data doubling every two years and we are down a couple of staff members, we're
sort of in the hopefully hang on until next year and hopefully we'll get some more money
plays.
If it's not broken, we do have regular dependency checking for security issues.
Of course, yeah.
If there's an issue, we do update it.
Basically, we're not in the place to take that on quite yet, but I did hear the news and
I'm excited to see what that might help us accomplish.
Yeah, absolutely.
You've mentioned cloud.gov.
I think that's a pretty interesting place.
Let's talk about a couple of other things.
Let's talk about this communities of practice at digital.gov.
You talked about working out in the open.
Let's talk about some of the things going on there that I think are pretty interesting.
What's the story of these community of practices?
I'm here.
I can see that there's an AI section and there's a blockchain section and the cloud and infrastructure,
which is getting close to cloud.gov stories and DevOps.
What is this place?
Yeah, so one of the things that surprised me about coming to government is how, so FEC is
an independent regulatory agency, which means it's not part of one of the big cabinet agencies.
And so kind of surprised me.
We're all sort of doing similar things and we sort of struggle to talk to each other about,
you know, what's working, what's not working.
It's not always clear who you might talk to just at other agencies about ideas and suggestions
they have.
We're pretty small potatoes compared to a lot of the bigger agencies and we have a lot
to learn from them, but I don't know how I'd feel about just sort of cold emailing someone
I've stalked on LinkedIn or something like that.
It can be hard to connect to other people working on very similar problems.
Yeah, because you could hear that maybe the people in, I don't know, IRS or something,
they're doing something cool and you would, because it's kind of related to what you're doing
in some sense.
It'd be neat to see how that might work for you all, but you don't know those people, right?
Like the government is huge and it's like a hundred thousand, I'm sure more than that,
a huge, huge company that many people have never met.
They're not close to each other, right?
So this is a way to set up some of those communities around ideas.
Is that right?
Absolutely.
Yeah, it's really wonderful.
And we've talked to people at the CFPB.
They use, I know they use Wagtail and we've sort of traded ideas about what's working well
and what's some of the challenges we're facing.
But the communities of practice are kind of a formal way of bringing people together.
And I've really enjoyed the DevOps community of practice.
And they have a monthly short half hour tech talk where they invite people to talk about
the types of things they're working on and successes and challenges.
Yeah, super neat.
Then another one was this digital services playbook.
So I think the digital services is something I just learned about a little while ago, a few
years ago, I think.
I had, oh gosh, I'm sorry, I don't remember the guest's name, but I had one of the developers
on from the US Digital Service, which I thought was a really interesting, just a really interesting
organization that I hadn't heard about.
Tell people what that is real quick.
And then what is this digital services playbook?
I think this seems pretty neat.
Sure.
So I'm not affiliated with the US Digital Service in any way.
However, I have met people who work there and I've always been super impressed by them.
And they have a similar mission to 18F.
And there are people out there who have written blog posts about the difference, but they're
through the executive office of the president.
And they do have a similar mission of helping modernize government technologies.
I think one of the biggest challenges in everyone I've ever spoken to who works in government is
contracting.
So if you were lucky enough to have staff of federal employees, you can have some more flexibility
around how you do the work.
But if you are, there are some advantages and disadvantages to contracting, but it can be very difficult to
get right.
And so one of the resources I often point my acquaintances towards that are trying to
modernize their technology is the digital US Digital Services Playbook.
And it goes through how to approach your modernizing your technology.
It talks about best practices for contracting and kind of gives you a roadmap.
And another resource I'd like to touch on in a second, there's a statistic that I think like
80% of all custom built software projects in government fail.
And I think that contracting is a really hard thing to get right.
It's sort of like, I imagine outsourcing for the private sector.
It has risks to outsource because it just has trade-offs.
And the amount of management and oversight you might need to have for a contractor or an
outsourced project might differ from if it's in-house.
So this is a really great, the Digital Services Playbook is a really great resource for other
federal employees or people working on federal projects.
And then also just anyone looking to modernize their systems might find this is helpful as
well.
Yeah.
And they've got 13.
Apparently, they're not superstitious, which is cool.
So they've got 13 digital service plays.
So things like that are pretty, I guess, pretty standard, but also good, you know, understand
what people need, make it simple and intuitive.
But then stuff that maybe people within government don't necessarily, they wouldn't necessarily
think about like default to open or, you know, choose a modern technology stack and things
like that.
You know, default to open, for example, right?
You might think, well, this is data about people and it's super sensitive.
That's one reason to keep it hidden.
I mean, obviously what you're giving away is supposed to be public at FEC.
And then another is, you know, for example, like if you share your source code, maybe people
who want to do bad things with it will come and, you know, use that to find some way to,
I don't know, do something that they shouldn't, right?
We've seen a lot of security in open source as well, right?
By having multiple eyes on it.
Absolutely.
And I think building in the open from the beginning is the best way to successfully
have an open source project because retroactively open sourcing that has been built using the
assumption that you'll have protection through obfuscation can be really risky.
And unfortunately, if you're building a closed source system, it can make it a little easier
to have some bad practices like hard coding credentials.
I think there can be some misunderstandings about how open source works.
Like no one can come in and just change the code for fec.gov.
They need to submit a request, which is reviewed by one of our team members and gone through
a security review as part of that process.
So it kind of open source sort of encourages best practices because you might think that you're the strongest network security that ever exists.
If someone could be phishing is, I think, one of the biggest weaknesses that organizations face.
And so I think that's sometimes called zero trust.
Yeah.
So assume they're going to get to the code either way.
So you might as well use best practices for securing your code.
And the best way to do that is from the very beginning, assume that you're going to have eyes on it.
Talk Python to me is partially supported by our training courses.
Flask is one of, if not the most popular Python web frameworks and developers are adopting it for the smallest and many of the largest Python based websites and APIs.
If you want to learn Flask, we built a fantastic course called Building Data-Driven Web Apps with Flask and SQLAlchemy.
In this course, we built a PyPI.org clone from scratch using Flask and SQLAlchemy.
You'll learn many of the major ingredients needed to build most web apps.
If this sounds amazing, just visit talkpython.fm/Flask or email us at sales at talkpython.fm.
Before we move off this digital services, Kim out there live stream says, David Holmes was your guest.
Thank you, Kim, for that.
Yes, I remember that now.
And says, as a non-U.S. citizen, he's from South Africa, do organizations like the FEC use the U.S. digital services or is it all in-house?
It sounds like you kind of are separate from them.
We partnered with 18F back in 2015.
Again, they're sort of like consulting for government by government.
18F rolled off our project in 2018-ish.
And we haven't partnered with the U.S. digital service.
I think they choose their projects that can be influenced by the executive office of the president.
So we haven't worked with them, but I've met, we do have an exchange of ideas.
If we're lucky to go to PyCon, for example, and meet other people who work there, we can talk about what's working well.
And so that is super valuable to be able to have that sort of relationship with them.
Yeah, fantastic.
One other resource that I think you were hinting at is this de-risking guide from 18F.
Is that right?
Yes.
I'm a huge fan of this.
They have a federal and non-federal version.
So if you're a state, that's a whole nother thing in the U.S. that we have to deal with where we have the federal government and the state government.
And just like all the federal agencies are out there replicating similar efforts, all the states are replicating similar efforts with their own budgets and their own challenges.
And in some ways, if we're looking at things like security, we have a disadvantage because we're splitting up our resources into many slices.
So I love that 18F also built a non-federal guide to help with the state and local governments as well.
Yeah, I think this also applies to large organizations that have maybe been around a while.
But here's a stat from government.
It says only 13% of large government IT projects succeed.
And implementing custom software projects can be extraordinarily costly and risky in a government setting.
And here's the big thing.
The waterfall software development remains the standard at all levels.
So in budgeting, in management, in rollout, all those things, which is very different from here's my PR.
Let's push that with CI to cloud.gov, right?
Yeah, absolutely.
I feel like someone said recently, government contracting is really designed to buy tanks.
And building software isn't really very, doesn't have a lot in common with buying a tank.
And so, yes, it is difficult to get right.
And this guide has some really practical recommendations for how to manage the process.
And one of my biggest things I'd like to highlight is the Quality Assurance Surveillance Plan, or QASP.
That's mentioned in this guide.
You can build it into your contract as a way to measure whether the quality of the software is acceptable.
It can help you as the person who's managing the contract and help your contractors understand what acceptable levels of quality are.
So, oh, it looks like the QASP is highlighted there at the top in one of the menus.
Oh, yeah.
There you go.
And they have a sample QASP, which is really cool.
And so, if you scroll down a little bit there on this page, you can see some example quality measurement ideas.
So, using an automated tool to check for accessibility and security vulnerabilities, you should be able to deploy with a single command following OWASP best practices.
Their tools, ZAP, can be used to check for vulnerabilities.
Doing user research at every sprint.
These are just really great recommendations.
Oh, yeah.
How neat.
And there's a little, how do you do this measurement of what is the acceptable quality level?
How do you measure it?
And so on.
This is really neat.
I like it.
Huge.
I'm a huge fan.
Yeah.
Cool.
Now, let me pull up one thing that I thought was pretty interesting here.
And I'm seeing this, excuse me, I'm seeing this all over the different sites that were.
I'm seeing it on, you know, the fec.gov and so on.
If I click here, it says right at the top, an official website of the U.S. government.
How do you know?
And it says, well, it's a .gov.
And then it talks about this little lock.
And if I look at the lock, it says this is all done through Let's Encrypt, which Let's Encrypt is awesome.
You know, it's a way to basically automate getting SSL certificates.
But it feels like very modern and cool and hip.
And to see the government rolling that out, that was kind of a surprise.
And I feel like this ties a little bit back to our next topic, which is cloud.gov.
Absolutely.
Yes.
The first thing I want to point out is the banner you mentioned at the top of the page is part of the U.S. Web Development Standards, U.S. Web Development Design System.
It's an initiative to bring the look and feel of government websites in line with each other so it can help build trust.
And users can be assured that they are in the right place.
If this is the first time you've heard of it, they have some really great resources.
U.S. W.D.S.
And I might be getting the acronym, like what it stands for slightly wrong.
But they have components you can use on your site.
It's all open source.
Web Design System.
There you go.
Thank you.
And so you can see on their website, they have an official website of the U.S. government.
Here's how you know at the top of their page, just like we do.
They even have versioning, like version 2.11.2.
Yeah.
This is a really great resource as well.
It's all open source.
So even if you aren't in government, if you wanted to borrow any design, it's a design system.
And people who know more about design systems would know more about what that means.
But we do use parts of that in our site.
But you mentioned Let's Encrypt as part of how we manage our certificates.
FEC.gov is run on cloud.gov.
And cloud.gov is a platform as a service that's available to government agencies.
And so when you build a cloud-based system, you have the option of building your own cloud
infrastructure in AWS or Azure, Google Cloud from scratch.
Or you could use a platform as a service.
And there are pros and cons, trade-offs of each approach.
But we find as a small agency, cloud misconfiguration is one of the leading causes of security vulnerability.
Like security breaches.
Yeah.
How often do we hear about here is a open server running in the cloud without authentication?
Or here is 10 million records of sensitive data in a non-secured public S3 bucket because
it was a pain to deal with security, to integrate that with the app or whatever, right?
These are real problems.
Absolutely.
And the cloud itself is very secure.
But you need to make sure you're configuring it properly, take advantage of those security
features.
And so that's not easy to do.
Or I haven't found it easy to do.
I always say like they give you enough.
Well, I probably won't use that phrase, phraseology, but it's very powerful and difficult to get
right.
Yeah.
I feel like AWS and Azure, those types of places, they're very powerful, but you go there
and it's just, you know, deer in the headlights.
So many choices.
Just even at the, what area do you want to go into?
And then you factor, well, what are the security implications of changing this switch down in
some level?
It's crazy.
Yeah.
I won't even sign up for a free tier account because I'm convinced knowing me, someone's
going to get access and mine Bitcoin.
I'm going to have a $10,000.
Like they make you put your credit card in there and I'm not doing that.
So it takes a lot of expertise to build your own infrastructure in the cloud in a secure
way.
Our agency being smaller, we could go ahead and do that, but cloud.gov has a lot of advantages.
And so what it does is it sits on top of AWS as a management layer.
And so it manages the cloud security.
It manages authentication.
You know, who has access to what it makes it super easy to deploy, really easy to keep
our infrastructure as code.
You can use infrastructure as code in AWS, but I still haven't really figured out how to do
that.
It seems wolf again.
So many haven't, I find it no matter how many trainings I take, I still find it difficult
to work with.
So cloud.gov has a lot of advantages over building your own infrastructure.
It has fixed pricing.
Again, like if you mess something up and are running a machine for too long and you get
a surprise $10,000 bill, like that doesn't happen in cloud.gov because they have cost
engineers and they have fixed pricing.
In government, if you want to build a technical system, you have to have an, it's called an
ATO, an authority to operate.
And it's all this paperwork that you need to do in order to prove that your system is secure
enough.
By using cloud.gov, you can inherit from their ATO and kind of save yourself a lot of paperwork.
You just put the checkbox.
I'm using cloud.gov.
I mean, essentially, they, I mean, you mentioned we use let's encrypt and my first reaction
was cool.
I guess we use let's encrypt.
Awesome.
Yeah.
They manage a lot for us.
They make it super easy to deploy.
We have scalability through cloud.gov and we can focus more of our time and energy on
the data and application side of development, which in and of itself is pretty time consuming
for us.
Yeah.
So maybe you could talk about two things.
Maybe you could talk about what does your deployment look like?
I think you're using AWS at the moment.
Is that right?
And then you're using Postgres.
And maybe just talk about how you're running all that, the APIs and stuff out there.
Sure.
So application architecture, we run our application tier on cloud.gov and our database tier is
actually, we have our own managed AWS Aurora Postgres that we run because while cloud.gov
does offer database services, our database is so massive that we wanted, and we have really
experienced DBAs who wanted more granular control over the settings on the database.
Yeah.
You talked about how you were in a place where if you wanted to scale horizontally by adding
more read servers, read replicas, that was a redeploy of your app and a big challenge.
And moving to Aurora made that sort of automatic behind the scenes, right?
Yes.
I did.
And I can probably put this in the show notes.
I did a talk on how we use Aurora for database scalability.
So AWS, RDS, Postgres, just kind of plain, I think it's like relational database service,
basically managed database service.
When you have read replicas, so at some points we need to run like three to four read replicas
equivalents.
I can go into like the difference there, but we need to run on numerous machines at the same
time on the database side in order to meet our demand.
I suspect that this data is very read heavy, the way it's being used.
Well, it is very read heavy, but we also have the data is always changing.
So we have some really big filers.
That's true.
If you're doing like a big campaign, people are every little contribution is streaming in,
right?
Yep.
And there's some big packs that'll have like 30 million records, 40 million records we need
to insert all in one night.
So we do have a lot of, it's pretty insert heavy as well.
But in the past, when you use AWS RDS, each read replica had its own connection string.
We are using the 12 factor framework for application development, which is a few is that's published
online and basically considered best practices for cloud based application development.
And it recommends using your attached services connecting to them through connection strings.
So basically, you should be able to swap out your database for one database to be able to swap out to another.
So this is backing services number four there.
Treat backing services as attached resources.
So our application connects to the database through environment variable connection string.
And when we were using plain RDS before, each replica had its own connection string.
So in order to scale, we could have set up auto scaling on that database side, but we would have had to come up with a fairly fancy way of consuming that new connection string, adding it to the environment variable, the application, and then redeploying the application to reflect that new connection string.
So basically, it wasn't functionally, it wasn't very feasible to us for us to have database auto scaling.
And after doing some research, we decided to move to AWS Aurora Postgres, which has one connection string that you connect to, and it handles auto scaling on its end.
So it doesn't call them replicas, it calls them instances, but they're functionally replicas.
So we connect to one connection string, and it could have two instances running.
And then if it has high CPU, it'll spin up some more machines to handle the database side.
Oh, cool.
So you can set up like auto scaling rules and things along those lines?
Yes.
Okay.
So that's the machine under the hood for the API.
Yeah.
And during one of your talks, was that presentation that I linked to, was that one given to the community of practice group?
Is that where that was?
It is.
Yes.
It's all coming together, I see.
So I'll link to that as well.
But during that talk, you showed some neat CLIs for deployment and configuring scale and things like that.
And it was really cool.
You go over there and say, hey, I just want five more web run-ins.
Boom.
And you have them, right?
That was quite neat how that works.
Yeah.
Cloud.gov is based off of Cloud Foundry, which is an open source platform management tool.
And so it can be a little bit difficult to run this on your own, but there are managed services that run Cloud Foundry.
And one of the big ideas, as I understand it, behind Cloud Foundry is making the developer experience really easy.
So you can deploy with a single command, CF push.
So we use on Cloud.gov all the Cloud Foundry commands.
And with one command, you can scale up the number of instances you're running your application.
And it's practically immediate.
I mean, I would say under three seconds to quadruple your number of machines.
Yeah, you made an interesting comparison to that about your previous model with physical servers.
Yeah.
So our legacy system was run on physical servers.
And it was really difficult to predict how much we were going to need.
Sometimes we'd overbuy.
Sometimes we'd underbuy.
It could take three months to get a new server set up.
And now it takes three seconds to get a new instance running.
In addition, when we moved to Cloud.gov, we saved about $1.2 million a year in infrastructure costs, which was really big for us as a small agency.
That's a huge thing.
And I believe one of the numbers you threw out there previously is $1.4 million was around the infrastructure costs before.
So to go from $1.4 to $200,000, that's a huge difference.
It's not like, well, it's $100 million and it went to $99 million.
It was much, much better, right?
Percentage-wise.
That's a great point.
Yeah.
And we were able to, again, like my big, the thing I love about Cloud.gov is not taking on that risk ourselves.
It's both the security risk and also just the risk of misconfiguring something and something going down.
If for a small agency like ours, it's really huge to be able to depend on them for that.
Yeah.
And you don't have to have a team of network and DevOps engineers just to keep it going, I suspect.
Like that's the cloud.gov side and they just deal with it.
Yeah.
I think when I talked to them, if we had taken all of this in-house, the amount of services that they offer, we would have needed to hire like six or seven more people.
And we have a team of nine.
So it was just not cost effective for us to bring that infrastructure in-house at this time.
All right.
So let me ask you a question.
I've been thinking about this as we've been talking about all these things, seeing all the cool stuff that you're doing, seeing how 18F came in and helped and all of those.
Now, I don't necessarily expect you to have an answer to this, but I just want to hear your thoughts.
So the rollout of the Affordable Care Act and all of the drama around the websites and the challenge of scaling that and stuff, do you think it would be different?
I think it would be smoother if like this world existed in a more polished way and we tried to roll it out again in the same way?
I think that there are a lot of challenges in developing and delivering digital services.
I think a lot of the expectations for digital service delivery are high.
You know, it's like, well, Google can find this for me.
And if you compare the operating budget between my agency and Google, it's like, yeah, Google just talked about rolling out like their own supercomputer for this other thing.
Right.
That's like a different set of capabilities or whatever.
Totally.
I would say to me, it all boils down to as boring as it sounds.
It boils down to contracting.
It really depends on how the contract is written and how it's managed.
And I don't know very much about it, but I do know that if you don't get it right, it's very difficult to course correct.
So absolutely, we can do better.
But there are still a lot of challenges to preventing those sorts of issues.
Yeah, of course.
All right.
Well, I think that's probably all the time we have to dive deep in this.
But this has been a super interesting look at what's happening over there.
And, you know, honestly, it's heartening to see how much the government is advancing and using modern tech and really putting these systems in place.
So well done.
Thank you so much.
It was a pleasure to speak with you today.
Yeah, you bet.
Now, before you get out of here, I got the two final questions.
So if you're going to work on the API, for example, write some Flask and Python code.
What editor do you use?
I use Sublime.
And VS Code does package management better, I think.
But I just can't make myself move.
And notable PyPI package.
You know, maybe something cool that you use during the project that you're like, oh, people should really know about this.
It doesn't get enough press.
Sure.
I'm a maintainer for Flask SQLAlchemy.
So it's a really great package.
Community has been super welcoming.
And it's the only one I've been able to really get my hands on.
So if you're looking for a project to work on, all the Flask projects are great to attribute to.
Oh, fantastic.
Tell people real quick, why would you choose Flask SQLAlchemy over just SQLAlchemy?
I think it depends on getting started.
It's pretty friendly to get started.
And you can maybe start with that and then reevaluate what sort of custom settings you might need.
Yeah, cool.
All right.
A couple of comments from the live stream.
Antonio says, no, don't go.
I really enjoyed the chat.
And then Brandon says, super interesting subject.
Thank you.
Yeah, thanks.
Thanks both for being here.
Final call to action.
You know, people broadly or maybe even people within governments around the world who want to do better work with modern tech stack.
And what do you tell them?
Check out fec.gov.
The link to our GitHub is in the footer.
We look at all the feedback as public.
Anything you can get out of our open source project.
Take a look at who's running for office.
And we just love it if you got involved in our project.
Very cool.
Well, like I said, well done.
Very nice work.
It seems like when I was working with the site, it was super snappy and beautiful and polished.
So excellent.
Thanks for being here, Laura.
I'll pass that along to my team.
Thanks so much.
Yeah, you bet.
Thanks.
Okay, bye.
This has been another episode of Talk Python to Me.
Our guest on this episode was Laura Buford.
And it's been brought to you by Square and us over at Talk Python Training.
With Square, your web app can easily take payments, seamlessly accept debit and credit cards, as well as digital wallet payments.
Get started building your own online payment form in three steps with Square's Python SDK at talkpython.fm/square.
Want to level up your Python?
We have one of the largest catalogs of Python video courses over at Talk Python.
Our content ranges from true beginners to deeply advanced topics like memory and async.
And best of all, there's not a subscription in sight.
Check it out for yourself at training.talkpython.fm.
Be sure to subscribe to the show, open your favorite podcast app, and search for Python.
We should be right at the top.
You can also find the iTunes feed at /itunes, the Google Play feed at /play, and the direct RSS feed at /rss on talkpython.fm.
We're live streaming most of our recordings these days.
If you want to be part of the show and have your comments featured on the air, be sure to subscribe to our YouTube channel at talkpython.fm/youtube.
This is your host, Michael Kennedy.
Thanks so much for listening.
I really appreciate it.
Now get out there and write some Python code.
Thank you.