#187: Secure all the things with HubbleStack Transcript

00:00 Michael Kennedy: How do you keep track of your security, configuration states, and even out of date system level packages in your servers? Well, what if you had 40,000 or more servers to manage? How would your process scale then? I'll tell you, mine would take a few tweaks. On this episode, you'll meet Colton Myers, who's a co-creator of HubbleStack. HubbleStack is an open-source security compliance framework. It provides on-demand profile-based auditing, real-time security event notifications, alerting, reporting, and yes, Colton's group has over 40,000 servers, and HubbleStack is watching over all of them. Learn about this cool Python-based framework on this episode of Talk Python To Me. It's Episode 187, recorded November 14th, 2018. Welcome to Talk Python To Me, a weekly podcast on Python, the language, the libraries, the ecosystem, and the personalities. This is your host, Michael Kennedy. Follow me on Twitter where I'm @mkennedy, keep up with the show and listen to past episodes at talkpython.fm, and follow the show on Twitter via @talkpython. This episode is sponsored by Linode and Rollbar. Please check out what they are offering during their segments. It really helps support the show. Colton, welcome to Talk Python. Let's get started with your story and how you got into Python.

01:25 Colton Myers: I've been using computers since my parents got a black and white Macintosh. I don't actually know what it... I should go back and figure out what Macintosh it was so I can tell people, but I would play Lode Runner on that when I was little. I'm what, 30? Yeah, 30, I can never remember. So, I'm younger than a lot. I guess I'm starting to get into the middle of the pack these days, but yeah, I remember playing with that when I was like seven on this little black and white Macintosh. Then we had like a Windows 3.1 machine that I'd play games on, and I mostly played games until high school. And then in high school, I actually, half of my senior year, I did this applied technology course in computer programming, which was really great actually.

02:03 Michael Kennedy: What language was it?

02:03 Colton Myers: It was C#. They had just switched from C++ to C#.

02:07 Michael Kennedy: Oh, that's a pretty big improvement, actually.

02:09 Colton Myers: Oh, yeah yeah yeah, I'm with you on that, and C# is a great language, and back then, I was Windows, Windows, Windows, you know, cause I played video games, right? So, and it worked out just fine, and it was a really good course, and it really gave me like a jumpstart and from there on, I knew that's what I wanted to do, so went into college.

02:26 Michael Kennedy: That's awesome. Let me ask you more about that, cause that sounds so awesome. This was a high school program?

02:32 Colton Myers: Yeah, yeah, it was like...

02:33 Michael Kennedy: And how long did you get to spend programming each day?

02:36 Colton Myers: Half the day, like, you could either do morning, they had a morning session or an afternoon session, and so like basically at lunchtime, you would switch either back to your high school or to the biotechnology center, and it was difficult because you had to like structure your whole schedule around it, like you had to start planning. You could technically do it as a junior and a senior if you like really planned, but then you'd have to, like... I was also in like band and stuff like that, and I'd have to have dropped that, and I was like no, I'm just going to do it as a senior, but you still have to like make sure that you have all your other things to graduate, you know, in the rest of your schedule, so.

03:06 Michael Kennedy: Your four years of math, or, you know, three years of English, whatever, right?

03:09 Colton Myers: But it was a super cool program, and our professor, I think he's now working for Microsoft or something, but he also used to teach at university. He was really good, and he did a great job of teaching us the right way, and I don't know, man, it was a really good introduction to programming for me.

03:27 Michael Kennedy: I'm so jealous, when I was, this wasn't high school, this was my last year of middle school, I think, I'm pretty sure, but it could have been the same in high school, my teacher, I took sort of intro to programming course, right, and my teacher was, it's kind of okay, and I was just really loving it, and I said, oh, I'd love to write this kind of program, and like, how can I learn more? And they're like, well, you can do that, but you've got to understand, like, real programming is actually super hard and I don't know if you really can do it. Like, I mean, that was my idea, like, okay, great. I'm like, oh well, that's awesome.

03:57 Colton Myers: I don't know, I'm blessed, programming just clicks with my brain, like it just is, it has never been a struggle for me to learn how to program, like, you know, none of the conditionals, none of the advanced logic, like obviously I struggle with, like, advanced algorithms like anybody does, but like, the basics of programming just came to me immediately, and so, and my teacher actually recognized that, and it was super nice because he, like, had me helping other students, which was frustrating as much as it was, but it was useful, right, like, I've always been a bad teacher because, with stuff that comes so easily to me, because I can't bridge that gap, but he had me helping other students, but then he also like had me doing some of like the second year curriculum, along with the first year curriculum, because he was like, like I would just finish so early on these projects and he'd just be like, well, you can either just sit here and do whatever you want or I can get you started on this other stuff, and it was a real blessing, like, to have that start, and to have somebody so supportive.

04:48 Michael Kennedy: Yeah, I've never heard of anything like that. I've heard of like the vocational stuff, like shop, and, you know, carpentry, but never software. So, aw man, that's awesome for you. All right, so that was C# back in the day, and Windows, and somehow your project is not in C#.

05:05 Colton Myers: No, so, college, I went straight into computer science. I knew exactly what I wanted to do, and I managed to make a couple of friends there that were super useful because I went to the University of Utah and at the time I went, it was still there, they would start in C#, and then you'd go to C++, there was a lot of Java involved, like through all their classes, but they didn't, like, Python hadn't at that point really pierced the, you know, the education, which is a whole different conversation as to whether we should actually be teaching Python as a first language, I almost think it's almost doing a disservice because it's so easy to program in Python, but I don't know, it's a hard conversation, but I've made some good friends who basically, we kind of pushed each other to explore outside of what our classes had, so like some classes you could choose language, and most people chose C# or Java because that's what we were learning, right? But we decided to try Python at one point, and it was really awesome and we were using Git long before they taught us how to use Git, and all these things that really like helped jumpstart that, and helped me get a base in stuff that maybe I wouldn't have otherwise, and found Python, I got my laptop stolen in, while I was on vacation in Hawaii, and I had a good friend at work who convinced me to buy a Mac, instead of a Windows machine, which was a great decision, I really like macOS in general, especially because it's just Unix underneath, and so, you know, that got me into more of the Unix side of things, and kind of away from Windows.

06:30 Michael Kennedy: And you're like, hey, this C# stuff doesn't really work so well, there's Mono, but it kind of is clunky over there, and so, now what, right?

06:37 Colton Myers: Well, once I found Python, like, like Python, you just get rid of so much of that boilerplate like Java and C# have all this boilerplate and it's just so verbose, and Python, like, any time, like, Python was really interesting because, like, I would be like, oh, I need to do this, I need to like open a file or whatever, and I'd be like, okay, so I'm betting it's like, this is the function name, and turns out, most of the time it was right, like it just, like it felt like it was written straight out of my brain and it was just so nice, but yeah, like with C# back then, you couldn't really write C# in Linux, like Mono was just getting off, just getting started, and so I didn't really, yeah, I basically avoided it wherever I could from then on.

07:18 Michael Kennedy: That's awesome.

07:18 Colton Myers: Yeah, it was really great.

07:20 Michael Kennedy: Yeah, and of course, you know those other libraries didn't have nearly the package management story and the ecosystem story that Python does.

07:25 Colton Myers: Sure, yeah, well, and then, the way, it's really interesting because I just feel like, I don't know, I just have, like, there are so many things in my life that, if they changed it a little bit, I could have been on such a different path, right, and I think that's true of all of us, but like, when I was getting close to graduating, I was on a local Python user group, and a local startup advertised on that user group, hey, let's, we're looking for Python developers, and I was like, well, I'd love to do Python, but like, my internship that I'm working on is in C# and C++, right, I'm working at this, you know, on this legacy codebase in this DOD contractor that's local or whatever, and I was like, I don't have any experience, like, I've just written some Python for like class projects, and they were like, well, come in and interview, and I was like, okay! And apparently I interviewed pretty well. They were impressed, so they actually hired me on part-time, until I graduated, with the expectation that, assuming I did well, you know, they'd hire me full-time, and it worked out, and I, that's how I joined SaltStack, so I worked for SaltStack for three years, and really, like, got my Python, you know, really solid and you know, was able to expand my ops knowledge, you know, because it's a configuration management tool.

08:35 Michael Kennedy: Yeah, it's not just the Python you got to learn, right? It's also all the Linux and dev ops side of things.

08:41 Colton Myers: Yeah, so that was like, that was hugely important to my career because, without that, like, I don't know, I could've found something, I'm sure, as like a junior Python dev, but it just really jump started it, and actually directly led to my position at Adobe, too, so. Yeah, that's kind of where my career path has been.

08:57 Michael Kennedy: No, that's a super interesting story, so, speaking of where you are today, you know, so, what do you do day-to-day at Adobe?

09:03 Colton Myers: So, I'm a senior software engineer at Adobe, and I actually mostly work on Hubble. I joined a, actually, the other creator of Hubble, it was kind of his brainchild, but he's not really a developer, so he was able to, like, you know, say hey, I want to do this proof of concept, can I hire somebody for it, and he hired me. We had met because he was actually in the SaltStack community, and his name is Christopher Edwards, by the way, I should give him credit, but this is his baby, and so he hired me to do this, and the idea was to base it on Salt and basically, we were trying to replace an existing vendor solution, with this proof of concept because the vendor solution wasn't very popular and was very expensive, and so yeah, we were able to, so he brought me in on this and it became just, it almost overnight, like we wrote this proof of concept, and it was good, right? And people liked it and all of a sudden, it was like, oh, hey, other people heard about it, and they're like, hey, by the way, our contract with Cloud Passage is up next year and we're losing this promotional pricing. It's going to get like three times more expensive, so, could you maybe make it so that Hubble is everywhere at Adobe within like, you know, the next six months? Like, that'd be great.

10:12 Michael Kennedy: You know, no pressure or anything, right?

10:17 Colton Myers: Oh, it was crazy, it was a crazy crazy time, but we were able to, we had an open source from the beginning, which was super awesome, and because it's grown so much inside of Adobe and is so important to our security posture, it has, it's my full-time job, I get paid full-time to write open source software in Python, so, you can't really go wrong with that.

10:35 Michael Kennedy: Yeah, I feel like that's a lot of Python developers' dreams, right there, like, I want to work on this open source thing I love and I want to get paid full-time for it, like, there's not that many people that get to do that.

10:45 Colton Myers: And I just like fell into it, like I, it's so much, yeah, I don't know. I don't even know how to approach how lucky I am that way, but it's been really great, so yeah, that's kind of what I do, I just work on HubbleStack.

10:58 Michael Kennedy: Okay, super, so, maybe let's start with a super high level conversation about this, just even, not just HubbleStack, Hubble and HubbleStack, but just this idea in general, so what it is is an open source security compliance monitoring and auditing system, like, tell us just sort of about that world, like, what kind...

11:18 Colton Myers: So many words.

11:18 Michael Kennedy: What kinds of things, yeah, I know, yeah, so just tell us, you know, what's that world like and what kind of role does this try to play in there?

11:26 Colton Myers: So there's a couple pieces, right? We, security has multiple facets, and one of those facets is compliance, and really, the reason compliance is so important, I'm going to put air quotes around important because the reason compliance is so important is for these certifications, right? If you deal with customer data, especially like credit card data, and that kind of stuff.

11:45 Michael Kennedy: PCI, all that.

11:45 Colton Myers: Things like PCI, right, exactly, you know, you might have SOXS compliance, you might have HIPPA, right, all these different things that if you can check that box, you can have customers that you might not otherwise, right? And each of these has different requirements for compliance. A lot of them are a lot less fleshed out than you'd expect, most of them, like, they just say, like, the auditors come in and the audit control says something like, you have a baseline security compliance check thing, that you audit against regularly, and make sure that everything is compliance, right, and it's just like, written like that, and so, at Adobe, we use the CIS compliance checks, Center for Internet Security. They are a nonprofit that publishes these profiles that allow you to, if you follow all these checks in here, then you'll be more secure than if you didn't. That's the idea, right?

12:35 Michael Kennedy: Right, so does it include like, known vulnerabilities, or is it more like guidelines on, like, hardening of the server or what's...

12:41 Colton Myers: Yeah, it's more along the lines of hardening, like I don't think CIS, I don't know if they actually do, but I don't think they publish like a vulnerability list or a CVE list or anything like that. It's more like, hey, do you have, root login enabled on your server, right, in SSH? And if you do, that's a problem, right? Do you have a password list login? Do you have a password login even, right? Like, or is it just SSH, like, you know, those are easy ones to point out for SSH, but it's also like, hey, do you have Telnet installed, right, do you have all these different things that are just like, you know, do you have FTP, or are you serving anything via FTP as opposed to SFTP, right, like, just the gimmes, right? And obviously, they get a lot worse, like, or, a lot worse, a lot more stringent, like, oh, you need to have SSH timeouts on your host set to these low values, right, so that it quickly kicks people off if they're not typing and that kind of stuff, and those, you know, obviously as you get further and further into these checks, these different levels, they actually split them up into levels, it becomes less convenient, right, because security is always a compromise between convenience and security, right? So we basically choose subset of those, and we audit against them, and that is actually where like the big money is, like that's where we get like customers, right, but if I'm being honest, it's not actually making us a lot more secure. Like, obviously, if we have root login enabled, we should not, right, but like you fix that once, but that doesn't mean you're secure now, right? And so the other piece, the rest of Hubble, is just to collect other data that our blue team can use to watch for actual, like, ongoing attacks, big holes that maybe an audit wouldn't have caught otherwise.

14:16 Michael Kennedy: Right, so give me an example. This is something I wonder all the time, if you don't know that's fine, but, give me an example of what an attack looks like, like if I log into my server, it seems okay, you know, nobody's saying, hey, our data seems to be gone, but like, how do I, you know, share to write like stolen but how do I look at the server and know, outside of, say, weblogs?

14:39 Colton Myers: Yeah, I mean, that's kind of a hard question and it's actually not my expertise, like I'm not a security engineer. But I work with them a lot. One that's pretty common in 2018 is if somebody gets access to your server they might install a cryptocurrency miner on your server, right? And so, you know, one of the ways we can look for that kind of stuff is oh, you know, in one piece of Hubble we collect, like, the established outbound connections on your host, right? Because certain services have outbound connections, like that's a thing that they should have, but a lot of them don't. A lot of them just accept inbound, right? Especially if it's just like a web server, right? It's accepting inbound connections but not reaching out. So if we see an outbound connection from a process we don't recognize or an uncommon process, then we might look into that and say hey, what's going on here?

15:25 Michael Kennedy: I see, that makes a lot of sense.

15:26 Colton Myers: Yeah, I mean, there are a lot of, like, tells. You could have increased CPU usage or like, you know, stuff slowing down obviously. They're trying not to leave tracks, right? So hackers will tend to try to not let you know that they're there. So it's hard, it's a hard problem. And that's why we have a whole security team, we have the team I work with just with the digital marketing, they must have 30 security engineers both, like, red team and blue team just constantly combing this data, looking for patterns. And we have a whole security operations center, looking for these alerts and it's a big thing because it's harder and harder, you know, in 2018 to keep your stuff secure.

16:03 Michael Kennedy: It definitely is. So maybe just define some terms for folks who are not super up-to-date on the security side of things. Red team is the penetration testers and the folks trying to break in, the blue team are people like using Hubble to try to say, you know, wait a minute, why is it making all these connections over this VPN to someone we don't recognize.

16:23 Colton Myers: Yeah, exactly. The red team is trying to break it, they're the hackers, they're the white-hat hackers, right, that say we're just going to spend all day trying to get into Adobe's hosts and then if we find a way in, then the blue team can patch that hole. And they actually have, like, these whole things where red team will set up these different things and make sure the blue team catches them. Like sometimes red team will work with access to the box and just say, let's assume I get access to the box. I don't have a way right now, we've patched all those holes, but let's assume and I'm going to just set up these breadcrumbs and make sure that blue team is catching them. Because blue team says they're watching for these things, but are they actually watching?

16:55 Michael Kennedy: Yeah yeah yeah.

16:56 Colton Myers: And then it's this back and forth, and then blue team, yeah, is trying to catch red team, but then they're also trying to catch the actual attackers because people are always trying to get into our data, right? It could be really lucrative. And we've had so many hacks that we see in the industry, and Adobe doesn't want to be next on that list. So we have a substantial team.

17:16 Michael Kennedy: Those headlines! This portion of Talk Python to Me is brought to you by Linode. Are you looking for bulletproof hosting that's fast, simple, and incredibly affordable? Look past that bookstore and check out Linode at talkpython.fm/linode. That's L-I-N-O-D-E. Plans start at just $5 a month for a dedicated server with a gig of RAM. They have 10 data centers across the globe, so no matter where you are, there's a data center near you. When you want to run your Python web app, host a private git server or file server, you'll get native SSD on all the machines and newly-upgraded 200 gigabit network. 24/7 friendly support, even on holidays, and a seven day money back guarantee. Do you need a little help with your infrastructure? They even offer professional services to help you get started with architecture, migrations, and more. Get a dedicated server for free for the next four months. Just visit talkpython.fm/linode. Do you think containers and Docker and things like that make this harder or easier to protect?

18:24 Colton Myers: It's a combination. One thing that we've been struggling with is the, Docker is 12-factor apps, right? Like, are you familiar with the 12-factor thing?

18:32 Michael Kennedy: Yeah, but maybe give the listeners a quick, just a super quick rundown.

18:35 Colton Myers: Oh, what is the site? There's like a site, 12factor.net, with like one-two-factor-dot-net. And basically it's like these systems of things that like make, let's read their introduction. Software as a service, 12 factor app is a methodology for building software as a service inputs that have these different things. And really, it's where containers come from and where these microservices come from. Like you want everything to be immutable, and like easy to set up fresh. Instead of changing a host when it starts to misbehave you just burn it down and create a new one.

19:03 Michael Kennedy: Don't embed all the configuration and information, put that, like, in the environment so it's separate from, say, your source code and your app code and whatnot.

19:11 Colton Myers: And there are some great advantages to that, because if you start with an image that like, for example, in auditing. If you start with an image that has all of those audit checks already passing, then you're just starting from a much better place, right? And you can make sure that everybody is starting from this hard and secure image. And then, past that, when you're not messing with hosts, you have a lot fewer snowflakes and you have a lot fewer inconsistencies across your host, in theory, right? And really, where the attackers get in is in those snowflake servers that have kind of been forgotten about. Sometimes they get in via dev, you know, whatever.

19:43 Michael Kennedy: Nobody wants to upgrade it because it might break the thing. That person used to maintain but they left the company and no one wants to make it their problem, so they don't touch it.

19:51 Colton Myers: But it has problems, too, like for example the established way to pass secrets into a container via the 12 factor method is via the environment variables. And security engineers don't like that, they don't. They don't like secrets in environment variables.

20:04 Michael Kennedy: Because as soon as you're out of the box you can just, you know, go into the environment variables and see all the logins and stuff, right.

20:10 Colton Myers: Exactly. And theoretically, you don't have the ability to log in to the actual container, right? Because you don't ever need to. And so, theoretically, and then of course you're using CoreOS or something that is theoretically more secure, right? So like, there's frameworks around that to try to prevent that from being as bad of a problem as it is. But it still is, you know, a hard thing that we're trying to deal with. So I do think 12 factor and containers and immutable images and those kinds of things do have a lot of advantages. And probably the advantages override the disadvantages. But the other problem is that not every use case fits into that, right? We have 40,000 some odd servers across our analytics product, and like using those, it's just not, it would take a huge rewrite of that product to make it a microservices product kind of a thing. So it's going to take many years, and even then, there are certain applications where just being on bare metal is just better, more performant, et cetera. So it's never going to be, like, the only thing that we do. I don't think, I don't know, I'm not going to try to guess. But I do think from a security standpoint it is better overall.

21:18 Michael Kennedy: Yeah, interesting. I think it probably is as well. It's easier to set one baseline and just create the images from that, but still I think it also, it seems like things change more often, and someone could flip a switch in that image that you don't realize or in a base image somewhere down that chain, and you know, all of a sudden everything is like that, not just that snowflake server, right.

21:38 Colton Myers: But that's also easy to undo when you find it, right? Because again, you just switch it on a higher level and the next time they build their image they get that, right. So I don't know, it's, but yeah.

21:47 Michael Kennedy: A bit of a diversion. But if you don't mind, tell us as much as you're happy to share, just a little bit about, you talked about 40,000 servers, something like that. When I think of Adobe, I think of Photoshop, and things like that that are more apps on like a desktop or a tablet or something. Give us a sense of what a cloud and server world looks like over there.

22:12 Colton Myers: So I have less of a sense on the, so we have two major business units. We have our digital media, which is what you referred to, the Photoshop, et cetera, et cetera. And then we have our digital marketing or digital experience as it's been rebranded recently, and that actually started from an acquisition of a company called Omniture here in Utah, and it turned into, like, Adobe is one of the big players in marketing analytics. Like they do everything from like helping you, helping customers see what's working as far as advertising on their websites to helping serve social media stuff. There's a whole suite of stuff that you can get from Adobe. And it's much less known, even though it's a pretty big business unit, and that's actually, on the digital media side, we have servers, right? Because we have now creative cloud, right. And we have to serve all that data, we have to, you know, you can keep your files in our creative cloud, all that stuff, right?

22:57 Michael Kennedy: Right.

22:58 Colton Myers: But it's much bigger on a digital marketing side because we're doing this software as a service for all these big companies, yeah. So on the digital marketing side we have something like 120,000 servers or so across a whole bunch of teams, you know we just acquired, Adobe just finished acquiring Marketo, which added a whole bunch of new infrastructure to our that we have to worry about.

23:18 Michael Kennedy: So how does that kind of stuff sort of shape what you're building and what Hubble is? Where somebody outside of the company says oh, we just acquired this other company, and now they use this technology and here's all their servers.

23:32 Colton Myers: It changed a lot. It affected us a lot. Livefyre we acquired a few years ago, and that one was a big one because they did a lot of autoscaling in AWS and, like, our original Hubble was built on top of Salt directly, right. You would deploy it as a series of modules inside of Salt. And that worked well, not nearly as well as today's product. But it worked well. But then we get to Livefyre and it's going to be prohibitively expensive to actually deploy SaltStack everywhere because what they use, they use AWS a lot like containers, right? They use AMIs, they use images, right, and they don't really modify their host. They don't really have a good configuration management story in place because they don't need to, right? They use it like containers. So what we basically realized that we needed a product that could be stand alone, that did not rely on Salt. As this went wider and wider, a lot of teams at Adobe don't use Salt. And we couldn't necessarily force them to, or it wasn't going to be prudent, right? So we created Hubble, which we use PyInstaller to compile so that it basically, we do our best to compile everything into one directory, all of our dependency, our packages have no dependencies except glibc which we can assume is going to be on a Linux host. It has no dependencies, it ships with its own Python, it doesn't mess with existing Pythons, it doesn't mess with existing Salt, it doesn't mess with anything. And that was hugely advantageous because now we can just say, hey teams, all you need to do is just install this package, put the configuration in place once, and restart Hubble and all of a sudden you were getting all of your reporting. And we're not adding a whole bunch of new stuff to overwhelm your Salt masters, right? We talk directly from those hosts to Splunk, which is the tool we use to collate all those results, and it turned out to be just way better. So eventually we, oh and by the way that version of Hubble does still use Salt, it just uses it as a library, it just imports Salt, and we were able to slim it down and make it so much more targeted and work so much better and make the onboarding story so much better that we eventually deprecated Hubble-Salt and just focused on the standalone.

25:38 Michael Kennedy: Right, so Hubble-Salt was the original one where it was just completely assuming this is going to be built on SaltStack, right? And now you kind of,

25:48 Colton Myers: Yeah, you just add it as like a remote in your SaltStack Master and it just pulls down the stuff, so.

25:53 Michael Kennedy: Right, right, but with all these acquisitions and disparate infrastructure, it's like, okay, well, that's not working so let's not require that, let's just do this straight Hubble. And what I'm really impressed with is your ability to take this and deploy it, no Python dependency on the machine, no pip install -r requirements, dependency on the machine. Just here is a binary, run it. Like that seems just golden for Python.

26:18 Colton Myers: Oh it's amazing. Like I, yeah, pip is great. But if you talk to SREs they don't like pip in general. They want to install via system packages.

26:30 Michael Kennedy: Or why wouldn't they want to just download arbitrary code from the internet and run it? That's so paranoid.

26:35 Colton Myers: pip has gotten so much better but it still, like, it has some uninstall issues, like these days it's a lot better than it used to be but yeah, like, they just want to do it as a package 'cause that's what they're doing with everything else, right, they just want to do RPM or Deb, like, that's it. So it's been really advantageous to do that and we also use, we cheat a lot, we actually don't use proper Debian standards for packaging, we just use FPM because FPM is amazing, just makes it so easy to package for any operating system. And it's just been really good, and it was so much better, like even if you're using Salt, like my team uses Salt. And it was like, upgrading Hubble-Salt was much more of a pain to get consistent across our infrastructure than just doing this package. And so I immediately switched us to Hubble instead and I was like, this is so much better. So we're actually in the process of completely phasing out HubbleSalt, it's mostly gone out of Adobe's infrastructure and Hubble is king, so.

27:31 Michael Kennedy: Is HubbleSalt still a thing in opensource or are you trying to push everyone over to Hubble outside of Adobe as well?

27:37 Colton Myers: It's not a thing, well, if somebody wanted to fork it, it's apache too, go for it. But we have stopped maintaining it, I put a note in the ReadMe, and said we're deprecating this. I encourage you to try Hubble because it's just better, it's just a better story for users.

27:54 Michael Kennedy: So obviously, I don't know if we said it explicitly, but Hubble is written in Python.

27:58 Colton Myers: Yes, yes.

27:59 Michael Kennedy: So that's great, but I do love that you don't have to, as a consumer of it for your servers, you don't have to care that it's in Python, you just take the thing and run it. It's great the way it's in Python for developing it, but I just don't feel like pip is the right answer for consumers and users in general. It's cool that you guys made that work.

28:22 Colton Myers: Yeah, we like it a lot.

28:24 Michael Kennedy: Yeah, so, when I go to the Github organization, I see three things at least. I see Hubble, and HubbleSalt, those are the two we talked about, right?

28:31 Colton Myers: Mm-hmm.

28:32 Michael Kennedy: And then there's also HubbleStack Data. Now, do you want to maybe tell us about that?

28:36 Colton Myers: So the idea behind Hubble is that we, because upgrading is expensive, even though it shouldn't be, even though it should just be an RPM update, right? It's expensive to work with all these teams to upgrade because they're busy with their own stuff. So the idea was to try to make it as much as possible that we could safely change the information that we're gathering from a source that security engineering controls.

28:58 Michael Kennedy: Right, so this is something like, new vulnerabilities or new things to check for?

29:03 Colton Myers: Exactly. So we find out that we need to check for this new file or whatever, we want to watch this file that we haven't been watching or we want to add this new audit check or whatever, we don't want to necessarily have to deploy new code to the servers. But on the other hand, we also don't want to give ourselves the ability to run arbitrary code on the servers. It's this compromise, right? We work as hard as we can to give ourselves as much flexibility in data gathering, but we also work as hard as we can to make sure we can't modify the server from HubbleStack Data, and that any changes to HubbleStack Data will be safe. So we have an agreement with our customers, which are other users at Adobe, that we won't push code to that HubbleStack data repo, and then we also will notify them and do it with a change request, a change management request inside of Adobe whenever we merge to, like, the master branch. So we treat it almost like a code deploy even though it's not, and that way we mitigate the risk and we do testing around it before we merge those things and that has just allowed us to create trust with our customers that we're not going to, that Hubble's not going to mess up their servers. HubbleStack Data is where all of the profiles, is the word we've chosen, that basically tells Hubble what data to collect. And inside of there we have a couple different components inside of Hubble, I guess I should have given like a high-level review earlier than this, probably, we're like 35 minutes in here, sorry.

30:22 Michael Kennedy: That's alright.

30:24 Colton Myers: Like I mentioned, Hubble does a couple things right. It does these audits which are success/fail checks against like, say, CIS or Red Hat STIG, any of these different compliance standards. And the code name for that is Nova. Hubble, you know, is named after the telescope. The idea's we are looking with our telescope into the black hole of our infrastructure, right? We hope it's not a black hole, but sometimes it feels that way.

30:45 Michael Kennedy: But it can be vast, right? I mean, 40,000 servers is not an obvious thing to do.

30:52 Colton Myers: It's crazy. And again, yes, most of those servers are fine but it's the edges that are hard and those edges are the hardest things for SREs to keep track of, right? So everything is space named. We may be moving away from that in the near future, just because it's really hard to keep track of. Like if I say Nova, you're like, I don't know what that is. But if I say audit, you can say oh, I can get an idea of what that is. So Nova is the audit piece, and that's all success/fail stuff. It's built on this, you basically write these YAML profiles in kind of an ugly format, we're going to hopefully improve that in the near future. But it works well, just returns, you know, you can see all the failures. And then we then can ticket in Jira the teams that need to make changes based on those failures. And then we have Nebula, which is kind of our data gathering segment of Hubble. It primarily uses osquery, which is a really cool opensource project by Facebook, and basically osquery allows you to query your system as if it was an SQL database.

31:51 Michael Kennedy: Right, I've seen that, that is amazing.

31:55 Colton Myers: Yeah, it's really cool! You can query anything from like network connections to install packages to, who knows, all sorts of stuff. It's crazy. It's actually developed, they fixed for a while, Facebook, we had some drama with Facebook around their, it was like a patents clause in like their projects and it was a bigger deal for Angular, is it Angular that they do, or is it? I don't remember what JS project it is.

32:19 Michael Kennedy: It's React, I think.

32:20 Colton Myers: React, thank you. And it's this big deal because if Facebook ever ends up in litigation with your company then this is revoked, you can't use our thing. So it's this big thing that a bunch of companies were like, we've got to get React out of our infrastructure because of all of this stuff, like, the lawyers were like no, it cannot be there! Anyway, Facebook has fixed that since, including with osquery, so it's been good. So yeah, it's a really cool product, and we use that to just collect terabytes of data every day from our hosts. Everything from, like, the easy gimmes like established outbound connections, to like, all the packages installed in the system to all the docker processes running, the shell histories, we just collect all this stuff and shove it into Splunk and then our blue team can sift through it and try to find actionable insights. That's not what makes us the money, but that's where the blue team is super excited about the data. That's the data that's useful to them.

33:12 Michael Kennedy: Right.

33:13 Colton Myers: And then we have Pulsar, which is our file integrity monitoring, our FIM. We use iNotify on Linux and MTFS Journaling on Windows, by the way Hubble runs on Windows.

33:22 Michael Kennedy: Yeah, I was going to ask you that. So obviously it runs on Linux, but it also runs on, like, Windows server? Mac, macOS?

33:29 Colton Myers: It could run on Mac, there's nothing stopping it from running on a Mac, it runs great, we don't have any audit profiles that are Mac specific in HubbleStack Data yet, and we don't have, like, packages and like all the key list stuff for services.

33:43 Michael Kennedy: Yeah, I mean, there's very few servers running macOS but there are some, I think, like some of these build systems.

33:50 Colton Myers: Well and theoretically, like, some people would like to use Hubble for, like, laptops and that kind of stuff.

33:57 Michael Kennedy: Like desktop management, for like, I want to know what my machines are up to.

34:03 Colton Myers: And it could be useful that way, it would take some work to make it more user friendly, like you can't just, like this is meant to be run on the CLI and it doesn't have a UI at all, you know what I mean? It would take some work, potentially, but I've tried to push back on that because I don't think it's useful. But anyway, so Pulsar.

34:20 Michael Kennedy: Okay, yeah, keep going on infrastructures.

34:24 Colton Myers: Yeah, Pulsar FIM, we basically just collect file changes on the host, and you can have it like, send checksums on the new files and that way you can compare, you know, does this match our white list of checksums for this specific binary or whatever. We don't have a good answer on that one for, like, changes that are supposed to happen, right? There's no, I don't have a good way to turn off our monitoring during, say, a configuration management push. Like obviously I could say, oh, if you lay down this file, then it'll say oh, that file is there. Let's stop collecting for a minute. But then an attacker could lay down that file, right? So there's not, like, we haven't figured out a secure way to like, a Salt master to tell Hubble to stop collecting because Hubble doesn't accept incoming connections, it only shoves data out and pulls data from Github or from S3 or from wherever else. And so those are the big pieces, the other code name is Quasar, which is, I don't know if it qualifies as being its own section. It's technically just the returners in Salt, that's the SaltStack terminology. But basically that's the blue that allows us to send the data to a destination. We support Splunk, that's what we use in Adobe, we also support Elk and Logstash, so, that's what most of our opensource users are using, I think, because Splunk is very expensive. Yeah, so that's kind of the high level overview of the pieces of Hubble.

35:41 Michael Kennedy: This portion of Talk Python to Me is brought to you by Rollbar. I've got a question for you. Have you been outsourcing your bug discovery to your users? Have you been making them send you the bug reports? You know there's two problems with that. You can't discover all the bugs this way, and some users don't bother reporting bugs at all. They just leave, sometimes forever. The best software teams practice proactive error monitoring. They detect all the errors in their production apps and services in real time and debug important errors in minutes or hours, sometimes before users even notice. Companies like Twilio, Instacart, and CircleCI use Rollbar to do this. With Rollbar, you get a realtime feed of all the errors, so you know exactly what's broken in production. And, Rollbar automatically collects all the relevant data and metadata you need to debug the errors, so you don't have to sift through logs. If you aren't using Rollbar yet, they have a special offer for you, and it's really awesome. Sign up and install Rollbar at talkpython.fm/rollbar, and Rollbar will send you a $100 gift card to use at The Open Collective, where you can donate to any of the 900+ projects listed under the opensource collective, or to the Women Who Code organization. Get notified of errors in real time, and make a difference in Opensource. Visit talkpython.fm/rollbar today. Alright, so give me an example of, maybe, how once this data's collected, like, what kind of questions would you ask or, so I'm not familiar with Splunk, you said this is not an opensource project, it's like a paid commercial thing that's kind of pricey?

37:13 Colton Myers: Yeah, so Splunk is, how do they?

37:15 Michael Kennedy: What do they call themselves?

37:17 Colton Myers: Describe themselves, search, analyze, and visualize the machine data generated by your organization. So the idea behind Splunk is it's kind of the same story as, like, an Elkstack. You have the collection, you know, with Logstash you have the coordination with Elasticsearch, that, sorry, not coordination.

37:34 Michael Kennedy: Query ability with.

37:36 Colton Myers: Yeah, correlations, I guess, is what I meant to say.

37:38 Michael Kennedy: Yeah, yeah, yeah, okay.

37:39 Colton Myers: With Elasticsearch, and then you have Kibana for, like, reports and stuff. And Splunk does all those things. You can do these elaborate searches and correlations across the data and then you can turn that into alerts or you can turn that into dashboards and they have machine learning aspects that try to help you do some of these security things without having to hire your own machine learning team. They're still young, but it's pretty expensive. I mean, I think your first few gigabytes are free or whatever.

38:13 Michael Kennedy: But in reality.

38:13 Colton Myers: It's basically billed by the amount of data that you send to it, and it can get out of hand pretty quickly.

38:20 Michael Kennedy: So you save your company money by only upgrading your servers, like, once a year, things like that?

38:25 Colton Myers: Yeah, I mean, I don't deal with the Splunk stuff. They just tell me I can send as much data as I want to them, basically.

38:31 Michael Kennedy: Send it here, and they go do what they do, right?

38:33 Colton Myers: Right, and so yeah, we send terabytes of data to our Splunk clusters. Like, the gimmes, I keep going back to this establish outbound connections. Because it's such an easy thing for people to see, like, if something is calling home that shouldn't, that's actually pretty easy to spot in general, right? Obviously, on something like, what's a good example of something that reaches out?

38:54 Michael Kennedy: Well, like, let's talk about, say, a webserver, right? So it's got like a uWSGI or a gunicorn worker process, but that worker process is probably talking to a database, maybe it's talking to like a Stripe credit card API or S3 bucket or something like that. I mean, that thing would be reaching out, maybe you've got, obviously, all the upgrades, right? If you're going to upgrade, just, apt upgrade, that's going to be reaching out. But there's a few, right?

39:23 Colton Myers: Those are easy to, like, limit to a specific white list, right? Like you know this server needs to reach out with the database servers. And you know what all the database servers are, so you say okay, in this search I'm going to eliminate all of those, we don't care about those. You know it's going to be reaching out to the package servers, right? And theoretically, hopefully, you're rehosting those package servers, right? So you know so you can keep that list, you don't have to whitelist every mirror in the world, right?

39:46 Michael Kennedy: Right, or maybe your cloud server is something like that.

39:49 Colton Myers: And then all these other things. But then past that, like, the only thing that it accepts from any location is incoming connections, right? On 80 or 443 or whatever, right? So if you create a search such that it says, hey, we know about all these different things on these servers that are good reaching out or okay reaching out, but if we see something from a new process that isn't in our whitelist or from an existing process, because somebody could say, oh, I'm apache, too, right? And if you see something from an existing process that is reaching out to somewhere we don't know about, then we send an alert to the security operations center and they contact the team and say, hey, is this normal? Is this something we should add to our whitelist? Should we look into this? You might find that you have a cryptocurrency miner on there or something. So that's like the easy one.

40:38 Michael Kennedy: Yeah, so you start with, let's just see what connections are being made. Alright, these look valid, hide them, don't worry about those. And then you just slowly either go through till there's none on your list, or you start to get worried and figure out what the abnormal ones are.

40:54 Colton Myers: Exactly. And it's a hard problem, right? Like I mentioned, we have like 30 security engineers and it's because every time you put something on that whitelist, you've given yourself a blind spot. So like, you still, ideally, we would have some crazy AI that could go through and be like, hey, let's investigate this data and it could notice things that we couldn't, you know. And that's something that Adobe is working on, it's something many companies are working on. Machine learning and AI is huge everywhere. But it's a really difficult problem and one that I'm not super qualified to talk about because it's not what I do all day.

41:32 Michael Kennedy: Yeah, no worries. So what we've spoken about so far, to me, sounds like one a configuration of the machine, and two, oh crap, something is bad here. And at that point I feel like it's not necessarily too late, but you're way far down the road of it's not good. If there's a process connecting to something else on your server and it really is malicious, like that's a long ways down the path to not good. So one of the things that might be really helpful is to just say, I'm using this version of JDango, or that version of Requests, and it turns out that there's a critical security problem found in it, we had better upgrade that before someone else figures it out.

42:11 Colton Myers: Right.

42:13 Michael Kennedy: Does it support searching for that kind of stuff?

42:14 Colton Myers: Yeah, so we don't actually use it for that in Adobe, we have a different tool that they've been using for many years, and so we didn't see the need to change on that one yet. But we do have support for it in Hubble. It's not perfect, we use a site called Vulners, vulners.com. Originally we were downloading, like, originally the AI had the ability to download the whole vulnerability list, basically, and then you could audit it locally and look against the existing packages, and see if you had any issues and then report those. That was nice because it's a hard sell to send your list of installed packages to an API, right? Because now they know if you have any vulnerabilities. I'm not saying that vulners can't be trusted, but I would hesitate to use them without paying for them.

42:59 Michael Kennedy: Just in general, yeah.

43:01 Colton Myers: Just so you can have somebody to blame, you know what I mean. Basically, with that original version we were just hammering their servers, and they were like, please stop. Please don't do this. And I was like, okay. And so, like, their new method is you actually get an API key from them and then you plug that into our Vulners module configuration and then it can basically, it will securely send your list of installed packages to them and they'll report if you have any vulnerabilities. And it's not perfect.

43:28 Michael Kennedy: Yeah, how does it do that? Does it look at the requirements, like the pin versions in the requirements? Does it look at your virtual environments or your system level one and see what's installed, or what's the process of even knowing what to send them?

43:39 Colton Myers: So I don't think it does stuff like Python packages. I think it's mostly dealing with system packages. Like we know, you know, so they can keep a list of.

43:48 Michael Kennedy: Like Bash or something.

43:50 Colton Myers: Yeah, CBE, a CBE came out on. And a lot of these things are repackaged, right? Like you have, you can install, you can do the yum install requests, Python Bash Requests, right. And you can get requests in its packaged Python version. And so, you know, that kind of stuff would be reported if there's a big requests vulnerability then you can know that you're vulnerable and you need to upgrade. But yeah, I don't think it does the Python stuff. So this is something where we don't have a super great answer for it, because it's arguably not a great thing to send all of your security data off to Vulners, who you maybe don't necessarily trust, right. And I haven't figured out a good way to do that because I don't want to be in the business of trying to compile that list of vulnerabilities, because that's why companies pay big bucks is because it would just consume all my time, right?

44:37 Michael Kennedy: Right, right.

44:38 Colton Myers: Whole companies like Vulners are just designed to just do that, right? And they make money doing just that. So I don't know where that's going to go in the future with Hubble, but you know, for now you can do it with Vulners, which is great as long as you're okay with that compromise.

44:52 Michael Kennedy: Sure, that's interesting. So one of the things I do for my servers and projects is I use a thing called PyUp.io, and basically it looks at your requirements.txt, pins the versions, and if there's new ones, it'll do a PR to automatically increment that and it'll let you know if there's a security vulnerability in them, but that's only in the Python side, and you still have the problem of you're sending them your requirements file, but.

45:18 Colton Myers: And Github has been doing great work recently, have you seen that?

45:21 Michael Kennedy: Yeah I have.

45:22 Colton Myers: Where they'll just be able to inspect your requirements file for you and say hey, there's a CVE on this, you should upgrade this, and it's like whoa, thank you Github, that's awesome.

45:29 Michael Kennedy: It is really good, and it's a huge warning across when you log in, they update it weekly.

45:34 Colton Myers: Yeah, you're not going to miss it.

45:36 Michael Kennedy: Yeah, that used to, it would have been around for about a year I think. But it was only for Javascript and Ruby.

45:41 Colton Myers: I think it's doing it for Python now.

45:43 Michael Kennedy: Yeah, now they added Python support which is great. Just, like, a few months ago. So yeah, that's a really good one as well. I guess maybe we could talk just a little bit about some of the pieces involved in some of the building blocks. So we talked about Salt, and we also talked about Vulners, so what else is involved here? It's your Python code, those two projects, and what else? Like how have you built it?

46:04 Colton Myers: We actually used Docker to create our packages, which I recommend highly. It's really great, like I can just basically do a Docker build and then a Docker run and I just have packages, it's beautiful. We have a security engineer who's all about, like, not trusting what's on the host. Which is smart, right? Like theoretically, an attacker could compromise the tools that you're using to try to track down the attacker, right? So like, in a security utopia, Hubble would have nothing that comes from the host. It would have all its own stuff, it would lock itself down immediately and, like, not be modifiable. So we've worked to, one of the easy ways to serve profiles to Hubble is via Git. And we include, oh, apparently I'm still going through puberty. We include the Git libraries, the Git Python libraries, we can use GitPython or Libgit2, we prefer Libgit2 and so we actually ship that with Hubble. But that also requires Git to be installed on the host, and Git is a really easy requirement to source. Like if you have added dependency for Git, it's almost certain that the host will be able to serve that dependency. Because that's a problem with something like osquery, which is not necessarily in all of the normal repos, so like that's harder to assume that it's going to be there. But even with stuff like Git, we actually, we recently added some code into our build process to just compile Git from source. So now we don't even rely on Git on the host. We compile osquery from source, not because we need it to be newer than what they do, than the stuff that they provide, but just because we can add some different compile flags that are more useful to our, like, little siloed osquery over here. So we compile a bunch of stuff from source as part of our build process, but yeah. The big pieces are Salt and its dependencies, the reason we use Salt SSH is because we don't need any of the transport stuff, which is where all of the difficulties in sourcing come from. Like all of the sudden when you're sourcing ZeroMQ, and these other things, you're talking about C Code and binaries, right? It's not just pure Python. Pure Python is easier, and so we use Salt SSH because it doesn't have any of those dependencies. Vulners, we have their, we now package their Python package in with our stuff so you can hit their API easily, we package osquery in there, and yeah, those are really the biggest pieces that we add.

48:21 Michael Kennedy: Yeah, that's cool. It's pretty stand alone, then, it doesn't have a huge list of dependencies. You know, you opened this whole section by saying, well, there's this trade-off, like security generally makes usability worse but you kind of want it. I know but, what's surprising is like, in your description here, your security considerations, like let's just package Git in there even though it's a pain for us, and let's package in osquery and all that, like that actually makes it easier to deploy and maintain. So it's not always true, right.

48:46 Colton Myers: You're certainly right, yeah. But, it does still have trade-offs, let's be clear. Because now our package is 50 megabytes instead of five.

48:51 Michael Kennedy: Right.

48:55 Colton Myers: So theoretically we could use the stuff that's already on the host, like that could make our packages smaller and stuff like that, but let's face it. It's 2018, and 100 megabyte package is just not really a big deal in general, like, in most cases.

49:09 Michael Kennedy: Yeah, yeah. What is that, two, three seconds to download on a good connection? It's fine.

49:13 Colton Myers: Yeah, and usually you're going to rehost these in your data center, you know, because usually you'll have mirrors of the repos or whatever and it's just not a big deal.

49:21 Michael Kennedy: So, it sounds like such a cool project. People might be interested in participating, maybe bringing it over into their organization or just working on it in general, are you looking for contributors? And if so, how, what, things like that?

49:35 Colton Myers: So we definitely are. Our repo is, I'll make sure it's in the show notes, but it's all opensource, it's all Apache 2, you can use it, you don't have to worry about licensing issues, it's not GPL or anything like that. And we love contributors. I have been so swamped with, like, actually doing this at Adobe that I have not been able to spend as much time community building as I wish, right? I've done a few talks, I've been on, you know, Floss Weekly earlier this year, and now this podcast. But, like, I'm not great at doing the social media stuff aspect and all those other things, so I apologize if it doesn't look like stuff's going on. But you can see in the PRs that we are actively developing and we would love people to report issues, we would love people to add pull requests, it's all Python. It's a little bit Salt-y, like there's some Salt stuff, Salt-isms in there which might put people off but like, when you get actually down to, like, the modules that do the work, it's pretty much just straight Python so it can be pretty fun to write. And we try to keep some of our issues labeled with like, the help wanted, good first issues labels, so that's a good place to start. We have a Slack workspace, which you can get to from our homepage, which will also be in the show notes. And we'd love to see you. The Slack workspace is the best place.

50:48 Michael Kennedy: How about writing tutorials or something like that?

50:51 Colton Myers: Yeah, sure, like documentation. I recently did a big overhaul of our documentation which was way out of date, and so it's on read the docs now, but obviously that doesn't update itself. Some of it does, if you base it off of the doc streams, but we'd love work there. If you get into Hubble and run into issues and are like, this was not clear in the documentation, please submit a PR and let's fix the documentation or at least this issue. Like I don't even care if you don't want to, like, submit a PR, that takes time if you're not set up for it. But if you just file an issue and say, hey, this could be better in this way, like, I'll love you forever. So yeah, we'd love to have people contribute, we'd love to grow the community and make this just like a really good tool and yeah, I'm looking forward to your feedback.

51:32 Michael Kennedy: Sounds good. So, when I was researching this project, I stumbled upon the Adobe Security Blog and it looks like you guys are actually doing a lot of interesting stuff out in public. Do you want to maybe just talk a little bit about that? Because it seems way more open and put together for public consumption than I would have expected.

51:52 Colton Myers: Yeah, Adobe is huge, right? And all of a sudden, it was so funny, because we did all of this Hubble stuff and we're like, deploying everywhere at Adobe and my coworker Christer who came up with the idea for Hubble, he's like hey, the Security at Adobe guys want to sponsor, have HubbleStack sponsor SCaLE over in California, Southern California Linux Expo. And they're going to sponsor a booth and give us some materials to take down and stuff like that, and I was like, that's a thing? We have, like, a marketing team dedicated to Security at Adobe? And he was like, yeah dude. And I was like, okay. So, like, they've been great.

52:26 Michael Kennedy: That's pretty cool.

52:27 Colton Myers: They blogged about Hubble a couple of times and they kind of ping us on their social media and stuff like that. Opensource at Adobe, you had a note in here about that, that was another thing that I was super surprised about because, like, I don't think about Adobe as an Opensource company, right? And it's really been in the last few years that it's really grown. We have, like, an Opensource committee inside that both goes for like opensource into, like, open opensource into the world, but then also encourages opensource inside of Adobe, right? Which, you know, sometimes teams make their stuff, like we have our own Enterprise Github instance. And, you know, you might be tempted to make your stuff private, even though it's Adobe only. And then they're like, no, let's make it public. Let's share, let's collaborate inside of Adobe. And it's a really cool effort, and in fact we're having our Adobe opensource summit internally tomorrow, which will be real fun. And they, like, have been helping teams through the legal stuff because, you know, you have like to get legal approval to opensource a project and make sure Adobe doesn't want to actually compete in that space, that kind of stuff. But anyway, so Adobe does have a network that's pretty substantial these days on Github, so it's github.com/adobe, and you can check out all of our opensource stuff there. Hubble is not on that org because we have multiple repos, so it doesn't gel well with having a giant, you know, being part of a giant org. If it was one repo, it would be there, but we have our own org, so.

53:47 Michael Kennedy: Yeah, cool. Yeah, I was surprised to see that, and it's a good sign. So let's close this out with sort of a high level conversation. You've been building all this software to defend the servers and sound the alarms when needed. Are you hopeful about security, cybersecurity stuff in 2018? Are things getting better, getting worse, what do you think?

54:06 Colton Myers: I think they're getting better. I mean, it's so hard with all the legacy stuff, right? That's really where the problems come in, is when you have this product that's been around for ten years, and rewriting it from scratch is super expensive and that's a hard sell until you actually get owned, right? And then it's like, well, okay, this is going away within the year, right. And so, you know, I think it's still okay. The thing that I probably worry about more is if quantum computing breaks SSL or something like that, I think that's probably the bigger risk to like the overall internet. Cause if you don't have SSL, I don't know what you do at that point.

54:43 Michael Kennedy: Economies come to a halt.

54:44 Colton Myers: I just try not to think about it too much.

54:47 Michael Kennedy: Yeah, that's like thinking about the9.7 magnitude earthquake. Well, it's like, there's nothing we can do about it, so let's just hope that doesn't happen. I feel like, not super pessimistic about quantum computing breaking the internet, because I feel like when that breakthrough happens, there's going to be like one or two quantum computers in the whole world, and there's going to be a long time before that becomes something in the hands of standard people.

55:12 Colton Myers: And maybe by the time it's something in the hands of standard people, we'll have something that is, we'll come up with another math problem that will solve that for us, you know what I mean? Like, you know?

55:21 Michael Kennedy: Yeah exactly, like TLS3 that's Quantum Safe, Quantum Safe Encryption.

55:25 Colton Myers: Yeah, I don't know.

55:26 Michael Kennedy: Or something like that. But yeah, that is a little concerning, but yeah, I'm generally hopeful for it but I don't know.

55:32 Colton Myers: I would say I'm generally hopeful for it as well. That's just the one thing that, like, niggles at the back of my mind, I'm like, oh, that would be really bad. This is like, apocalypse fiction level stuff right here. But hopefully.

55:43 Michael Kennedy: Yeah it is.

55:44 Colton Myers: I mean, we've had plenty of those kinds of problems that we've, I don't know if we've had problems like that, but we've been able to weather a lot of things already so I'm not super worried about it. I think we'll always have something that we can add on top and weather the next storm.

55:58 Michael Kennedy: It would definitely send people scrambling. It would make the front page of a lot of news if that happened, though. Cool, well, I guess we'll leave it there for now. We could talk about this last part for a whole another episode I'm sure. Before we move on, though, let me ask you the final two questions. When you write Python code, what editor do you use?

56:14 Colton Myers: I use Vim, I keep trying to, if I was going to switch to an IDE it would be PyCharm, they have the best Vim integration, or, Vim emulation that I've ever used, and they have a lot of other cool tools. But I hate not having it in my terminal, so I stick to vim.

56:28 Michael Kennedy: Yeah, I think a lot of people like the, it's just always around everywhere, so I'm going to use that. But here, I'll push on. And then, notable PyPI packages? Obviously you can install Hubble from PyPI but you're probably better off getting the prebuilt binaries, right?

56:46 Colton Myers: I definitely recommend using the prebuilt binaries because that way you get osquery, you get all these different things. I have been lax, and like, there's no actual reason why Hubble shouldn't be on PyPI and I just haven't done it yet. So look for it in the near future, hopefully, because I just need to do it. It's not that hard, but yeah, technically you can source install Hubble without any difficulty, so.

57:07 Michael Kennedy: Yeah, cool. And so another one besides Hubble?

57:10 Colton Myers: I don't have any others off the top of my head, I love requests. Like I know the whole world loves requests but it's great. Hubble is like my main one right now, I worked for a long time on SaltStack which is a pretty good product. I don't love some of the directions that the company's going that backs it, but it is opensource Apache 2 so check out SaltStack as well.

57:28 Michael Kennedy: Right on. Alright, so, final call to action. Maybe there's a bunch of companies or organizations out there that are nowhere near as organized as you guys are at Adobe. They maybe want to start using Hubble, they want to start putting some of these practices into place, what do you tell them?

57:43 Colton Myers: It's never too early to start thinking about security. That's, I guess, is where I go. It's expensive, I know, like to actually start looking at this stuff you have to like hire security engineers and stuff and you know, how do you have the money to do that? But if you lose a whole bunch of customer data it's going to be way more expensive, so. Think about it early, the tools are so good these days. You know, HubbleStack included, I think Hubble is a pretty great tool and it's pretty easy to deploy. So there's things you can do from the beginning just to do a baseline thing that will just at least give you something, right? And so just start, that's all.

58:16 Michael Kennedy: Alright, cool. Thanks for being on the podcast.

58:18 Colton Myers: Thanks so much.

58:19 Michael Kennedy: Alright, bye. This has been another episode of Talk Python to Me. Our guest on this episode was Colton Myers, and it's been brought to you by Linode and Rollbar. Linode is bulletproof hosting for whatever you're building on Python. Get four months free at talkpython.fm/linode. That's L-I-N-O-D-E. Rollbar takes the pain out of errors, they give you the context and insight you need to quickly locate and fix errors that might have gone unnoticed until your users complained, of course. Track a ridiculous number of errors for free as Talk Python to Me listeners at talkpython.fm/rollbar. Want to level up your Python? If you're just getting started, try my Python Jumpstart by Building 10 apps, or our brand new 100 Days of Code in Python. And if you're interested in more than one course, be sure to check out the everything bundle. It's like a subscription that never expires. Be sure to subscribe to the show. Open your favorite podcatcher and search for Python, we should be right at the top. You can also find the iTunes feed at /itunes, Google Play feed at /play, and direct RSS feed at /rss on talkpython.fm. This is your host, Michael Kennedy. Thanks so much for listening, I really appreciate it. Now get out there and write some Python code!