How do you keep track of your security, configuration states, and even out-of-date system-level packages in your servers?
Well, what if you had 40,000 or more servers to manage?
How would your process scale then?
I'll tell you, mine would take a few tweaks.
On this episode, you'll meet Colton Myers, who is a co-creator of Hubblestack.
Hubblestack is an open-source security compliance framework.
It provides on-demand profile-based auditing, real-time security event notifications, alerting, reporting.
And yes, Colton, the group, has over 40,000 servers, and Hubblestack is watching over all of them.
Learn about this cool Python-based framework on this episode of Talk Python to Me.
It's episode 187, recorded November 14th, 2018.
Welcome to Talk Python to Me, a weekly podcast on Python, the language, the libraries, the ecosystem, and the personalities.
This is your host, Michael Kennedy.
Follow me on Twitter, where I'm @mkennedy.
Keep up with the show and listen to past episodes at talkpython.fm.
And follow the show on Twitter via at Talk Python.
This episode is sponsored by Linode and Rollbar.
Please check out what they're offering during their segments.
It really helps support the show.
Colton, welcome to Talk Python.
Let's get started with your story and how you got into Python.
I've been using computers since my parents got a black and white Macintosh.
I don't actually know what it is.
I should go back and figure out what Macintosh it was so I can tell people.
But I would play Lode Runner on that when I was little.
I'm, what, 30?
Yeah, 30.
I can never remember.
So I'm younger than a lot.
I guess I'm starting to get into the middle of the pack these days.
But yeah, I remember playing with that when I was like seven on this little black and white Macintosh.
Then we had like a Windows 3.1 machine that I'd play games on.
And I mostly played games until high school.
And then in high school, I actually, half of my senior year, I did the supplied technology course in computer programming, which was really great, actually.
Like half the day I would send over here.
It was C#.
They had just switched from C++ to C#.
Wow, that's a pretty big improvement, actually.
Oh, yeah, yeah, yeah.
I'm with you on that.
And C# is a great language.
And back then I was Windows, Windows, Windows, you know, because I was played video games, right?
So it worked out just fine.
And it was a really good course and really gave me like a jump start.
And from there on, I knew that's what I wanted to do.
So went into college.
That's awesome.
Let me ask you more about that because that sounds so awesome.
This was a high school program?
Yeah, yeah.
It was like...
And how long did you get to spend programming each day?
Half the day.
Like you could either do morning.
They had a morning session or an afternoon session.
And so like basically at lunchtime, you would switch either back to your high school or to
the biotechnology center.
And it was difficult because you had to like structure your whole schedule around it.
Like you had to start planning.
You could technically do it as a junior and a senior if you like really plan.
But then you'd have to like...
I was also in like band and stuff like that.
And I'd have to have dropped that.
And I was like, no, I'm just going to do it as a senior.
But you still have to like make sure that you have all your other like things to graduate,
you know, in the rest of your schedule.
Your four years of math or three years of English, whatever, right?
But it was a super cool program.
Like and our professor, I think he's now working for Microsoft or something.
But he also used to teach at university.
Like he was really good.
And he did a great job of like teaching us the right way.
And I don't know, man.
It was a really good introduction to programming for me.
I am so jealous.
When I was...
This wasn't high school.
It was my last year in middle school, I think.
I'm pretty sure.
But, you know, it could have been the same in high school.
My teacher, you know, I took sort of an intro to programming course, right?
And my teacher was kind of okay.
And I was just really loving it.
I said, oh, I'd love to write this kind of program.
And like, how can I learn more?
And they're like, well, you can do that.
But you got to understand, like real programming is actually super hard.
And I don't know if you really can do it.
Like, I mean, that was my idea.
Like, okay, great.
Oh, well, that's awesome.
I don't know.
I'm blessed.
Programming just clicks with my brain.
Like, it's just is...
It has never been a struggle for me to learn how to program.
Like, you know, none of the conditionals, none of the advanced logic.
Like, obviously, I struggle with like advanced algorithms like anybody does.
But like, the basics of programming just came to me immediately.
And so...
And my teacher actually recognized that.
And it was super nice because he like had me helping other students, which was frustrating as much as it was.
But it was useful, right?
Like, I've always been a bad teacher because with stuff that comes so easily to me because I can't bridge that gap.
But he had me helping other students.
But then he also like had me doing some of like the second year curriculum along with the first year curriculum.
Because he's like, like, I would just finish so early on these projects.
And you just be like, well, you can either just sit here and do whatever you want.
Or I can get you started on this other stuff.
And it was a real blessing like to have that start and to have somebody so supportive.
Yeah, I've never heard of anything like that.
I've heard of like vocational stuff like shop and, you know, carpentry, but never software.
So, man, that's awesome for you.
All right.
So, that was C# back in the day and Windows.
And somehow, your project is not in C#.
No.
So, college, I went straight into computer science.
I knew exactly what I wanted to do.
And I managed to make a couple of friends there that were super useful because I went to the University of Utah.
And at the time I went, it was still, you know, there.
They would start in C#.
And then you go to C++.
There was a lot of Java involved, like, through all their classes.
But they didn't like Python hasn't hadn't at that point really pierced the, you know, the education, which, which is a whole different conversation as to whether we should actually be teaching Python as a first language.
I almost think it's almost doing a disservice because it's so easy to program in Python.
But I don't know.
It's a hard conversation.
But I made some good friends who basically we kind of pushed each other to explore outside of what our classes had.
So, like some classes where you could choose the language and most people chose C# or Java because that's what we were learning, right?
But we decided to try Python at one point.
And it was, you know, really awesome.
And we were using Git long before they, you know, taught us how to use Git and all these things that really, like, helped jumpstart that and helped me get a base in stuff that maybe I wouldn't have otherwise.
And I found Python.
I got my laptop stolen in while I was on vacation in Hawaii.
And I had a good friend at work who convinced me to buy a Mac instead of a Windows machine, which was a great decision.
I really like macOS in general, especially because it's just Unix underneath.
And so, you know, that got me into more of the Unix side of things and kind of away from Windows.
And you're like, hey, this C# stuff doesn't really work so well.
There's mono, but it kind of is clunky over there.
And so now what, right?
Well, once I found Python, like Python, you just get rid of so much of that boilerplate.
Like Java and C# have all this boilerplate and it's just so verbose.
And Python, like, any time, like, Python was really interesting because, like, I would be like, oh, I need to do this.
I need to, like, open a file or whatever.
And I'd be like, okay, so I'm betting it's, like, this is the function name.
And it turns out most of the time it was right.
Like, it just, like, it felt like it was written out of my, straight out of my brain.
And it was just so nice.
But, yeah, like, with C# back then, you couldn't really write C# in Linux.
Like, mono was just getting off, just getting started.
And so I didn't really, yeah, I basically avoided it wherever I could from then on.
That's awesome.
Yeah, it was really great.
Yeah.
And of course, you know, those other libraries didn't have nearly the package management story and the ecosystem story that Python does.
Sure.
Yeah.
And then the way, it's really interesting because I just feel like, I don't know, I just have, like, there are so many things in my life that if they'd changed just a little bit, I could have been on such a different path, right?
And I think that's true of all of us.
But, like, when I was getting close to graduating, I was on a local Python user group.
And local startup advertised on that user group, hey, let's, we're looking for Python developers.
And I was like, well, I'd love to do Python.
But, like, my internship that I'm working on is in C# and C++, right?
I'm working, like, at this, you know, on this legacy code base and this DoD contractor, this local or whatever.
And I was like, I don't have any experience.
Like, I've just written some Python for, like, class projects.
And they're like, well, come in and interview.
I was like, okay.
And apparently, I interviewed pretty well.
They were impressed.
So, they actually hired me on part-time until I graduated with the expectation that, assuming I did well, you know, they'd hire me full-time.
And it worked out.
And I, that's how I joined SaltStack.
So, I worked with, for SaltStack for three years and really, like, got my Python, you know, really solid and, you know, worked, was able to expand my ops knowledge, you know, because it's a configuration management tool.
Yeah, it's not just the Python you got to learn.
Right.
It's also all the Linux and, you know, DevOps side of things.
Yeah.
So, that was, like, that was hugely important to my career.
Because without that, like, I don't know.
I could have found something, I'm sure, is like a junior Python dev.
But it just really jump-started it and actually directly led to my position at Adobe, too.
So, yeah.
That's kind of where my career path has been.
No, that's a super interesting story.
So, speaking of where you are today, you know, so what do you do day-to-day at Adobe?
So, I'm a senior software engineer at Adobe.
And I actually mostly work on Hubble.
I joined a, I joined actually, the other creator of Hubble.
It was kind of his brainchild, but he's not really a developer.
So, he was able to, like, you know, say, hey, I want to do this proof of concept.
Can I hire somebody for it?
And he hired me.
We had met because he was actually in the SaltStack community.
And his name is Christopher Edwards, by the way.
I should give him credit.
But this is his baby.
And so, he hired me to do this.
And the idea was to base it on Salt.
And basically, we were trying to replace an existing vendor solution with this proof of concept because the vendor solution wasn't very popular and was very expensive.
And so, yeah, we were able to...
So, he brought me in on this.
And it became just almost overnight.
Like, we wrote this proof of concept.
And it was good, right?
And people liked it.
And all of a sudden, it was like, oh, hey.
Other people heard about it.
And they're like, hey, by the way, our contract with CloudPassage is up next.
And we're losing this promotional pricing.
It's going to get like three times more expensive.
So, could you maybe make it so that Hubble is everywhere at Adobe within like, you know, the next six months?
Like, that'd be great.
It's like...
No pressure or anything, right?
Oh, it was crazy.
It was a crazy, crazy time.
But we were able to...
Like, we had an open source from the beginning, which was super awesome.
And because it's grown so much inside of Adobe and is so important to our security posture,
it's my full-time job.
I get paid full-time to write open source software in Python.
So, you can't really go wrong with that.
Yeah.
I feel like that's a lot of Python developers' dreams, right?
They're like, I want to work on this open source thing I love.
And I want to get paid full-time for it.
Like, there's not that many people that get to do that.
And I just like fell into it.
Like, I...
So much, so much...
Yeah, I don't know.
I don't even know how to approach how lucky I am that way.
But it's been really great.
So, yeah, that's kind of what I do.
I just work on Hubble Stack.
Okay.
Super.
So, maybe let's start with a super high-level conversation about this.
Just even...
Not just Hubble Stack and Hubble and Hubble Stack, but just this idea in general.
So, what it is an open source security compliance monitoring and auditing system.
Like, tell us just sort of about that world.
Like, what kind of...
So many words.
What kinds of things...
Yeah, I know.
Yeah.
So, just tell us, you know, what's that world like?
And what kind of role does this try to play in there?
So, there's a couple pieces, right?
We...
Security has multiple facets.
And one of those facets is compliance.
And really, the reason compliance is so important...
I'm going to put air quotes around important.
Because the reason compliance is so important is for these certifications, right?
If you deal with customer data, especially like credit card data and that kind of stuff,
then you need like PCI.
Right.
Exactly.
You know, you might have SOX compliance.
You might have HIPAA, right?
All these different things that if you can check that box, you can have customers that you might
not otherwise, right?
And each of these has different requirements for compliance.
A lot of them are a lot less fleshed out than you'd expect.
Most of them, like, they just say, like, the audit...
The auditors come in and they...
And the audit control says something like,
you have a baseline security compliance check thing that you audit against regularly and
make sure that everything is in compliance, right?
And it's just like written like that.
And so at Adobe, we use the CIS compliance checks, Center for Internet Security.
They are a nonprofit that publishes these profiles that allow you to...
You know, if you follow all these checks in here, then you'll be more secure than if you
didn't.
That's the idea, right?
Right.
So does it include, like, known vulnerabilities or is it more like guidelines on, like, hardening
of the server or what's...
Yeah, it's more along the lines of hardening.
Like, I don't think CIS...
I don't actually know if they do, but I don't think they publish, like, a vulnerability list
or a CVE list or anything like that.
It's more like, hey, do you have root login enabled on your server, right?
In SSH.
And if you do, that's a problem, right?
Do you have password list login?
Do you have password login even, right?
Like, or is it just SSH?
Like, you know, those are easy ones to point out for SSH, but it's also like, hey, do you
have Telnet installed, right?
Do you have all these different things that are just like, you know, do you have FTP?
Are you serving anything via FTP as opposed to SFTP, right?
Like, just the gimmies, right?
And obviously, they get a lot worse, like...
Or a lot worse.
A lot more stringent.
Like, oh, you need to have SSH timeouts on your host set to these low values, right?
So that it quickly kicks people off if they're not typing and that kind of stuff.
And those, you know, obviously, as you get further and further into these checks, these
different levels, they actually split them up into levels.
It becomes less convenient, right?
Because security is always a compromise between convenience and security, right?
Right.
So we basically choose a subset of those and we audit against them.
And that is actually where like the big money is.
Like, that's where we get like customers, right?
But if I'm being honest, it's not actually making us a lot more secure.
Like, obviously, if we have root login enabled, we should not, right?
But like you fix that once.
But that doesn't mean you're secure now, right?
And so the other piece, the rest of Hubble is just to collect other data that our blue team
can use to watch for actual like ongoing attacks, big holes, you know, that maybe an audit wouldn't
have caught otherwise.
Right.
So give me an example.
This is something I wonder all the time.
If you don't know, that's fine.
But give me an example of what an attack looks like, right?
Like if I log into my server, it seems okay.
You know, nobody's saying, hey, our data seems to be gone.
But like, how do I, you know, shared, right, like stolen, but how do I look at the server
and know outside of say web logs?
Yeah, I mean, that's kind of a hard question.
And I and that's actually not my expertise.
Like I'm not a security engineer, but I work with them a lot.
One that's pretty common in 2018 is if somebody gets access to your server, they might install
a cryptocurrency miner on your server, right?
And so, you know, one of the ways we can look for that kind of stuff is, oh, you know, in
one piece of Hubble, we collect like the established outbound connections on your host, right?
Because certain services have outbound connections, like that's a thing that they should
have.
But a lot of them don't.
A lot of them just accept inbound, right?
Especially if it's just like a web server, right?
It's accepting inbound connections, but not reaching out.
So if we see an outbound connection from a process we don't recognize or an uncommon process,
then we might look into that and say, hey, you know, what's going on here?
I see.
That makes a lot of sense.
Yeah, I mean, there are a lot of like tells like you can.
You could have increased CPU usage or like, you know, stuff slowing down.
Like, obviously, they're trying not to leave tracks, right?
So they will tend, you know, attackers will tend to try to not let you know that they're
there.
So it's hard.
Like, it's not it's a hard problem.
And that's why we have a whole security team.
We have the team I work with just with the digital marketing.
They must have 30 security engineers, both like red team and blue team, like just constantly
combing this data looking for patterns.
And we have a whole security operations center looking for these alerts.
And it's like it's a big thing because it's harder and harder, you know, in 2018 to keep
your stuff secure.
It definitely is.
So maybe just to find some terms for folks who are not super up to date on the security
side of things.
Red team is the penetration testers and the folks trying to break in and the blue team
are people like using Hubble to say, you know, wait, wait, wait.
Why is it making all these connections to this VP over this VPN to somewhere we don't
recognize?
Yeah, exactly.
The red team is trying to break it.
They're the hackers.
They're the white hat hackers, right?
That say, how can we we're just going to spend all day trying to get into Adobe's hosts.
And then if we find a way in, then we can patch that hole.
The blue team can patch that hole.
And they actually have like these whole things where like red team will set up these different
things and make sure the blue team catches them.
Like sometimes red team will work with access to the box and just say, let's assume I get access
to the box.
I don't you know, I don't have a way right now.
We've patched all those holes, but let's assume and I'm going to just set up these breadcrumbs
and make sure that blue team is catching them because blue team says they're watching for
these things, but are they actually watching?
You know?
Yeah.
And it's this back and forth.
And then blue team is trying to catch red team.
But then they're also trying to catch like black, you know, like like the actual attackers,
you know, because people are always trying to get into our data, right?
Like they it could be really lucrative.
And so we've had so many hacks that we see in the in the industry.
And, you know, nobody doesn't want to be next on that list.
So we have a substantial team.
You do not want to be in those headlines.
That's right.
So no, no.
This portion of Talk Python to me is brought to you by Linode.
Are you looking for bulletproof hosting that's fast, simple and incredibly affordable?
Look past that bookstore and check out Linode at talkpython.fm/Linode.
That's L I N O D E.
Plans start at just $5 a month for a dedicated server with a gig of RAM.
They have 10 data centers across the globe.
So no matter where you are, there's a data center near you.
Whether you want to run your Python web app, host a private get server or file server, you'll
get native SSDs on all the machines, a newly upgraded 200 gigabit network, 24 seven friendly support, even on holidays and a seven day money back guarantee.
Do you need a little help with your infrastructure?
They even offer professional services to help you get started with architecture, migrations and more.
Get a dedicated server for free for the next four months.
Just visit talkpython.fm/Linode.
Do you think containers and Docker and things like that make this harder or easier to protect?
It's a combination.
One thing that we've been struggling with is the so, you know, Docker is 12 factor apps, right?
Like, are you familiar with the 12 factor thing?
Yeah.
But maybe give the listeners a quick, just a super quick rundown.
What is the site?
It's like there's like a site 12 factor dot net with like one, two factor dot net.
And basically it's like this, the systems of things that like make, let's read their introduction.
Software as a service, 12 factor app is a methodology for building software as a service inputs that have these different things.
And really it's where containers come from and where these microservices come from.
Like you want everything to be immutable and like easy to, you know, set up fresh, you know, instead of changing a host when it starts to misbehave, you just burn it down and create a new one.
Don't embed all the configuration information and put that like in the environment.
And so it's separate from, say, your source code and your app code and whatnot.
And there are some great advantages to that because if you start with an image that like, for example, in auditing, right?
If you start with an image that has all of those audit checks already passing, right?
Then you're just starting from a much better place, right?
And you can make sure that everybody is starting from this, you know, hardened, secure image, right?
And then past that, when you're not messing with hosts, you have a lot fewer snowflakes.
I mean, you have a lot fewer inconsistencies across your hosts in theory, right?
And really where the attackers get in is in those snowflake servers that like have kind of been forgotten about, you know, sometimes they get in via dev, you know, whatever.
Nobody wants to upgrade it because it might break the thing.
Right.
That person used to maintain, but they left the company and no one wants to make it their problem.
So they don't touch it.
But it has problems too.
Like, for example, the established way to pass secrets into a container via the 12 factor method is via environment variables.
And security engineers don't like that.
They don't like secrets and environment variables.
Because soon as you're on the box, you can just, you know, query the environment variables and see all the logins and stuff, right?
Exactly.
And theoretically, you know, you don't have the ability to log into the actual container, right?
Because you don't ever need to.
And so theoretically, and then of course, you're using CoreOS or something that is theoretically more secure, right?
So like there's frameworks around that to try to prevent that from being as bad of a problem as it is.
But it still is, you know, a hard thing that we're trying to deal with.
So I do think 12 factor and containers and immutable images and those kinds of things do have a lot of advantages.
But and probably the advantages override the disadvantages.
But the other problem is that not every use case fits into that, right?
Containers don't like, you know, we have 40,000 some odd servers across our analytics product.
And like using those, like it's just not, you know, it would take a huge rewrite of that product to make it a microservices, you know, product kind of a thing.
So like, so like it's going to take many years.
And even then there are certain applications where like just being on bare metal is just better, more performant, etc.
So it's never going to be like the only thing that we do.
I don't think, I don't know.
I'm not going to try to guess.
But I do think from a security perspective, it is better overall.
Yeah, interesting.
I think it probably is as well.
It's easier to set one baseline and just create the images from that.
But still, I think it also, it seems like things change more often.
And somebody could flip a switch in that image that you don't realize or in a base image, you know, somewhere down that chain.
And, you know, all of a sudden, everything's like that.
Not just that snowflake server, right?
But that's also easy to undo when you find it, right?
Because again, you just switch it on a higher level.
And the next time they build their image, they get that, right?
So I don't know.
But yeah.
Yeah, a bit of a diversion.
But so let's, if you don't mind, tell us as much as you're happy to share.
Just a little bit about, you talked about 40,000 servers, something like that.
When I think of Adobe, I think of Photoshop and things like that that are more apps on like a desktop or a tablet or something.
Give us a sense of what a cloud and server world looks like over there.
So I have less of a sense on the, so we have two major business units.
We have our digital media, which is what you referred to, the Photoshop, etc, etc.
And then we have our digital marketing or digital experience as it's been rebranded recently.
And that actually started from an acquisition of a company called Omniture here in Utah.
And it turned into like, Adobe is one of the big players in marketing analytics.
They do everything from helping customers see what's working as far as advertising on their sites to helping serve social media stuff.
There's a whole suite of stuff that you can get from Adobe.
And it's much less known, even though it's a pretty big business unit.
And that's actually, on the digital media side, we have servers, right?
Because we have now Creative Cloud, right?
And we have to serve all that data.
We have to keep your...
You can keep your files in our Creative Cloud, all that stuff, right?
Right.
But it's much bigger on the digital marketing side because we're doing the software as a service for all these big companies.
Yeah.
So on the digital marketing side, we have something like 100, 120,000 servers or so across a whole bunch of teams.
We just acquired...
Adobe just finished acquiring Marketo, which added a whole bunch of new infrastructure to our...
That we have to worry about.
So how does that kind of stuff sort of shape what you're building and what Hubble is, where somebody outside of the company says,
Oh, we just acquired this other company.
And now they use this technology and here's all their servers.
It changed a lot.
It affected us a lot.
LiveFire, we acquired a few years ago.
And that one was a big one because they did a lot of auto scaling in AWS.
And our original Hubble was built on top of Salt directly, right?
You would deploy it as a series of modules inside of Salt.
And that worked well.
Not nearly as well as today's product, but they worked well.
But then we get to LiveFire and it's going to be prohibitively expensive for them to actually deploy SaltStack everywhere.
Because what they use, they use AWS a lot like containers, right?
They use AMIs.
They use images, right?
And they don't really modify their host.
They don't really have a good configuration management story in place because they don't need to, right?
They use it like containers.
So we basically realized that we needed a product that could be standalone, right?
That did not rely on Salt.
Because as this went wider and wider, a lot of teams at Adobe don't use Salt.
And we couldn't necessarily force them to.
Or it wasn't going to be prudent, right?
So we created Hubble, which we use PyInstaller to compile.
So that it basically, we do our best to compile everything into one directory.
All of our dependencies, our packages have no dependencies, except glibc, which we can assume is going to be on a Linux host.
It has no dependencies.
It ships with its own Python.
It doesn't mess with existing Pythons.
It doesn't mess with existing Salt.
It doesn't mess with anything.
And that was hugely advantageous because now we can just say, hey, teams, all you need to do is install this package, put the configuration in place once, and restart Hubble.
And all of a sudden, you were getting all of your reporting, right?
And we're not adding a whole bunch of new stuff to overwhelm your Salt masters, right?
We talked directly from those hosts to Splunk, which is the tool we use to collate all of those results.
And it turned out to be just way better.
And so eventually, we actually...
Oh, and by the way, that version of Hubble does still use Salt.
It just uses it as a library.
It just imports Salt.
Right.
But we were able to just slim it down and make it so much more targeted and make it work so much better and make the onboarding story so much better that we eventually deprecated Hubble Salt and just focused on the standalone.
Right.
So Hubble Salt was the original one where it was just completely assuming this is going to be built on SaltStack, right?
And now you kind of...
Yeah, you just add it as like a remote in your SaltStack master and it just like pulls down the stuff.
Right, right.
But with all these acquisitions and disparate infrastructure, it's like, okay, well, that's not working.
So let's not require that.
Let's just do this straight Hubble.
And what I'm really impressed with is your ability to take this and deploy it.
No Python dependency on the machine.
No pip install dash R requirements dependency on the machine.
Just here is a binary, run it.
Like that seems just golden for Python.
Oh, it's amazing.
Like I, you know, yeah, pip is great.
But like being able to like a lot of if you talk to SREs, they don't like pip in general.
Yeah.
They want to install via system packages.
Why wouldn't they want to just download arbitrary code from the internet and run it?
That's so, so paranoid.
It's just, I mean, and pip has gotten so much better.
But like, you know, it still doesn't even, it has like some uninstall issues.
Like these days, it's a lot better than it used to be.
But yeah, like they just want to do it as a package because that's what they're doing with everything else.
Right.
They just want to do RPM or deb.
Like that's it.
And so it's been really advantageous to do that.
And we actually, we also use, we cheat a lot, right?
We actually don't use proper Debian standards for packaging.
We just use FPM because FPM is amazing and just makes it so easy to package for any operating system.
And it's just been really good.
And it was so much better.
Like even if you're using salt, like my team uses salt.
And it was like upgrading Hubble salt was much more of a pain to get consistent across our infrastructure than just doing this package.
And so, you know, I immediately switched us to Hubble instead.
And I was like, this is so much better.
So, you know, we're actually in the process of completely phasing out Hubble salt.
It's mostly gone out of Adobe's infrastructure and Hubble is Hubble's king.
So is Hubble salt still a thing in open source?
Or are you trying to push everyone over to Hubble outside Adobe as well?
It's not a thing.
Well, if somebody wanted to fork it, it's Apache 2.
Go for it.
But we have stopped maintaining it.
I put a note in the readme and, you know, said we're deprecating this.
Like I encourage you to try Hubble because it's just better.
Like it's just a better story for users.
So obviously, I don't know if we said it explicitly, but Hubble is written in Python.
Yeah.
Yes.
So that's great.
But I do love that you don't have to, you know, as a consumer,
of it for your servers.
You don't have to care that it's in Python.
You just take the thing and run it.
It's great that we have pip and it's in Python and all that for developing it.
But it's, I just don't feel like pip is the right answer for consumers and users in general.
And it's cool that you guys made that work.
Yeah.
We like it a lot.
Yeah.
So when I go to the GitHub organization, I see three things at least.
I see Hubble and Hubble salt.
Those are the two we talked about, right?
And then there's also Hubble stack data.
Now, do you want to maybe tell us about that?
So the idea behind Hubble is that we, because upgrading is expensive, even though it shouldn't
be, even though it should just be a, an RPM upgrade, right?
It's expensive to work with all of these teams to upgrade because they're busy with their own
stuff, right?
So the idea was to try to make it as much as possible that we could safely change the information
that we're gathering from a source that, you know, security engineering controls.
Right.
So this is something like the vulnerable, new vulnerabilities or new things to check
for.
Exactly.
So we find out that we need to check for this new file or whatever.
We want to watch this file that we haven't been watching, or we want to add this new audit
check or whatever.
We don't want to necessarily have to deploy new code to the servers, right?
But on the other hand, we also don't want to give ourselves the ability to run arbitrary
code on the servers.
It's this compromise, right?
We work as hard as we can to give ourselves as much flexibility in data gathering, but we also
work as hard as we can to make sure that we can't modify the server from Hubble stack data
and that any changes to Hubble stack data will be safe, right?
So, you know, we have an agreement with our customers, which are other users at Adobe, that
we won't push code to that Hubble stack data repo.
And then we also, you know, we'll notify them and do it with a change request, a change management
request inside of Adobe whenever we merge to like the master branch, right?
So we treat it almost like a code deploy, even though it's not.
And that way, you know, we mitigate the risk and we do testing around it before we merge
those things.
And that has just allowed us to create trust with our customers, right?
That we're not going to, that Hubble is not going to mess up their servers.
And so, yeah, Hubble stack data is where all of the profiles is the word we've chosen.
That basically tells Hubble what data to collect.
And inside of there, we have a couple of different components inside of Hubble.
I guess I should have given like a high level review earlier than this, probably right?
35 minutes in here.
It's all right.
Like I mentioned, Hubble does a couple of things, right?
It does these audits, which are success fail checks against like, say, CIS or Red Hat Stig,
any, you know, these different compliance standards.
And that the code name for that is Nova.
Hubble, you know, is named after the telescope.
The idea is where we are looking with our telescope into the black hole of our infrastructure,
right?
We hope it's not a black hole, but sometimes it feels that way.
But it can be vast, right?
I mean, like trying to maintain 40,000 servers is not a not an obvious thing to do.
It's crazy.
And again, you know, yes, most of the servers are fine, but it's the edges that are hard.
And those edges are the hardest thing for SREs to keep track of, right?
So we everything is space themed.
We may be moving away from that in the near future just because it's really hard to keep
track of.
Like if I say Nova, you're like, I don't know what that is.
But if I say audit, you can say, oh, I get an idea of what that is.
So Nova is the audit piece and that's all success fail stuff.
It's built on this.
You basically write these YAML profiles in kind of an ugly format.
We're going to hopefully improve that in the near future.
But but it works well.
Just returns, you know, you can see all the failures.
And then, you know, we then can ticket, you know, in JIRA, the teams that need to make changes
based on those failures.
And then we have Nebula, which is kind of our like data gathering segment of Hubble.
It primarily uses OS query, which is a really cool open source project by Facebook.
And it basically OS query allows you to query your system as if it was an SQL database.
So you can query everything.
Right.
I've seen that.
That is amazing.
Yeah, it's really cool.
You can query everything from like network connections to like installed packages to
all sorts of stuff.
It's crazy.
It's actively developed.
They fixed for a while.
Facebook had we had some drama with Facebook around there.
Oh, it was like a patents clause and like they're in their projects.
And it was it was a bigger deal for Angular's Angular that they do.
Or is it?
I don't remember what JS project it is.
It's react.
I think react.
Thank you.
And it was this big deal because it's like if you if Facebook ever ends in ends up in
litigation with your company, right, then we this is revoked and you can't use our
thing.
Right.
And so it's this big thing that like, you know, a bunch of companies were like,
we got to get react out of our infrastructure because of all this stuff like the lawyers
were like, no, it cannot be that.
And anyway, Facebook has fixed that since including with those queries.
So it's it's been good.
But yeah, it's a really cool product.
And we use that to just collect terabytes of data every day from our hosts, everything like
from the easy gimmies like established outbound connections to like all the packages installed
on the system to all like the Docker processes running to shell histories to like we just collect
all this stuff and shove it into Splunk.
And then our blue team can sift through it and try to try to find actionable insights.
That's not what makes us the money.
But that's where the blue team is super excited about the data, right?
That's the data that's useful to them.
Right.
And then we have Pulsar, which is our file integrity monitoring or FIM.
We use iNotify on Linux and NTFS journaling on Windows.
By the way, Hubble runs on Windows.
Yeah, I was going to ask you that.
So obviously it runs on Linux, but also runs on like Windows server.
Yeah, yeah.
Mac, macOS.
It could run on Mac.
There's nothing like there's nothing stopping it from running on Mac.
It runs great.
We don't have any audit profiles that are Mac specific in Hubble stack data yet.
And we don't have like packages and like all the P list stuff for services.
Yeah.
I mean, there's very few servers running macOS, but there are some, I think, like some of these
build systems.
Yeah.
Well, and theoretically, like some people, some people would like to use Hubble for like,
like laptops and that kind of stuff.
And it's really not like desktop management for like, I want to know what my machines
are up to.
And it could be useful that way.
It would take some work to make it like more user friendly.
Like you can't just like this is meant to be run on the CLI and doesn't have a UI at all.
You know what I mean?
Like it would take some work, but potentially, but I've tried to push back on that because
I don't think it's useful.
But anyway, so Pulsar.
Okay.
Yeah.
Keep going on the infrastructures.
Yeah.
Pulsar, FIM.
We just basically collect file changes on the host and you can have it.
Like send checksums of the new files.
And that way you can compare, you know, is this, does this match our whitelist of checksums
for the specific binary or whatever?
We don't have a good answer on that one for like changes that are supposed to happen.
Right.
There's no, I don't have a good way to turn off our monitoring during say a configuration
management push.
Right.
Like, obviously I could say, oh, if you lay down this file, then we'll say, oh, that
file's there.
Let's stop collecting for a minute.
But like then an attacker could lay down that file.
Right.
So there's like, we haven't figured out a secure way for like a salt master to tell Hubble to
stop collecting because Hubble doesn't accept incoming connections.
Right.
It only shoves data out and pulls data from, from GitHub.
So, or from S3 or from wherever else.
And so those are the big pieces.
The other code name is Quasar, which is, I don't know if it qualifies as being its own
section.
It's basically just, that's the returners in salt.
That's the salt stack terminology.
But basically that's the glue that allows us to send the data to a destination.
We support Splunk.
That's what we use in Adobe.
We also support elk and log stash.
So that's what most of our open source users are using, I think, because Splunk is very
expensive.
Yeah.
So that's, that's kind of the high level overview of the pieces of Hubble.
This portion of Talk Python to Me is brought to you by Rollbar.
Got a question for you.
Have you been outsourcing your bug discovery to your users?
Have you been making them send you bug reports?
You know, there's two problems with that.
You can't discover all the bugs this way.
And some users don't bother reporting bugs at all.
They just leave sometimes forever.
The best software teams practice proactive error monitoring.
They detect all the errors in their production apps and services in real time and debug important
errors in minutes or hours, sometimes before users even notice.
Teams from companies like Twilio, Instacart, and CircleCI use Rollbar to do this.
With Rollbar, you get a real-time feed of all the errors so you know exactly what's broken
in production.
And Rollbar automatically collects all the relevant data and metadata you need to debug the
errors so you don't have to sift through logs.
If you aren't using Rollbar yet, they have a special offer for you and it's really awesome.
Sign up and install Rollbar at talkpython.fm/Rollbar.
And Rollbar will send you a $100 gift card to use at the Open Collective, where you can donate
to any of the 900 plus projects listed under the Open Source Collective or to the Women Who
Code organization.
Get notified of errors in real time and make a difference in Open Source.
Visit talkpython.fm/Rollbar today.
All right.
So give me an example of maybe how once this data is collected, like what kinds of questions
would you ask?
So I'm not familiar with Splunk.
You said this is not an Open Source project.
It's like a paid commercial thing that's kind of pricey.
Yeah.
So Splunk is...
How do they...
What do they call themselves?
Describe themselves.
Search, analyze, and visualize the machine data generated by your organization.
Right?
So the idea behind Splunk is it's kind of the same story as like an elk stack, right?
You have the collection, you know, with Logstash.
You have the coordination with Elasticsearch that...
Sorry, not coordination.
Queryability with...
Yeah.
Correlations, I guess, is what I meant to say.
Yeah, yeah.
Okay.
With Logstash...
Or sorry, with Elasticsearch.
And then you have Kibana for like reports and stuff.
And Splunk does all of those things, right?
You can do these elaborate searches and correlations across the data.
And then you can turn that into, you know, alerts.
Or you can turn that into dashboards.
And they have, you know, machine learning aspects that try to help you do some of these security
things without having to hire your own machine learning team.
They're still young, but it's pretty expensive.
I mean, I think your first few gigabytes are free or whatever.
But then it's basically billed by the amount of data that you send to it.
And, you know, it can get out of hand pretty quickly.
So you save your company money by only upgrading your servers like once a year, things like that?
Yeah.
I mean, I don't deal with the Splunk stuff.
They just tell me I can send as much data as I want to them, basically.
Send it here and they go do what they do, right?
Right.
And so, yeah, we send terabytes of data to our Splunk clusters.
Like the gimmies.
I keep going back to this established outbound connections because it's such an easy thing
for people to see.
Like if something is calling home that shouldn't, right?
That's actually pretty easy to spot in general, right?
Yeah.
Obviously, on something like, what's a good example of something that reaches out?
Well, let's talk about, say, a web server, right?
So it's got like a micro whiskey or a G unicorn worker process.
But that worker process is probably talking to a database.
Maybe it's talking to like a Stripe credit card API or a S3 bucket or something like that.
Maybe that thing, right, would be reaching out.
Maybe you've got, obviously, all the upgrades, right?
If you're going to upgrade, just apt upgrade, you know, that sort of thing that's going to
be reaching out.
But those are the few, right?
Those are easy to like limit to a specific whitelist, right?
Like, you know, the server needs to reach out to the database servers and you know what all
the database servers are.
So you say, okay, in the search, I'm going to eliminate all of those.
We don't care about those.
You know, it's going to be reaching out to the package servers, right?
And theoretically, hopefully you're rehosting those package servers, right?
So that you know, so you can keep that list, you know, you don't have to whitelist every
mirror in the world, right?
Right.
Or maybe your cloud server is, right?
Like something like that.
And then, you know, all these other things.
But then past that, like the only thing that it accepts, like from any location is incoming
connections, right?
On 80 or 443 or whatever, right?
So if you can create a search such that it says, hey, we know about all these different
things on these servers that are good reaching out or okay reaching out.
But if we see something from a new process that isn't in our whitelist or from an existing
process, because somebody could say, oh, I'm a I'm Apache too, right?
Like, you know, if you see see something from an existing process that is reaching out to
somewhere we don't know about, then we send an alert to the security operations center and
they contact the team and say, hey, is this normal?
Like, is this something we should add to our whitelist?
Can we look into this?
You might find that you have a cryptocurrency miner on there or something, right?
Like it's, you know.
Sure.
So that's like the easy one.
Yeah.
So you start with, let's just see what connections are being made.
All right.
These look valid.
Hide them.
Don't, you know, don't don't worry about those.
And then you just slowly either go through till there's no more.
There's none in your list or you start to get worried and figure out what the abnormal ones
are.
Exactly.
And it's a hard problem, right?
Right.
Like I mentioned, we have like 30 security engineers.
And it's because every time that you put something on that whitelist, you've given yourself a
blind spot.
Right.
Yeah.
So like, you still like, you know, ideally, you know, if we would have like some crazy AI
that could go through and be like, hey, let's investigate this data.
And it could notice things that we couldn't, you know, and that's and, you know, that's something
that Adobe is working on.
And so many companies are working on, right?
Machine learning and AI is huge everywhere.
Right.
But yeah, it's it's it's a really difficult problem.
And one that I'm not super qualified to talk about because yeah, no worries.
Not what I do all day.
So what we've spoken about so far to me sounds like one, a configuration of the machine and
two, oh crap, something is bad here.
At that point, I feel like you're it's not necessarily too late, but you're way far down the road of
it's not good.
You know, if there's a process connecting to something else on your server and it really is
malicious, like that's a long ways down the path to not good.
So one of the things that might be really helpful is to just say, I'm using this version
of Django or that version of request.
And it turns out that there's a critical security problem found in it.
We had better upgrade that before someone else, you know, figures it out.
Right.
Does it support searching for that kind of stuff?
Yeah.
So we don't actually use it for that in Adobe.
We have a different tool that they've been using for many years.
And so we didn't see the need to change on that one yet.
But we do have support for it in Hubble.
It's not perfect.
We use a site called Vulners, Vulners.com.
Originally, we were we were downloading like their original API had the ability to download
the whole vulnerability list, basically.
And then you could audit it locally and look against the existing packages and see if you
had any issues and then report those.
And that was nice because it's a hard sell to send your list of installed packages to an
API.
Right.
Because now they know if you have any vulnerabilities.
Right.
I'm not saying that Vulners can't be trusted, but I would, you know, I would hesitate to use
them without without paying for them.
Right.
Like it just in general.
Yeah.
Just so that you can have somebody to blame.
You know what I mean?
So basically, with that original version, we were just hammering their servers and they
were like, please stop.
Please don't do this.
And I was like, OK.
And so like their new method is you actually get an API key from them and then you plug
that into our Vulner's module configuration and then it can basically it will securely send
your list of installed packages to them and they'll report if you have any vulnerabilities.
And that's not perfect.
Yeah.
How does it do that?
Does it look at the requirement like the pin versions and the requirements?
Does it look at your virtual environments or your system level one and see what's installed
or what's the process of even knowing what to send them?
So I don't think it does stuff like Python packages.
I think it's mostly dealing with system packages.
Right.
Like we know, you know, so they can keep a list of, hey, like bash or something.
Yeah.
CVE came out on.
Yeah.
And, you know, a lot of these things are repackaged.
Right.
Like you have you can install you can do yum install Python dash requests.
Right.
And you can get requests in its packaged Python version.
Right.
And so, you know, that kind of stuff would be reported.
Right.
If there's a big request vulnerability, then you can know that you're vulnerable and you need
and you need to upgrade.
But yeah, I don't think it does the Python stuff.
So so like this is something where we don't have a super great answer for because it's
arguably not a great thing to send all of your security data off to vulners who you maybe
don't necessarily trust.
Right.
Yeah.
And I haven't figured out a good way to do that because I don't want to be in the business
of trying to compile that list of vulnerabilities because that's why companies get paid big bucks
is because it would just consume all my time.
Right.
Like, you know, that's right.
Right.
Whole companies like vulners are just designed just to do that.
Right.
And they make money doing just that.
So I don't know where that's going to go in the future with Hubble.
But, you know, for now, you can do it with vulners, which is great.
You know, as long as you're OK with that compromise.
So sure.
That's interesting.
So one of the things I do for my servers and projects is I use a thing called pyup.io.
And basically, it looks at your requirements.txt, pins the versions.
And if there's new ones, it'll do a PR to automatically increment that.
And it'll let you know if there's a security vulnerability in them.
But that's only the Python side.
And you still have the problem if you're sending them your requirements file.
And GitHub has been doing great work recently.
Have you seen that?
Where they'll just like, they'll inspect your requirements file for you and say,
hey, there's a CVE on this.
You should upgrade this.
And it's like, whoa, thank you, GitHub.
Like, that's awesome.
It is really good.
And it's a huge like warning across when you log in and they update it weekly.
You're not going to miss it.
Yeah.
That used to would have been around for about a year, I think.
But it was only for JavaScript and Ruby.
I think it's doing it for Python now.
Yeah.
Now they added Python support, which is great.
Just like a few months ago.
So yeah, that's a really good one as well.
I guess maybe we could talk just a little bit about some of the pieces involved and some
of the building blocks.
So we talked about salt and we also talked about vulnerable.
So what else is involved here?
It's your Python code, those two projects.
And what else?
Like, how have you built it?
We actually use Docker to create our packages, which I recommend highly.
It's really great.
Like I can just basically do a Docker build and then a Docker run.
And I just have packages.
It's beautiful.
We have a security engineer who's all about like not trusting what's on the host, which
is smart, right?
Like theoretically, an attacker could compromise the tools that you're using to try to track
down the attacker, right?
So like in a security like Utopia, Hubble would have nothing that comes from the host.
It would have all its own stuff.
It would lock itself down immediately and like, you know, not be modifiable.
So we've worked to one of the easy ways to serve profiles to Hubble is via Git.
And we include, apparently I'm still going through puberty.
We include the Git libraries, you know, the Git Python libraries.
We can use Git Python or LibGit2.
We prefer LibGit2.
And so we actually ship that with Hubble.
But that also requires Git to be installed on the host.
And Git is a really easy requirement to source, right?
Like if you add a dependency for Git, it's almost certain that the host will be able to
serve that dependency, right?
Because that's a problem with like something like OS query, which is not necessarily in
all of the normal repos, right?
So like, that's harder to assume that it's going to be there.
But even with stuff like Git, we actually, we semi recently added code into our build process
to actually just compile Git from source.
So we don't, we don't even reply, rely on Git on the host, right?
We compile OS query from source, not because we need it to be newer than what they do, than
the stuff that they provide.
But just because we can add some different compile flags that are useful to our like little siloed,
you know, OS query over here.
So we compile a bunch of stuff from source as part of our build process.
But yeah, the big pieces are salt and its dependencies.
The reason we use salt SSH is because we don't need any of the transport stuff, which is where
all of the difficulties in sourcing come from.
Like all of a sudden, when you're sourcing zero MQ and these other things, you're talking about
C code and binaries, right?
Like it's not just pure Python and pure Python is easier.
Right.
And so we use all this as H because it doesn't have any of those dependencies.
Volners, we have their, we now package their Python package in with our stuff so you can hit
their API easily.
And we package OS query in there.
And yeah, those are really the biggest pieces that we add.
Yeah.
And that's cool.
It's pretty standalone then.
It doesn't have a huge list of dependencies.
You know, you open this whole section by saying, well, there's this trade-off like security
generally makes usability worse, but you kind of want it.
I know, but what's surprising is like in your description here, your security considerations,
like let's just package Git in there, even though it's a pain for us.
And let's package it in OS query and all that.
Like that actually makes it easier to deploy and maintain.
So it's not always true, right?
You're certainly right.
Yeah.
Like, but it does still have trade-offs.
Like let's, let's be clear because now our package is, you know, 50 megabytes instead
of five, right?
Like, so theoretically, if we could use the stuff that's already on the host, like that could
make our packages smaller and stuff like that.
But let's face it, it's 2018 and a hundred megabyte package is just not really a big deal
in general, like in most cases.
Yeah.
What is that?
Two, three seconds to download on a good connection.
Yeah.
It's just, I mean, yeah.
And, and, you know, usually you're going to rehost these in your data center, you know,
cause you usually will have mirrors of the repos or whatever.
And it's just, it's just not a big deal.
Yeah.
So it sounds like such a cool project.
People might be interested in participating, maybe bringing it over into their organization
or just working on it in general.
Are you looking for contributors?
And if so, how, what things like that?
So we definitely are.
Our repo is, I'll make sure it's in the show notes, but it's all open source.
It's all Apache too.
You can use it.
You don't have to worry about licensing issues.
It's not GPL or anything like that.
And we would love contributors.
I have been so swamped with like actually doing this at Adobe that I have, I have not been able
to spend as much time community building as I wish.
Right.
I've given a few talks and been on, you know, FOSS weekly earlier this year and now this podcast,
but like, I'm not great at, you know, doing the social media stuff, the aspect and all those
different things.
So I apologize if it doesn't look like stuff's going on, but you can see in the PRs that we
are actively developing and we would love people to report issues.
We'd love people to add pull requests and it's all Python.
It's a little bit salty.
Like it's a little, there's some salt stuff, salt isms in there, which might put some people
off.
But like when you get actually down to like the modules that do the work, it's pretty much
just straight Python.
So it can be pretty fun to write.
And we try to keep some of our issues labeled with like the help wanted good first issues
labels.
So that's a good place to start.
We have a Slack workspace, which you can get to from our, from our homepage, which will also
be in the show notes.
And we'd love to see it.
Like the Slack workspace is the best place.
How about writing tutorials or something like that?
Yeah, sure.
Like documentation.
I recently did a big overhaul of our documentation, which was way out of date.
And so it's all hosted on read the docs now.
And, but obviously that doesn't update itself.
You know, some of it does if you, if you base it off of the doc strings, but like we'd love
work there.
You know, if you, if you get into Hubble and run into issues and you're like, this was
not clear in the documentation, please submit a PR and let's fix the documentation or at least
an issue.
Like I don't even care if you don't want to like submit a PR that's, that takes time if you're
not set up for it.
But if you just file an issue and say, Hey, this could be better in this way.
Like I'll love you forever.
So yeah, we'd love to have people contribute.
We'd love to grow the community and make this just like a really good tool.
And yeah, I'm looking forward to your feedback.
Sounds good.
So when I was researching this project, I stumbled upon the Adobe security blog and it looks like
you guys are actually doing a lot of interesting stuff out in public.
Do you want to maybe just talk a little bit about that?
Cause it seems way more open and sort of put together for public consumption than I would
have expected.
Yeah.
Adobe is, is huge.
Right.
And, and all of a sudden it was so funny cause we did all this Hubble stuff and we're like
deploying it everywhere at Adobe.
And then my, my coworker, Krister, who came up with the idea for Hubble, he's like, Hey,
the security at Adobe guys want to sponsor, have Hubble stack sponsor scale over in California,
Southern California Linux expo.
And they're going to sponsor a booth and give us some materials to take down and stuff like
that.
And I was like, we, that's the thing.
Like we have like a, like a marketing team dedicated to security at Adobe.
And he's like, yeah, dude.
And I'm like, okay.
So like, they've been great.
That's pretty cool.
They blogged about Hubble a couple of times and they kind of ping us on their social media
and stuff like that.
Open source at Adobe.
You had a known here about that.
That was another thing that I was super surprised about because like, I don't think of Adobe as
an open source, as an open source company.
Right.
And it's really been in the last few years that it's really grown.
We have like an open source committee inside that both goes for like open source into the
like open, open source to the world, but then also encourages like open source inside of
Adobe.
Right.
Which, you know, sometimes teams make their stuff.
Like we have our own enterprise GitHub instance and, you know, you might be tempted to make your
stuff private, even though it's in Adobe only.
And they're like working, no, make it public.
Like let's, let's share, let's, you know, collaborate inside of Adobe.
And, and it's a really cool effort.
And in fact, we're having our Adobe open source summit internally tomorrow, which will be
really fun.
And they like have been helping teams through the legal stuff, you know, cause you have to
get legal approval to, you know, open source project, make sure that Adobe doesn't want
to actually, you know, compete in that space, that kind of stuff.
But anyway, so Adobe does have a, an org that's pretty substantial these days on GitHub.
So it's github.com/Adobe.
And you can check out all of our open source stuff there.
Hubble stack is not on that org because we have multiple repos.
So it doesn't gel well with having a giant, you know, being part of a giant org.
If it was one repo, it would be there, but we have our own org.
So yeah, cool.
Yeah.
I was surprised to see that.
And it's, it's a good sign.
So let's close this out with sort of a high level conversation.
You know, you've been building all the software to defend the servers and sound the alarms when
needed.
Are you hopeful about security, cybersecurity stuff in 2018?
Things get better.
Are they getting worse?
What do you think?
I think they're getting better.
I mean, it's so hard with all the legacy stuff, right?
Like that's, that's really where the problems come in is when you have this product that's
been around for 10 years and rewriting it from scratch is super expensive.
And that's a hard sell until you actually get owned.
Right.
And then it's like, okay, well, this is, this is going away in the year.
Right.
And so, you know, I think it's still okay.
The thing that I probably worry about more is if quantum computing, you know, breaks SSL
or something like that.
Like, I think that's probably the bigger risk to like the overall internet.
Like, cause if you don't have SSL, like, I don't, I don't know what you do at that point.
Yeah.
Economies come to a halt.
I just try not to think about it too much.
Yeah.
That's like thinking about the 9.7 magnitude earthquake.
It's like, well, there's nothing we can do about it.
So let's just hope that doesn't happen.
Yeah.
I feel like I'm not super pessimistic about quantum computing breaking the internet because
I feel like when that breakthrough happens, it's, there's going to be like one or two
quantum computers in the world.
And there's going to be a long time before that becomes something in the hands of standard
people.
And maybe by the time it's something in the hands of standard people, we'll have something
that is, we'll come up with another math problem that will solve that for us.
You know what I mean?
Like, you know.
Yeah, exactly.
Like TLS three, that's quantum safe, you know, encryption or something like this.
But yeah, that is, that is a little, a little concerning, but yeah, it's, I'm generally
hopeful for it, but I don't know.
I think I would say I'm generally hopeful as well.
That's just the one thing that like niggles at the back.
of my mind.
I'm like, Oh, that would be really bad.
This is like, this is like apocalypse, you know, fiction level stuff right here.
But hopefully, yeah, it is.
I mean, we've had, we've had plenty of those kinds of problems that we've, I don't know
if we've had problems like that.
We've been able to weather a lot of things already.
So I'm not, I'm not super worried about it.
And I think, I think we'll always have something that we can, that we can add on top and, and
weather the next storm.
So it would definitely send people scrambling and it would make the front page of a lot of news.
Yeah.
If it happened though.
Yeah.
Cool.
All right.
Well, I guess we'll leave it there for now.
We could talk about this last part for a whole nother episode.
I'm sure.
So before, before we move on though, let me ask you the final two questions.
When you write Python code, what editor do you use?
I use Vim.
I keep trying to, if I were going to switch to an IDE, it would be PyCharm.
They have the best Vim integration I've ever, or Vim emulation that I've ever used.
And they have a lot of other cool tools, but I hate not having it in my terminal.
So I stick to Vim.
Yeah.
I think a lot of people like the, it's just always around everywhere.
So I'm going to use that, but I hear you about PyCharm.
Okay.
And then notable PyPI packages.
Obviously you could install Hubble from.
Yeah.
PyPI, but you're probably better off getting one of the prebuilt binaries, right?
I definitely recommend using the prebuilt binaries because that way you get OS query, you get all
these different things.
I have been lax and like, there's no actual reason that Hubble shouldn't be on PyPI and
I just haven't done it yet.
So look for it in the near future, hopefully.
Cause I just, I just need to do it because it's not that hard, but yeah, technically you
can source install Hubble without any difficulty.
So yeah, cool.
And so another one besides Hubble, I don't have any others off the top of my head and I love
requests.
Like I know the whole world loves requests, but it's great.
Well, Hubble is my, is my main one right now.
I worked for a long time on SaltStack, which is a pretty good product.
I don't love some of the directions that the company's going, that backs it, but it is
open source Apache too.
So, and it has a great community.
So, check out SaltStack as well.
Right on.
All right.
So final call action, maybe there's a bunch of companies or organizations out there that
are nowhere near as organized as you guys are at Adobe.
They maybe want to start using Hubble.
They want to start putting some of these practices in place.
What do you tell them?
It's never too early to start thinking about security.
That's, I guess, I guess is where I'd go.
Like it's expensive.
I know like to actually start looking at this stuff, you have to like hire security engineers
and stuff.
And you know, how do you have the money to do that?
But if you lose a whole bunch of customer data, it's going to be way more expensive.
So yeah, think about it early use, you know, the tools are so good these days, you know,
Hubble stack included.
I think Hubble is a pretty great tool and it's pretty easy to deploy.
So there are things you can do from the beginning just to do a baseline thing that will just at
least give you something.
Right.
And so just start.
That's all.
Sounds good.
All right, Colton.
Thanks for being on the podcast.
Yep.
Thanks so much.
Yep.
Bye.
This has been another episode of Talk Python to Me.
Our guest on this episode was Colton Myers, and it's been brought to you by Linode and Rollbar.
Linode is bulletproof hosting for whatever you're building with Python.
Get four months free at talkpython.fm/Linode.
That's L-I-N-O-D-E.
Rollbar takes the pain out of errors.
They give you the context and insight you need to quickly locate and fix errors that might have
gone unnoticed until your users complained, of course.
Track a ridiculous number of errors for free as Talk Python to Me listeners at talkpython.fm
slash Rollbar.
Want to level up your Python?
If you're just getting started, try my Python jumpstart by building 10 apps or our brand new
100 days of code in Python.
And if you're interested in more than one course, be sure to check out the Everything Bundle.
It's like a subscription that never expires.
Be sure to subscribe to the show.
Open your favorite podcatcher and search for Python.
We should be right at the top.
You can also find the iTunes feed at /itunes, Google Play feed at /play, and
direct RSS feed at /rss on talkpython.fm.
This is your host, Michael Kennedy.
Thanks so much for listening.
I really appreciate it.
Now get out there and write some Python code.
I'll see you next time.
Thank you.