When you think of data science, Jupyter Notebooks and associated tools probably come to mind.
But I want to broaden your tool set a bit and encourage you to look around at other tools that
are literally at your fingertips, the Terminal and the Shell command line tools. On this episode,
you'll meet Joran Janssen, who wrote the book Data Science on the Command Line. And there are a bunch
of fun and useful small utilities that will make your life simpler that you can run immediately in
the terminal. For example, you can query a CSV file with SQL right on the command line.
That and much more on this episode 392 of Talk Python to Me, recorded November 28th, 2022.
Welcome to Talk Python to Me, a weekly podcast on Python. This is your host, Michael Kennedy.
Follow me on Mastodon, where I'm @mkennedy and follow the podcast using @talkpython,
both on fosstodon.org. Be careful with impersonating accounts on other instances. There are many.
Keep up with the show and listen to over seven years of past episodes at talkpython.fm.
We've started streaming most of our episodes live on YouTube. Subscribe to our YouTube channel over
at talkpython.fm/youtube to get notified about upcoming shows and be part of that episode.
This episode is sponsored by Sentry. Don't let those errors go unnoticed. Use Sentry. Get started
at talkpython.fm/sentry. And it's brought to you by Microsoft for Startups Founders Hub.
Check them out at talkpython.fm/foundershub to get early support for your startup.
Transcripts for this episode are sponsored by Assembly AI, the API platform for state-of-the-art
AI models that automatically transcribe and understand audio data at a large scale. To learn more,
visit talkpython.fm/assemblyai.
Yerun, welcome to Talk Python to Me.
Hey, thank you. I'm very happy to be here.
I saw your book and the title was Data Science at the Command Line. I thought, okay, that's
different. You know, there's a lot of people that talk about data science tools and Jupyter
Labs, amazing. And like, if you look over the fence, like our studio and those kinds of things.
And yet, so much of what we can kind of do and orchestrate and create as a building block
happens in the terminal. And bringing some of these data science ideas and some of these
concepts from the terminal to support data scientists, I think is a really cool idea. So
we're going to have a great time talking about it.
Yeah. Yeah. I love to talk about this. And yeah, you're right. I still find it an interesting
juxtaposition of these two terms, data science and the command line. One being, well, nowadays,
let's say 10 years old, at least the term is. And the other one, the command line is over 50 years old.
The command line is, it's ancient in computer terms, right? It's one of the absolute very first
ways of interacting with computers. You've got cards where you programmed on paper,
and then you had the shell, right? Right after that.
Exactly. Before there were any screens, really.
Yeah. When we all, when computers were green, they were all green. It was amazing. So I'm looking
forward to diving into that. And it's going to be a lot of fun. And I just want to also put out there
for people who are like, but I'm not a data scientist. So should I check out? I actually
think there's a ton of cool ideas in there from just for people who do all sorts of Python and other
types of programming. It's not just data scientists, right?
No, absolutely. And I mean, even, I mean, I don't really care much for titles, but even when you're
an engineer or a developer, you would be surprised if you really think about how much data you actually
work with. I mean, just log files on the server. Yeah.
That's data too. So there are still a lot of opportunities to use the command line, even if
you don't consider yourself to be a data scientist per se.
Yeah, I totally agree. All right. Let's start with your story. How'd you get into programming
and data science and Python? I know you do Python and R and some other things. So how'd you get into
programming? Yeah, it actually started when I was about 12 years old. We got this old computer. It
was already old by then, 286. And I opened up this program. I wanted to write a story. And I was just
typing. I was journaling. Then I got all these error messages. Turns out the program that I had opened
was QBasic. And I didn't really like what I had to say. And then I started reading the help. And then
I realized like, hey, I can make this computer do things. It just needs a particular language.
And that's really how I got into programming. Yeah. And then of course, there's a whole range of
programming languages that then come by. Visual Basic at a certain point. Pascal, Java, C.
And you know what? I've forgotten most of it. So if this sounds intimidating,
please don't worry. But yeah, nowadays, Python plays a big role in my professional career. Also R,
right? And those two happen to be the most popular programming languages for doing data science and
JavaScript, obviously, when you're doing some more front-end work.
JavaScript finds its ways into all these little cracks. You're like, why JavaScript? Come on. I was
just looking at programmable dynamic DNS as a service. And the way you program it is you, I know,
and you jam in little bits of JavaScript to make a decision on how to like route a DNS query. I'm like,
JavaScript.
Oh, yeah. I'm now using Eleventy, which is a static site generator. And ironically enough,
it uses JavaScript.
Yeah, sure.
So yeah.
That's fantastic. I've heard really good things about Eleventy. And I just started using Hugo,
which is also a static site generator, but that one's written in Go. And I just decided I care
about writing in Markdown and I want a static site. And I don't care as long as I run a command on the
terminal. Actually, I want to tell the story a little bit about sort of coordinating over the
shell for some of these static site things. But I decided I don't care if it's the guts of it are in
a language that I can program. It's a tool. If it's a good tool for me, I'm going to use it.
Okay. So that's how you got into programming. How about day to day? What are you doing these days?
Yeah. So at this very moment, as we're recording this, I have my own company called Data Science
Workshops, where I give training at companies to developers and researchers and occasionally managers.
But I have decided to stop with that. Okay. So in a couple of weeks, I'll actually join another company.
So in the past six years, I have, we can talk about how that company came about. And it's probably
related to, it's related to everything, of course. But, but I just want to say that this is actually the
very first time that I'm talking about this, but I'm going to be a machine learning engineer.
Okay. Two reasons why I decided to, to stop with, with my own company is that first of all, I really
miss working with people, working with colleagues. Yeah. And secondly, I miss building things. So
that's why I'm, I'm joining, well, January 1st, I'm joining Xomnia, a consultancy based in Amsterdam,
the Netherlands. Excellent. Well, that sounds, that sounds really fun. I also run my own company and I'd
really enjoy it, but I completely get what you're saying. It's sometimes it's nice to be with a team
and you also, it makes you learn different skills or hone different skills to show up at a client
company where they've got, you know, a million requests in an hour trying to answer something,
which machine learning versus, you know, doing some research and talking about how to improve the
shell, right? These are very, two very different jobs. And so it's, it's cool to sort of mix,
mix up the career across those. Say again? It's great to mix up your career and do,
do some of both, right? Because if all you do is work at a consultant, you'd be like,
I can't wait to start my own company and do something else. Right, right, right. And you know,
the company was just, just happened. That's actually thanks to the book that I wrote a long time ago.
I, once it was done, the first edition that is in 2014, and we're talking about data science at the
command line here, I was asked to give a workshop and I'd never given a workshop before, but I was asked by,
a games company in Barcelona to give a one day workshop. And I liked it. And I liked it so much
that I started doing this more often. Yeah. So I decided to do this full time. So I didn't choose
the company life, the startup life. Although you can't really think of it as a startup, but yeah.
Yeah, sure. Yeah. These things happen.
The independent, the independent life. That's right.
Yeah, exactly.
Yeah. Cool. All right. Well, let's talk about the terminal. People on Windows might know it as the
command prompt. Although you, as I would also recommend that people generally stay away from
the command prompt in, at least for some of these tools, but we do have the Windows terminal,
which is relatively new and much, much nicer, much, much closer to the way Mac.
Yeah. Well, there's the PowerShell, but then there's also a new Windows terminal application.
And then it can even do things like bash into Windows subsystem for Linux, right? So if you
wanted to use some of these tools, you could fire up Windows subsystem for Linux, and then you would
literally have the same tool chain because it's just Ubuntu or something.
Oh, right. Yeah. I mean, I have, I'm familiar with WSL, but I haven't tried out this new Windows.
Yeah. The new Windows terminal is pretty nice. Well, let me see if I can pull it up for everybody.
Windows terminal, but yeah, it's just in the, the Windows 11 store, I guess you call it. I don't
know, but it's, it's a lot closer to a lot closer to the other tools. So if you're on Windows,
you owe it to yourself to not use cmd.exe, but get this instead. So what I want to talk about just
real quickly to set the stage is I just went through a period of, oh, my computer has been
the same setup for a couple of years. It's getting crufty. I'm going to just format it,
not restore from some backup, but format it and reset up everything. So it's completely fresh and
like better. Cause I really made some mistakes when I first set it up. Now it's better, but I opened up
the terminal and it's this tiny font, dreadful white background with white, with black text. And
it has some old version of bash. And so I kind of wanted to get your thoughts on like,
what do you do to make your terminal better? Right. You probably do something. You probably
install some extras and other things to make your experience on the terminal nicer.
Yeah. I'm guessing you're on macOS then? Yes. I do macOS and I do a Linux for the servers.
And I think some combination thereof is pretty common for most of the listeners.
So for macOS, the biggest gain you get when you install iTerm. Okay. A different terminal,
right? The application that would launch your shell.
The macOS term. Yeah. The terminal emulator. I, what do they call it? The macOS terminal replacement.
Yeah. This is, I'd say the most popular one on macOS. There are others, but yeah, that's what I start
with. You mentioned the shell, which is it still bash? Is that still the default one on macOS?
Yeah. It's still bash by default. Yeah. Yeah. And I think it's an old version of bash.
So yeah, there are other shells out there. The Z shell is quite popular, largely compatible with bash.
I've heard good things about fish. Yes. Fish is good. Yeah. Yeah. Which actually it's not really
POSIX compliant as they say. So, so it's quite different from what you might get from bash or the
Z shell. But from what I've seen, the syntax, especially a looping might appeal to the Python
developer out there. It's closer to Python, but I haven't tried it myself.
There's also the conch shell. Is that how you say it? X-O-N dot. If you're willing to give up
POSIX, then this is like literally Python in the shell. You can just type like import JSON and do a
for loop, right? Yeah.
I've never, I've not gone this far. I'm still, I'm on the Z shell side of things. I really like how that
works. But if you really wanted to embrace the sort of Python in the shell.
Exactly. It's this trade-off of how far do you want to go? How much do you, do you want to deviate from
what is then considered to be the default, right? Because you mentioned you also work a lot on servers.
And there you're then presented with a completely different shell, perhaps, and set of tools.
It's a trade-off. And also how much time do you really want to spend customizing this? Because
yeah, our time is precious.
Yeah. Yeah. And William out in the audience says for the, for the Windows people,
Oh My Posh, which, have you done stuff with Oh My Posh? This is also really nice.
So I guess Posh is for the, oh no, any shell. So not just the PowerShell.
Yes. This, I think it came out of the PowerShell. So the Posh part, I think it originally was for that,
but I use this with Z shell and Oh My ZSH together. And it's basically that controls my prompt and Z,
Oh My Z shell is like all the plugins and, you know, complete your Git branches type of thing.
But yeah, this, this is really, this is really pretty neat too. Works well for, for Windows people.
I'd say then indeed the, the, so if your terminal is one thing where you can get a lot of benefit from
customizing your prompt so that it gives you a little bit more information and context of where
you are or what your state, in which state your Git repo is in or which virtual environment you're
working, that can be helpful too, because that is something that you lose easily when you're working
at the command line is, is context.
Right. I ran this command and it's not working because actually I forgot to activate the virtual
environment. So it doesn't have the dependencies or the environmental variables that I set up in that
virtual environment. Right.
Exactly.
Let me give one more shout out for one other thing while people are thinking about making their,
their stuff better is nerd fonts.
I'm always eager to learn these things. There is so much out there.
So nerd fonts, if you're going to get like, Oh, my posh and some of these other extensions that you
want to make your shell better. So many of them depend on having what are called nerd fonts. Because
if you look at say on the, Oh, my posh page, there's like these arrows with gaps in them. I mean,
what font could possibly have like a Git branch symbol and as these like connecting arrows that have
colors and are woven and all that stuff is nerd fonts. So if you're going to try to run them,
download and install one of these nerd fonts, and then those will work. Otherwise they're like
those, I don't understand Unicode square blocks, you know, like when emojis go bad.
Oh, you still got to install individual fonts. Yeah. So, so it's kind of like you take Consolata or
something or some other font and it's patched with these additional.
Some, yeah, something like that. Yeah. So it does, you only need one, but you have to set your terminal
to one of these to make a choice, set it to one of them and then it'll work. But if you don't,
then you'll, you'll end up with just like these, these, a lot of these extensions don't work.
This portion of Talk Python, I mean, is brought to you by Sentry.
How would you like to remove a little stress from your life? Do you worry that users may be
encountering errors, slowdowns, or crashes with your app right now? Would you even know it until
they sent you that support email? How much better would it be to have the error or performance details
immediately sent to you, including the call stack and values of local variables and the active user
recorded in the report? With Sentry, this is not only possible, it's simple. In fact, we use Sentry on all
the Talk Python web properties. We've actually fixed a bug triggered by a user and had the upgrade ready
to roll out as we got the support email. And that was a great email to write back. Hey, we already saw
your error and have already rolled out the fix. Imagine their surprise. Surprise and delight your
users. Create your Sentry account at talkpython.fm/sentry. And if you sign up with the code
Talk Python, all one word, it's good for two free months of Sentry's business plan, which will give
you up to 20 times as many monthly events as well as other features. Create better software, delight your
users and support the podcast. Visit talkpython.fm/sentry and use the coupon code Talk Python.
Yeah. So when it comes to customizing your shell, then if you still want to talk about that.
Yeah, yeah, yeah. Let's keep going.
Right. One of the things I think everybody does most often is navigating around. So moving from one
directory to another. And it can be quite cumbersome to keep on retyping all these long and deeply nested
directories. So there are a number of solutions that can help with that. I use FASD. Okay.
So that keeps track of what you've been visiting most often, most recently. So it also, I don't think if,
I wonder if that one also allows you to set bookmarks. That's what I used to do. I would keep,
I would have this set of custom shells, shells functions, which actually made it into a plugin,
about nine years ago into OhMyZSH. So if you, if you have OhMyZSH and the jump plugin is still
in there. Yeah. That's a, I see. Yep. Yep. So you can just jump around. I see.
Yeah. You would say, you would say mark, mark this directory under this alias, although it's not really
an alias, but it's like a bookmark. And then you say, okay, jump to this directory. So that really helps.
Right. So maybe the source directory for Talk Python, I would just mark it as TP. And I could say on the,
on the terminal, I could say J space TP. And it would take me this super long, complex directory,
just bam, you're there, right? Exactly. So I like it. Okay. I might need to try this out. And it comes
with OhMyZSH? This one? No, this one, it doesn't. It's a separate tool. Okay. I believe.
Although it might even be a plugin. By now, I don't even know. It's been, it's been a long time since I
installed it, but FASD. That's what you want to look for. Okay. Very, very cool. I have one that I
use a lot called McFly. Have you, have you seen McFly? No. So it's similar. And what you do is,
you know, if you type control R, it'll give you reverse incremental search or whatever that is.
And I'm, so this overrides that. So if you type control R, it brings up an Emacs, like autocomplete
type thing that has fuzzy searching. So you could type SSH and then like part of a domain name,
and it would find you type SSH, you know, root at some that, that, that, that domain name. And it'll,
it'll give you a list of like all these smart options looking through your history.
Yeah. Yeah. That's amazing. And even now, as we talk, I've learned like a dozen new things.
One thing I have noticed though, is that, you know, you may, the next time you're installing,
you're setting up your system, you may feel very productive and, and lead like,
right. When you're installing all these tools, but you still got to make use of them, right?
Yeah.
You got to turn that into some kind of a habit. And what I have noticed for me, at least what works
best is to just take it one tool at a time, make a little cheat sheet for yourself on a piece of paper,
and just see if that, if you like that tool, if you can, you know, if you get any benefit from it.
Yeah, absolutely. So related to this actually is, is the concept of aliases, right? In a more generic
sense, in the pure shell sense that you can define an alias that would then be expanded into some command
with zero or more arguments. Yeah. So if you have, if you have commands that you would often type like
a LS for listing your, your files, and you have all these arguments that you don't want to keep on
typing, then aliases is the way to go. I go crazy with aliases. I absolutely love this. Yeah, I have
probably 100, 150 aliases in my RC file. Oh, that's nice. Yeah, that's nice. So at some point,
I, what you, well, what you may have done is go through, through your history and then see how often
you use these aliases. Yeah. That's always a fun thing to do. Yeah. For me, it's kind of frustration.
I'm like, God, I want to do this. Or, you know, I've got to remember, I got to type, oh no, I got
to go into this directory. And then I got to first type this command, and then I can do this other
thing. So for example, we talked about the static site generators. So one of the things I have to do
in order to create new content and see how it looks in the browser is I have to go to a certain directory,
directory, not where the content is, but a couple up, run Hugo dash D server there, and then it'll
auto reload. And as I edit the markdown, it'll just refresh. So instead of always remembering how to find
that directory and then go into the right sort of parent directory and run it, I just now just type
Hugo right. And that's an alias. And just, it does that. Boom. It just, it just pops open and it's okay.
It's, it's running. I do my thing. I'm going to, then I got to do a whole bunch of automation in Python
on top of it and then build it and ship it to the Git and push it for a continuous deployment.
Now I have just Hugo publish. Boom. And these are all like aliases. The other thing you talked about,
a single commands is maybe talk about chaining commands and multiple commands.
Yeah. Because you just mentioned automation in Python. And then I, of course, immediately go like,
hmm, what's going on there?
Yeah. So I've got a couple of, I guess they're go commands because they're Hugo. And then I've got
some Python code that generates a tag cloud and then a Git command that'll publish it. So it's like,
Hugo, Hugo, Git. No. Hugo, Git. Hugo, Python, Hugo, Git is that all in one alias, right? Which is,
is beautiful. Oh, nice.
Yeah. It's beautiful. You know, I don't know if we've exactly, I guess I opened a little bit talking
about your book, but one of the really core ideas of your book is that the shell can be the integration
environment across technologies like Go, Python, and Git.
Exactly. Exactly. The command line doesn't care in what language something has been written. It's like a
super glue or duct tape, more really, that binds everything together.
Yeah.
Yeah. To a certain extent, right? Like duct tape.
Yeah. Well, it's, it's a, you know, loosely bound, but it's, there's a ton of flexibility in there. And
if you think, well, I really just want to do these four things, maybe that would be a macro in Excel,
or some kind of like scripting replay in Windows. But this is, it's on the terminal programs can run
it. You can run it. It's clearly editable. It's not some weird specific type of macro, right? You're
like, I want to do these four things. I just type thing and go, I'm sure many people know,
but if you have multiple commands, you want to run one than the other, you can just say
ampersand, ampersand between them. And it'll say, run this first thing, then run the other.
Those are independent. You can also pipe inputs and outputs between them. Right. I see that.
That's correct.
You've got some really interesting ways to do that multi-line stuff in your book as well.
Yeah. Well, yeah. So it depends on what kind of tools you want to combine, right? So you,
you just mentioned a double ampersand. So that should be used when you only want to run the
second command when the first one has succeeded, right? If you want to run the second one, regardless
of what the first one did, you can just use a semi-column. Or if you only want to run the second
command when the first one failed, there might be a situation where you want to do that. You can use
double pipe. So for four. Interesting. Okay. And then, yeah. And then you just mentioned piping
and that's, well, a whole nother story. That's when you want to use the output from the first command
as input to the second command. And this is where, or data again, comes into play. And this is, so you just
also mentioned macros, right? Another way to think of them are our functions that you then combine.
Yeah. Incredibly powerful, but that goes a little bit beyond then of course you should be working with
commands that produce some text that you want to then further work on. Yeah. You also talk about
creating bash scripts, which is pretty interesting. I think many people probably know about that or shell
scripts, .sh files. I guess it could be zshell scripts as well. Yeah. So you gave an interesting
presentation back at the Strata conference and you had a lot of fun ideas that I think are relevant
here. So maybe let me just throw out some one-liners and you could maybe riff on that a little bit.
Okay. Yeah. Yeah. Sure. One of the reasons you said you gave 50 reasons that the shell was awesome.
And I want to just point out a couple, highlight a couple, let you speak to them. So you said the
shell is like a REPL that lets you just play with your data. We know the REPL from Python and also
from Jupyter, but I never really thought of the shell as a REPL, but it kind of is, right?
Yeah. I think that the shell is the mother of all REPLs. The read, eval, print loop.
Yep. Right? Having this short feedback loop of doing things and seeing output and then elaborating on
that I think that is tremendously valuable. And Python users, of course, may recognize this from
Python itself, right? If you just execute Python, you get a REPL, IPython or Jupyter console. And to a
certain extent also, Jupyter notebook or JupyterLab is there are some similarities there where you again,
have this quick feedback loop. And it's a very different experience from writing a script from top to
bottom or starting at the top and then executing that script from the start every time you want to test
something. So yeah, it's a different work, different way of working. And I'm not saying one is better than
the other. But what I do want to say is that there are situations where having such a tight feedback loop can
be very efficient. Yeah. Especially in the exploration stage, right? Yeah, exactly. Once you go to production,
right? Once you whatever that means, right? Once you want things to be a bit more stable,
you don't want to just use duct tape, but you want to use a proper construction. Then, then yeah,
then, of course, the command line can have different roles there. Yeah. Yeah. But it's kind of the,
the, the, the rad GUI, the rapid application development GUI, but for data exploration, right? These,
these REPLs and you know, that's, that's probably why Jupyter is so popular. It just lets you play and see and
then try. And it just was that, that quick feedback loop is amazing. Another reason said,
it's awesome. Close to the file system. Yeah. I mean, in the end, it's all files,
right? Whether you're producing code that lives somewhere, it's in a file, or whether you're working
with images or log files that get written to something, or you have some configuration, it's all
files. And we got to do things with these files. We have to move them around. We have to rename them,
delete them, put them into Git. Yeah. So you want to be close to your file system. You don't want to be
importing a whole bunch of libraries before you can start doing things with these files.
Also, when you're doing data science, often it starts with this kind of ingest and understanding
files, right? CSV or text or others. Yeah. I mean, I sometimes try to immediately do read CSV in Pandas,
but then, you know, very often I get presented, I get some Unicode error, or it turns out it's the comma
is not the file, is not the limiter being used. And yeah, you can do that in a sort of trial and error
way. You can fix that. But it really helps to just being able to look at a file as it is, no parsing,
just boom, there's my file. And then, yep, once you're comfortable, once you're confident, like,
okay, this is what my file looks like. This is its structure. Then of course, you can always move on
to using some other package like Pandas. Okay. Another one that you've said,
another recommendation you had or sort of way for playing with this was to use Docker. I don't know
how many people out there who haven't done this for are really familiar, but basically when you start
up a Docker image, you might say dash it bash or ZSH. And what you get is just you get a basic shell
inside the Docker container. But in that space, then you can kind of go crazy and do whatever you want
to the shell and try it out, right? Exactly. Yeah. So there are two scenarios that I can think of. So
when you're just starting out with the command line, it's a very intimidating environment. And it's quite
easy to wreck your system if you're not careful. So being inside an isolated environment that
is sort of shielded off your host operating system can be comforting. So that's one recommendation
that I would say that why I think you should use Docker. And the other one is reproducibility. Also in
Python, right, we're dealing with packages that get updated, that get different version numbers,
where APIs change. And being able to reproduce a certain environment so that you get consistent results
is also very valuable. Yeah. And I'd like to sort of highlight the converse as well. You said playing
with Docker containers is a cool way to experiment with the shell. If you care about Docker containers,
you need to know the shell to do things to it. Because you might think, oh, I'm just going to make a Docker
file. I don't need to know the shell. Like what goes in the Docker file, a whole bunch of commands that
many of them look like exactly what you would run on the shell. You just put it in a certain location
or as a command argument to some configuration thing in there. And so you really, if you're going to do
things with containers, the way you speak to them is mostly through shell-like commands.
This portion of Talk Python to me is brought to you by Microsoft for Startups Founders Hub.
Starting a business is hard. By some estimates, over 90% of startups will go out of business in just their first
year. With that in mind, Microsoft for Startups set out to understand what startups need to be successful and to
create a digital platform to help them overcome those challenges. Microsoft for Startups Founders Hub was born.
Founders Hub provides all founders at any stage with free resources to solve their startup challenges.
The platform provides technology benefits, access to expert guidance and skilled resources, mentorship and networking
connections, and much more. Unlike others in the industry, Microsoft for Startups Founders Hub doesn't require startups to be
investor-backed or third-party validated to participate. Founders Hub is truly open to all.
So what do you get if you join them? You speed up your development with free access to GitHub and Microsoft
cloud computing resources and the ability to unlock more credits over time. To help your startup innovate,
Founders Hub is partnering with innovative companies like OpenAI, a global leader in AI research and development,
to provide exclusive benefits and discounts. Through Microsoft for Startups Founders Hub,
Founders Hub is no longer about who you know. You'll have access to their mentorship network,
giving you a pool of hundreds of mentors across a range of disciplines and areas like idea validation,
fundraising, management and coaching, sales and marketing, as well as specific technical stress points.
You'll be able to book a one-on-one meeting with the mentors, many of whom are former founders themselves.
So one of the cool tools that you had in that presentation was you talked about explainshell.com.
Yeah. What is this?
Well, you can try out. So what you see here on the screen is explainshell.com and it will break down
a long command and start explaining. So it will, it will, what I think the authors have done is they have
used all these manual pages and extracted bits and pieces that they then present to you in a, in an order
that corresponds to the command that you're pasting into this. So if you see, you know, on Stack Overflow,
you see this, this incantation and you're like, all right, what does it mean? And you don't want to go through the manual page yourself.
Right. Okay. So what's, what does dash F mean? What is this X, Z, V, F for the tar command mean?
Then explainshell can do this trick for you.
Yeah. It's amazing. When I first thought, I thought, okay, well this, what this is going to be is this
is going to be like the man page. So if you type LS, it'll show you a simple list directory contents and
you click on it, it'll give you additional arguments you can pass. But you could then say, like you said,
you could say dash L and it'll say the LS means list contents. The L means use the long listing format.
And you're like, oh, okay, hold on. What if I said, get, get checkout main. And you'll say, okay,
well get checkout does this. And then main it'll actually parse it apart. And there's some really
wild examples on here that like right on the page that are highlighted on the homepage of that site.
You click it and boom, it gives you this cool graph of like, what the heck? It even shows like
the ampersand at the double ampersand and the double or combining, as you mentioned before.
Yeah, it is. It is. It's really useful, especially when you're just, you know,
getting started with the command line and you're overwhelmed, like we all are in the beginning,
and sometimes still are then, you know, adding adding some context like this really helps.
I once wrote a utility that allowed you to use explain shell.com from the command line. So you would
just, you wouldn't leave the command line. I don't think it works any longer. But yeah, that was a fun
exercise. Yeah. Oh, yeah. Very neat. One of the things that I learned was parallel.
Oh, yeah. So tell us about parallel. Like this is a command you can run on the terminal. And it
sounds like it does stuff in parallel. That sounds amazing.
Yeah. Yeah. Like the name implies parallel is a tool. And we're talking about GNU parallel here.
There's another version out there that is similar, but different. GNU parallel, this tool that doesn't
doesn't do anything by itself, but it multiplies. It's a force multiplier for all the other tools.
So what this tool is able to do is will parallelize your pipeline. It will be able to run jobs on
multiple cores and even distribute them to other machines if you have those available. Right. So,
Michael, you mentioned you're working on a server. Well, if you can SSH into other servers as well,
you can leverage those. That's something that GNU parallel can do. The way it works is that you feed
it a list of something. Could be a list of file names. Could be a list of URLs. Could be your log files.
If you can then think of the problem that you want to solve. If you can break it down into smaller chunks,
then GNU parallel might be able to help you out there. So these jobs should be working independently
from each other. Yeah. There can be, yeah, it's nearly impossible to have those two jobs communicate
with each other. But let's say you have for your blog, right? In Hugo, you have a whole bunch of
ping files that you want to convert to to JPEGs. WebP or something. Yeah, sure. Yeah. I mean,
it's a bad example because this particular tool that I would then use already supports doing multiple
files. But let's just assume that this tool can only handle one file at a time. Yeah. Then you would
specify your command and then at certain places where necessary use placeholders. Yeah. So,
okay, this is where the file name goes and this is the file where the file name goes with a new
extension. So it's one of my favorite tools really. Yeah. That's fantastic. So for example, if you had
a bunch of web pages and you wanted to compute the sentiment analysis, right, as a data scientist,
you want to download it, compute the sentiment analysis, and then save that to a CSV or pin it to a CSV.
Yeah. You know, maybe somebody gave you that script and it's only written to talk to one thing and you
don't want to rewrite it or touch it or get involved with it, right? This is your way to unlock the
parallel of something, right? Yeah. In fact, let's talk a little bit more about this because I think
this is an important point in that I'm sure that we've all come across when we're working in Python
and you're thinking like, okay, I can speed this up. I want to do things in parallel. You know what?
I'm going to do multi-threading or what is it that you use these days in Python?
Yeah. Async and await maybe if it's I/O or something like that. Yeah.
You've got your pool of workers or I don't know. Basically, you're programming it yourself from the
ground up. Right. Multi-processing potentially.
Probably the closest. Right. Right. Right. Right. Right.
The trick then is to realize that there is already a tool out there that can do that for you. All that you
need to do is make sure that your Python code becomes a command line tool. Yeah.
And we can talk a little bit more about that, but there are just five, six steps needed to make that
happen. Once you realize that, then you can start turning existing Python code into command line tools
and start combining it with all the other tools that are already available, including parallel.
Yeah. It's awesome. I think it's a really cool idea because maybe the person working with the code
doesn't understand multi-processing and thread synchronization and all these tricky concepts.
Like, just give me a thing that does it once with command line arguments and I got it.
it. You know, like you, or you've, you picked it up from somewhere out in the audience. The question
is, is there a GIL associated with this? And I mean, technically yes, but it's not interfering with the
computation because it's multiple processes. It's not threads within a process. Right. So it should be
able to just run. Yeah. There will be one GIL per Python process. Right. Yeah. That's right. And so it
doesn't matter because if you say there's five jobs, you have five processes, right? There's no contention there.
Yeah. Yeah, absolutely. All right. Oh, so yeah. Let's talk a little bit about this idea of turning Python
scripts into command line tools. Yeah. I think that that's really valuable for people. It is. And
we can then put it in the show notes. I might have already given a talk about this. I'm actually not
sure if it's publicly available. Anyway, there are only a couple of steps and it's not that difficult.
So first of all, let's assume that you have some Python code out there. Yeah. You have it in a
file and let's just for simplicity's sake, assume it's a single file. Right. So what would you then
need to do to turn this into a command line tool, something that can be run on the command line. So
the way that you can currently run this is by saying, okay, Python, and then the name of the file,
right? That doesn't really sound like it's a command line tool. So the very first thing here then is to
add one line at the very top that would then start with a hash and an exclamation mark or a
hash bang or a shebang as it's called. These are two special characters and they instruct the shell.
This can be executed. What is the binary that's going to do the executing? Right. Yeah.
Executing. Right. Yeah. Exactly. That's what then would come after that. So you would have hash bang,
and then it would point to the Python executable. Yeah. Right. There's some details there.
It could be a certain version. It could be out of a virtual environment, potentially. And I think it
could go wherever, right? You don't want to overcomplicate it probably, but like you could point
to, you could point to different versions of Python. You could point, because you, you give it a full path to
executable. Exactly. Exactly. There's some, there's some compatibility issues there, but essentially is
you tell your shell, okay, which program should interpret my code. And that is some Python out
there that you have installed. So that's the first step. Then after you've done this, you no longer
need to type Python anymore because the file itself contains which executable should be, should be run.
But then you'll notice that you don't have the necessary permissions. What you need to do is you
need to enable the execution bit. This would give you as the user permission to actually execute this file.
You do that, of course, with a command line tool. It's called SHMOD, C-H-M-O-D for change mode. And then
U plus X, the name of the file, right? These details are, if you're really interested, one place where you can
find them is in chapter four of my book, data science at the command line, which you can read for free.
Okay. But let's say that you've enabled these, the execution bit. Now you can,
now you can run it. You would still need to type period and a slash because this file is presumably
not yet on your search path. So your search path is a, is a list of directories where your shell will be
looking for the executable that you want to run. Where is your tool located? Well, it should be
somewhere on the search path. So either you add another path to the search path or you move the tool
to one of the existing directories out there. That's about it for making your code executable. But then you
want to change one or two things about the code itself. Yeah. So one thing to do is look for any
hard coded values that you actually want to be, want to make dynamic, right? These should be turned
into command line arguments. And actually you can take that one step further. If one portion of your file
is doing something that can be done by another command line tool, then consider removing that. For example,
downloading a file. Yeah. There is of course a tool for that on the command line. Why would you then
write this yourself? Of course, there's a time and a place for that, but let's say, okay, a very
contrived example is a Python program that would count words. Yeah. Right. Right. If your code has some
hard coded website. Yeah. I mean, why you would make your tool more generic by getting rid of that hard
code URL and we'll turn it into a command line argument. Okay. Which website would you like to
download or to go one step further is to think, okay, you know what? I don't really care where the
text is coming from. I just want to count words. Give me text somehow. Yeah. Sorry. Just give me the
text. Don't tell me the URL. Yeah. So your tool should then be reading from standard input, which is a special
channel from which you can receive data. And this is also where the piping would come in. Yeah. So you
would first use a tool that would get this text, right? Maybe it's some log file. So you want to count
your errors or it's another website and you want to do stuff to that. It doesn't really matter, but you
would then that would write to its standard output. Yeah. And you would combine the standard output from
the first tool with your standard input using the pipe operator. So that's then basically it for,
I mean, of course, if you want to take this further, you can think about, you know, adding some help,
some nice looking help. Yeah. Think about the arguments themselves. Do you want to use short options or long
options? Exactly. Right. So something like Typer or Click or one of these formal CLI frameworks. Yeah.
Probably really. Python, of course, has ArgParse, but there are packages out there that can really help
you build beautiful command line tools. Typer is one of them. I'm currently using Click. Also,
Click combination with Rich. Mm-hmm . So of course, the author of Rich was on the show a couple of episodes ago.
Yeah. Will McGoogan. Very good stuff. Yeah. Why we're talking about that? You know,
the other thing that's really pretty interesting is the Rich CLI. Have you played with Rich CLI?
Which, oh, oh yeah. Okay. So that's indeed a command line tool in itself that can do a whole bunch of
things. Yeah. You want to tell us something about that? No, I haven't done much with it,
but you can do things like if you install the Rich CLI, then you can say things, there's lots of ways to
install it. You could say like Rich and then a Python file or a JavaScript file or a JSON file,
and it'll give you pretty printed color, you know, syntax highlighted printout. You can say,
Rich, some CSV file, and it'll give you a formatted table inside your terminal with colors and everything
of it understands markdown and like renders markdown. And there's all sorts. So if you're kind of
exploring files and you're happy with Python things and like installing the Rich CLI is a pretty neat way
to go as well. Yeah. It's a nifty tool, but just not to get confused. So this tool is provided by Rich
and it uses Rich to produce, you know, nice looking output. But just imagine that you can write your own
command line tools that would also produce this nice looking output. And for that, you can then use
this package called Rich. Right. In combination, perhaps with things like Typer or Click. And
DocOpt is another way you can go. There are so many tools out there. Yeah, there absolutely are. One other
thing I would like to point out that, so just taking the script and making it executable and put it in the
path, that's kind of a great way to take scripts that you have and make them CLI commands for you.
If you want to like formalize this a little bit more, I recently ran across this project called
the Twitter Archive Parser. And I don't know if you've noticed, but there's a lot of turmoil at Twitter.
And so what you can do is you go to Twitter and download your entire history of like thousands of
tweets or whatever as HTML file and some JSON files, and you can save them for yourself.
But the content of like all of the links are the shortened to.co Twitter short links. And if
Twitter were to go away, you'd have no idea what any of your links you've ever mentioned ever were.
And also the images that you get are the low res images, and you can get the high res images if you
know how to download them. So this guy, Tim Hutton created this really cool utility that you can
down, you can take that downloaded archive and upgrade it to standalone with high res images and full
full resolved links, not shortened links. Pretty cool, right? But if you look at the way to like,
how do you use it? Okay, where does it say this? Not sure where it is. Yeah. So how do I use it? I
download my Twitter archive and unzip it fine. And then I download the Python file to the working
directory. And then I go in there and I type Python that file. Wouldn't it be better if I could just,
you know, it has dependencies that has to install in order for it to run? Wouldn't it be better if I
could just use this as a command? So what I did is I forked this. And I said, I'm going to add a
pyproject.toml to turn this into a package. And then under the pyproject.toml, you say project.scripts,
Twitter archive markdown, Twitter archive images, and you, you map into your package and then functions
that you want to call. And then once you pip install this, these commands become just CLI commands.
And it doesn't matter how that happened long as your Python packages are in the path, which they
generally have to be anyway, because you want to do things like pytest and black, then if you just
pip install this project, it adopts all these commands here, which is pretty cool. Nice. Is it
then necessary to add this bin directory once to your search path? Because it lives, it would live
somewhere on their site packages, right? Yes, exactly. And so if you have a Python installation and you try
to pip install something, you'll get a warning that the site packages are not in the path. So you do have
to do that. And then go one further, you could do pip x. I don't know if you played with pip x. pip x is
awesome. So it'll generate the package environments and install the dependencies in an isolated
environment. And it'll set up the path if you just say ensure path. Then so if you pip x install the
thing with the commands in it, those automatically get managed and upgraded by pip x as just part of
your CLI, which like that's a perfect chain of like a four, but you've got to have a formal package
and like a place to install it from like get or pipe I or whatever. But it's still, it's still a,
like a neat pro level type of thing. I think. Yeah. Yeah. You can take this pretty far,
make it really professional. And before you know it, you start maintaining it for.
Yeah, exactly. Why am I doing PRs on this silly thing? I don't know.
Yeah. But just to clarify, if you say for a one off or a two off, you want to make something that is
reproducible, right? So a reusable command line tool, not reproducible, reusable. You don't really need
any other packages. You can use a sys.argv, right? You import sys and then you have your sys.argv.
And I do that. I do that a fair amount of times. Yeah. It's only for me, I've created an alias. So
it always gets the right argument. There's like, there's no ambiguity. Sys.argv bracket one, let's go.
Exactly. Yeah. Yeah. We've talked a lot about sort of around all the cool things we can do with
the command line. But in your book, you actually talked about a bunch of surprising tools. So like,
one of the things you talked about is obtaining data and you hinted at this before, like you can
just use curl for downloading those kinds of things. But if you get a little bit farther,
like under scrubbing data, you talk about grec, rep, and awk that a lot of people maybe know.
But then if we go a tad further over to say exploring data, then all of a sudden you can
type things like head of some CSV file and it kind of does the same thing as Jupyter. Or there's things
like CSV cut and SQL, CSV, CSV, SQL. Talk about some of these maybe more direct data science tools that
people can use. Right. So let's see then where to begin. You mentioned a couple of tools, right?
The head and awk and grep. Those are, you know, I would consider them the classic command line tools,
right? I would too. Part of core utils, GNU core utils, right? You can, if you have a fresh install,
then you can expect those tools to be present. Yeah. If you're not on Windows. So those tools,
they operate on text, on plain text, and they have no notion of any other structure that might be
present in this data. Say CSV for when you have some rectangular structure or JSON, when you can have
a potentially deeply nested data structure. These tools know nothing about that. That doesn't make
them entirely useless, right? There are ways to work around them, around that issue. But there are nowadays
plenty of tools available that are able to work with this structure. Right. And one of them is actually a
suite of tools. It's called CSV kit. And you can install it as a Python package. Okay. Through pip,
which of course we do at the command line. CSV kit, you say? Yeah, exactly. And then you get a whole bunch of
tools that understand that lines are rows. The first line is a header and all these fields are delimited
by default by a comma. And then you can do things like extract columns or sort a file according to a
certain column. Yeah. So this is more difficult for when you're working with core data utils. And of course,
all of these things you can do in Pandas, and it might even be faster in Pandas as opposed to these
CSV tools, not as opposed to the classic command line tools. But I mean, in order to get started with
Pandas, right, just imagine that you're given this file by your colleague and you're asked to quickly
to sum things together, right? And in order to just get started with Pandas, what are then the things
that you need to do? Yeah, fire up JupyterLab, import Pandas, and maybe a bunch of other things. There is,
of course, also a time and a place for that. Definitely, definitely. I always use the tool that gets
the job done. Don't get me wrong here. But it's just so incredibly powerful to just, if it solves the job,
just whip up a command on the command line using a couple of tools there. If you're going that route,
then CSV kit is not the only suite of tools that you should know about. XSV written in Rust, but yeah,
you shouldn't care about that because the command line doesn't care. It's generally faster. One thing
that CSV kit can do, by the way, and I'm actually kind of proud that I have been able to contribute
that tool to the suite of tools is CSV SQL. And it allows you to run a SQL query directly on the CSV file.
Yeah. So if you are familiar with the SQL, then you can leverage that knowledge directly at the command
line without first having to create a new database and import that CSV file in there and so forth.
All right. So one of the things you can do on the command line is basically just give it,
like, here's a SQLite file database, and now go insert all the things from the CSV file into it.
Here in this example, it has this create table statement. Does it figure that out from the CSV,
or do you need to write that? It figures it out. Yeah. It does some,
it looks at the first, say, a thousand rows and then figure out like, okay, this is a number. This is text.
I see. Yeah. Oh, cool.
But I was actually talking about the other tool and that's SQL to CSV. I always mix those up.
The reverse. Yeah.
Yeah. Yes, exactly. This one. And there it still uses SQLite under the hood,
but you don't need to worry about that. It takes care of the, of all that boilerplate for you.
You just say, okay, you know, select these columns from standard input, order them by this column. This
is the file or I've piped. Yeah. That's cool. Yeah. It's pretty cool. Yeah. I mean, maybe you've got
like some production database and you want to filter out. I just need this table with this particular
query, right? It's like, I only want to focus on my region of this data, give it to me as a CSV file,
and then you can go work on it all you want. You don't have to be connected to the database or near it
or any of those things, right? Potentially, if it doesn't have any sensitive data, you could share that,
right? You would never share the connection string to your database. That'd be insane.
Yeah. Yeah, exactly. Okay. Very cool. So what are some of the other tools? Well, if we go back,
if I go back to the CSV kit, you can see there's some of these you talked about. There's into CSV.
That one takes an Excel, XSL or XSLX and converts it to a CSV just on the command prompt or the terminal,
right? Yep. Yeah. Okay. Also, I should point out that I'm not the, the author of CSV kit,
right? I just contributed a small portion to it because of the ingredients that were already there.
Still proud of it though, but it's being created by many other people.
Sure. Of course. Some other things it has is like, CSV stat and CSV rep. Yeah. A lot of,
a lot of cool command line options to point at these things, right? Let's see. I pulled out, some others.
Rush. So one of the areas that they, the graph, the basically plotting to do, we're basically out of
time, but I want to, I want to talk about two things really quick. Right. The, some of this,
which chapter did you put it under where you have the pictures? Oops. seven. So seven,
visualizing, exploring data and then yeah. So if you, so tell us a little bit about this,
like you can plot stuff in your terminal. Yeah. Yeah. It's kind of crazy. I should say that rush
is a proof of concept, right? It's one of those projects that have a lot of potential, but don't
necessarily have enough users. And I don't necessarily have enough time to maintain it properly, but it does
prove the concept rush the name. I mean, it's for when you're in a rush, it's R on the shell and,
what it does, it, it leverages R under the hood and, for plotting, it leverages a particular R package,
uh, GG plot two, which is the data visualization package for when you're working with R. Yeah.
Kind of the, the sibling or where Matt plot lib is a little bit derived from that, I believe. Right.
Well, well, well, now you're mentioning that actually map, it is very different. It's a map
out. It is very low level and gives you a lot of flexibility, but also requires a lot of work.
Now, if you're, if you want to visualize data in Python in a similar way that GG plot uses,
then I can recommend plot mine. So that's a Python package that is, modeled after GG plot to API,
but, that was a little bit, a little segue there. Now somebody else created a backend for
GG plot that allows you to create visualizations on the command line. What I then did was create this
interface. So something that would translate arguments and their values to the appropriate
function call. And also does a lot of border plate when it comes to reading in the CSV file that you
provide, right? If you were to do this in R itself, it would require, let's say about five lines of code
in order to get started. Right. And then the same also for Python, right? So similar concept, right?
import the appropriate packages or modules, reading in some file and there's all this setup. And you know,
again, that is probably what you want when you want things to be a little bit more robust, but when you
want to get stuff done quickly, yeah, it really helps to be able to do that as a one liner on a command line.
So I make use of all this, yeah, elaborate machinery, you know, in R just to, use that at the command line.
So a beautiful little wrapper around this complex thing, but it hides the complex complexity, right?
Exactly. Exactly.
Yeah. So you can do beautiful, like bar plots. There's a lot of neat stuff in here. I really like this.
It is really nice. And, now that I see this again, I get excited again. There is
definitely potential there, but you know, it's, it's again, yet another open source project that has to be,
uh, maintained. And unfortunately my time is limited like, like everybody else's.
Yeah, of course. Of course. Yeah. All right. The last, last thing we have time for is this
polyglot data science. Tell us a little bit about this. Yeah. So polyglot data science is the idea
that in order to get things done, you might need to use multiple tools, multiple languages really. And,
throughout the book up until then, up until that chapter, we have mainly been focusing on using other
languages from the command line, but this chapter considers the other way around, right? Using the
command line from another language. So there might've been a situation where you're working in Python and
then all of a sudden like, ah, now I got to do this, this regular expression, or I got to do some globing
and, or I got to call, I have to call this, this other tool that is not written in Python, but can be
called from the command line, right? You would maybe use sub process, sub process module for that. These are
situations where you want to leverage the command line, where you want to break out of Python and do parts of
your computation on the command line. And in that chapter, chapter 10, I demonstrate this not only for
Python itself, but also in other languages and tools, including Jupyter lab, where you can pass around,
say a variable as standard input or, and also retrieve the output then so that you can continue working in
Python again with the output. So, and, what is still very interesting to me is that even new
languages and tools somehow still offer a way to leverage the command line. So Spark, Apache Spark,
has a pipe method where you can pass an entire dataset, right? RDD through a command line tool. And that,
I think that is just, it is, maybe it was just a fun little hack what the authors did. I don't know.
I tried to, to view it as a, as a compliment, like, okay, some, sometimes we just need to go back to the
basics and, and use the command line because once you're there, you're back in this environment where
you can use everything else. So everything we've spoken about so far is now accessible as a command,
be it go or Python or your own script or whatever. Exactly. So let's say,
you've written this, you've come across this really nice tool, but it's written in Ruby. Oh no. What
you're gonna, what are you gonna do? Are you gonna all of a sudden become, you know, involved into Ruby?
No. Assuming that this tool can be used from the command line, you can of course, relax, just use the
sub process module and still you incorporate that Ruby tool into your own script. That's the idea.
Yeah. I do want to maybe point out just really quickly here, like this has got a little bit of a,
a little Bobby tables warning asterisk by it, you know?
Yeah. Yeah. Yeah. Right. So for example, one of the things that's awesome here is I could run
Jupyter console as you show. And then if you say exclamation mark command that pumps it straight to
the shell. So you could say bang date, and it would tell you the day you go a bang pip install --upgrade
request and that'll go and execute that command. Don't do that with user input. Right? Who knows
what they're going to do. You can also do that within Jupyter notebooks you point out, right? So you can do,
what is it? Percent percent bash and then some interesting complicated thing there, right?
That's pretty. Yeah. Yeah. That's indeed the, the magic command that you can use in, in Jupyter notebook.
Right. And then higher cell is bash. Yeah. And so then you take what's left of that and then you head
over to explain shell and figure out what the heck it means. Yeah. Maybe do that before you run it. Yeah.
Yeah. That's a good idea. And then also in, in Python using sub process is something that I've done
several times. I need to automate generating some big import of say 150 video files across a bunch of
directories to build a course that we're going to offer. Well, and to the database I have to put,
how long is each one of those? I have no idea how to get the duration out of an MP4 or MOV file.
You know what? There's a really cool command line program I can run. It'll tell me. So I just use
sub process and call that. And then I can script out the rest in Python and it's, you know,
sub process is not to be underestimated. I think. Yeah. Yeah, exactly. No, it makes a lot of sense. I mean,
at a certain point, shell scripts can get a little bit too hairy to work with being able to automate
your things and use Python as your super glue, right? So a little bit stronger than duct tape,
I think makes a lot of sense. Yeah. We talked to the beginning about how
you're in this exploration stage and you just want to just run a bunch of stuff on the command line and
figure it out. But when you go to production and you said, whatever that means, like this could be one
of one thing that it means we're going to write formal Python code and then use sub process to
kind of bring in some of this functionality potentially. Yeah, exactly. I mean, the command
line is by definition, very ad hoc in nature. Still, if you're doing things in production, meaning you're
interacting with, with other environments, with, with the servers, or you have some kind of continuous
integration going on, there are these places, these are places where the command line keeps popping up.
Yeah. Right. So even there, so it is useful to at least be comfortable with this stark and unforgiving
environment. I think it's really excellent. I think there's a lot of cool stuff that we talked about. I
think there's a lot of, a lot of value for people to learn this. I guess, you know, maybe we close this
out with just one comment that I remember from your Strata presentation. You said the command line is
like wine. Maybe it takes a while to appreciate, but it gets better with age. I certainly, my first
experience was like, okay, I'm going to go from windows and Mac dev development over to setting up and
running servers over SSH. It was like, I am beyond lost. I have no idea even like just how to get started,
right? Many years ago. And, and now it's like, well, of course that's a beautiful way. And it just,
just you slowly build up these, these skills and it's, it's really lovely.
Yeah, it is. No, it took me a long time to get comfortable with the command line actually,
or Linux in a more generic sense. I, for the longest time I was running windows and Linux in a,
in a dual boot machine. And so I just couldn't make the jump. And this was, over 10 years ago,
but, no, it didn't definitely didn't come overnight and I wasn't born with it. So I also believe that
everybody is able to, to embrace the command line, if you will, but you just gotta, you know,
make yourself a little bit comfortable there as well. We talked about that in the beginning,
right? The right terminal, the right aliases can get you a long way.
They get you so far and tools like Oh, my Z shell and some of these others, the fast that will help
you remember the thing you needed to type or, or like you said, aliases and kind of bring it all
together. And like, ah, I know I did that thing. Let me just do a quick search for, yeah, there it is.
Five, you know, five weeks ago I ran this and that's how I, this is how I restart the web server. Oh yeah.
Now I remember. Yeah. Yeah. Yeah. I can talk about this all night. I think we're probably out of time
though. Let me ask you the final two questions before you get out of here. You're going to do some
editing or write some code. What editor do you choose these days? I am torn between Visual Studio Code,
doom Emacs and Neo Vim. And I, but wherever I am in these editors, I always have my Vim key bindings
set up. So it kind of depends on the project, but yeah, as long as I have my Vim key bindings, I'm happy.
Yeah. Yeah, absolutely. And then notable, normally I ask notable PI project or library, but maybe broaden
a little bit. Like if you could recommend one tool, one library people could install for the command prompt
or the shell, what would you say? One tool or one command line tool for that they could install on the shell.
Just something, it doesn't have to be the most popular. Something like people, if I ran across this,
it was delightful. People should know about X. Yeah. Can do parallel. Let's do it. Let's say,
yeah, we talked about it. So it doesn't require any further explanation. It's the tool that, that,
you know, makes every other tool way cooler. Yeah. So yeah, if you have that one in your arsenal,
you can become very powerful. That's a good recommendation. All right. Well, final call to action.
People are excited about this. They want to learn more about it. What do you tell them?
Yeah. A couple of things they can do. So my book, Data Science at the Command Line is freely available.
Yeah. So the second edition came out a year ago. You can read it for free on data science at command
line.com. I also offer a cohort based course that I do twice a year. The next cohort is coming up in April.
And this is, yeah, there, we, we have six live sessions and then I will, I help, you know,
a group of researchers and developers, you know, embracing the command line. It's a, a, a, a very
different experience than reading a book. If you want to know more about that, then also data science at
the command line.com has a link to that. Apart from that. Yeah. I mean, if you just follow
hacker news, you'll come across now that you're aware of the, of all these tools, you'll come across
quite a lot of tools every now and then there's not a week in which there's not a tool being mentioned.
There are tools being developed every day, even though it's a, you know, the technology is over
50 years old. It's impossible to keep up. It's only getting cooler. It is only getting cooler.
Yeah, definitely. So yeah, that's my, my recommendation there. All right. Fantastic. Well,
thanks for being here. It's been great. Congrats on the book and putting all this together. Yeah.
Thank you very much for having me. Yeah. You bet. Bye. Bye.
Bye.
This has been another episode of Talk Python to me. Thank you to our sponsors. Be sure to check
out what they're offering. It really helps support the show. Take some stress out of your life. Get
notified immediately about errors and performance issues in your web or mobile applications with
Sentry. Just visit talkpython.fm/sentry and get started for free. And be sure to use the promo code
talkpython, all one word. Starting a business is hard. Microsoft for startups, founders,
hub provides all founders at any stage with free resources and connections to solve startup
challenges. Apply for free today at talkpython.fm/foundershub. Want to level up your Python? We have one of the
largest catalogs of Python video courses over at Talk Python. Our content ranges from true beginners
to deeply advanced topics like memory and async. And best of all, there's not a subscription in sight.
Check it out for yourself at training.talkpython.fm. Be sure to subscribe to the
show. Open your favorite podcast app and search for Python. We should be right at the top. You can
also find the iTunes feed at /itunes, the Google Play feed at /play, and the direct RSS feed at /rss on
talkpython.fm. We're live streaming most of our recordings these days. If you want to be part of
the show and have your comments featured on the air, be sure to subscribe to our YouTube channel at
talkpython.fm/youtube. This is your host, Michael Kennedy. Thanks so much for listening. I really
appreciate it. Now get out there and write some Python code.
I'm out.