Refactoring your code is a fundamental step on the path to professional and maintainable software.
We rarely have the perfect picture of what we need to build when we start writing code,
and attempts to overplan and overdesign software more often lead to analysis paralysis rather than ideal outcomes.
Join me as I discuss refactoring with Brendan McGinnis and Nick Thoppen,
as well as their tool, Sorcery, which adds automatic refactoring in the popular Python editors.
This is Talk Python to Me, episode 266, recorded May 21st, 2020.
Welcome to Talk Python to Me, a weekly podcast on Python, the language, the libraries, the ecosystem, and the personalities.
This is your host, Michael Kennedy. Follow me on Twitter where I'm @mkennedy.
Keep up with the show and listen to past episodes at talkpython.fm, and follow the show on Twitter via at Talk Python.
This episode is sponsored by Datadog and Linode.
Please check out what they're offering during their segments. It really helps support the show.
Brendan, Nick, welcome to Talk Python to Me.
Thank you very much.
Thank you for having us.
Yeah, it's great to have you here.
I'm such a huge fan of refactoring and code quality and all these ways of taking living software and making it evolve, right?
I think long gone are the days of we have to plan this perfectly, and then we're going to build the perfect thing that we've thought up, right?
And so having this idea of continuously evolving and improving code, it just frees you from worrying about trying to get it all right and you can just get started.
And so I'm really excited to talk about sorcery and refactoring with you guys.
Awesome. Yeah, we think iteration is super important as well.
Sort of trying to get a skeleton of the thing up and running and then sort of tidying up later.
Yeah, absolutely.
That's kind of how we like to work.
No one really understands the domain when they first write the code anyway.
You have to write the code, find out all the mistakes that you've made, and then tidy it up, clean it up.
Over time, like you say, it evolves into what looks to be a nice quality code base and solves the needs of the users.
Yeah, I think that's a super good point because understanding, you don't fully understand it until you've gotten mostly into it.
But if you tried to understand it all, it's just so much work.
Even if you get it right, it's so much work to get to that point that you might as well have just written it three times.
So that's awesome.
But before we jump into more of that, let's just get your stories.
And Nick, I guess we'll start with you first.
How do you all get into programming into Python?
So I guess the first program I did was back in school on these graphics calculators we had back in the 90s.
And I remember reading a book about complex numbers and being super proud of getting my calculator to draw a fractal.
I think it took about 12 hours to draw it.
But I end up with a Mandelbrot set on my little calculator.
Like the Mandelbrot set or something cool like that.
Could you zoom in?
I think you could zoom in, but it would take another 12 hours to show you.
You could only do it twice because then the battery would run out.
Pretty much.
I did a little bit of basics and stuff, but I never did programming in a serious way until joining a software company after university.
What are you studying at university?
So it's maths and philosophy.
Okay.
That's a cool combination, actually.
Yeah.
And there's like a lot of logic, which I guess is quite key to programming.
And then the first languages I learned in the software company were this mainframe language called IBM RPG.
I don't know if you've come across that one.
I could not look at RPG source code and tell you what it is.
Like I couldn't identify it from code.
Yeah, I've heard of it.
That's incredible.
It's sort of, it's like punch cards.
Everything has to be in the same, in the right column kind of thing.
Okay.
Because it was this finance company.
Brendan went there as well.
It sounds as useful, but I think I'll stick with Python.
Yeah.
I had this code base from the eighties.
Okay, cool.
And you had to program it in a green screen terminal.
So that didn't put me off.
Yeah.
If you go through that, you're definitely good for this industry.
If you can make it through that, Steve, you'll be good.
And then it was Delphi, which I also look back on as not being super amazing.
And then Java.
And then it was only when I joined Imperial and started doing sort of machine learning that I got into Python, I guess, mid 2010s.
Cool.
You're like, why does this language have semicolons?
What happened to it?
Yeah.
I mean, after Java, it just was like a breath of fresh air, really.
And when I had to go back to doing Java, it seemed so verbose.
Yeah.
You're like, why?
I can't read this.
There's symbols all over what it's trying to tell me.
Yeah.
Curly braces and parentheses and semicolons.
Yeah.
And the way they do the libraries and things, actually, it seems to be all done in a very verbose way.
Yeah.
Indeed.
Lots of boilerplate.
Nice.
Cool.
Well, Brendan, how about you?
The first programming I did was in my final year math degree.
So we had a project to do, which was to simulate various mathematical equations.
So the ones that I chose were the heat equation.
And it would simulate one wall being hot and another wall being cold and various obstacles between the two walls.
And then after it ran, it would show this chart of the temperature at different places in the room.
And the other thing I simulated was the Schrodinger equation, which quantum mechanics.
Yeah.
Quantum mechanics is so interesting.
And it's just like, it seems like such a weird Twilight Zone alternative reality.
And yet it seems to be applying to real reality.
It's such a weird world.
I love quantum mechanics.
Yeah.
It's amazing how accurate the predictions are.
And yet when you try to understand it, it just has no similarity to reality.
Just making that up, aren't you?
There's no way this is real.
When you get so small, it doesn't match up with what you understand.
Yeah, for sure.
How can this object be in two places at the same time?
What does that even mean?
Exactly.
Yeah.
Well, the equation, I was just simulating a single object in the waveform of it.
So it was quite simple.
But the way they did it was, here's min GW, go and program it in C, and then just send us off to go and do it.
And we knew nothing about how to program.
So my programs were just hundreds of lines of statements, absolutely no functions, nothing, no structure.
Yeah.
I think we probably all got-
Probably some memory leaks in there as well if it was written in C.
Oh, yeah.
The whole thing, yeah.
The whole thing was pretty terrible.
But you get to the end of the project and you simulate it correctly and you think, I can program now.
I know how to program.
Even if that's not entirely true, I think you do come away with this feeling of like, oh my gosh, look what I built.
Like, this is so awesome, even if it's not really.
It's just like that feeling of creating that thing is so cool in those days when you're getting started.
Absolutely.
I think that, yeah, the first time that I managed to output the heat equation onto the screen where you see the different temperature across the room,
they provided us with a charting library to use.
That was amazing.
And I was like, wow, I can program.
I can do anything.
I know computers now.
Yeah.
Yeah, yeah.
Super cool.
So, yeah.
After I finished university, I joined the same company as Nick.
I learned RPG and I learned Delphi.
I learned Java.
And then I achieved quite a senior role in the company.
I was part of the architecture team.
And I managed to introduce Scala to the company, which, yeah, we did a language comparison and it was between JVM languages.
So, one of the languages was actually J Python or Jiton.
We didn't go for that or JRuby because it wasn't first class language on the platform.
So, really, the main contenders were Java, Scala and Clojure.
And we had someone come in and talk to us about Scala, who was really, really amazing.
And he convinced us to use it.
So, yeah, I ended up leading the project to bring Scala into the company and re-architect a lot of the systems in that.
When I left that company, I was very much down the functional programming route.
So, I did some more Clojure, which is very functional.
I did some Haskell.
But then I joined Nick, Imperial College London.
And that's where he said he got into the Python.
That's where I got into it as well.
So, I joined a reading group that was all about deep learning.
And I went to the reading group and we read a paper, which was called Differential Neural Computer.
And I had no understanding what it meant.
And I was like, OK, I'm going to go home and I'm going to implement it.
So, I went home and I cracked out some Python, cracked out TensorFlow.
And it took me weeks and weeks to implement it because I had no idea what was going on.
That was the start.
So, I just started implementing more and more of these papers as I went to more and more reading groups.
And it tended to be all of the Atari playing reinforcement learning algorithms, which is really fascinating to me.
OK, cool.
So, I learned how to play Breakout and learned how to play Pong.
Right.
Maybe some Pitfall in there.
I never tried it on Pitfall, actually.
That was a good one.
Nice.
I'm really fascinated with these ways to train AIs around video games.
I mean, I'm blanking on one of the options.
But there's a handful of libraries that you can kind of plug your AI into the virtual world.
So, it has somewhere to interact with and things to interact with.
And yeah, that's fascinating.
Yeah.
I really did find it fascinating.
That's why I was doing it every week.
And then just around that time, AlphaGo came out as well.
And it was like a shock to the world.
Suddenly, the hardest game that humanity can play is beaten by a computer.
Yeah.
And it's one of the first, it might have been the first real AI opponent that used like
strategizing rather than just deep exploration of the paths.
Right.
Like the chess one is like, well, I can hold 12 steps ahead in every direction in my mind.
That's more than the chess master.
So, right.
Like we'll just play them all out, all the possible futures out and then go down the best
one.
Right.
But that's not how AlphaGo worked, which is, I think, part of the magic.
Yeah, it's got intuition in a sense.
That's the interesting bit.
Yeah.
It was just fascinating the way they did that.
I ended up writing a version of AlphaGo to play Kinect 4.
Okay.
Which is actually pretty strong.
That was a really fun project as well.
I mean, it can beat me.
You're a very strong player.
That's awesome.
Yeah.
I'm an average player out there.
Yeah.
How cool.
All right.
So what are you both working on the same project yet again?
What are you doing day to day?
What are you doing now?
We're both working on Sorcery full time.
We kind of started working on it back at the end of 2018 from Brendan's flat.
So I'd turn up and he'd be still in his pajamas eating cereal.
Sit down and code away.
Yeah.
We're kind of totally focused on making Sorcery as good a refactoring tool as can be really.
Yeah.
Cool.
We'll get into what Sorcery is more later, but what's the quick elevator pitch before
we dive into just more general software stuff?
I guess we've been pitching it as kind of Grammarly for code.
And if you don't know what Grammarly is, it's like it improves the style of the code without
changing the sort of meaning or the content.
I guess that's what refactoring is.
You improve this quality in the structure without changing the functionality.
And that's what we aim to do.
So as you're writing it, we analyze it and we suggest refactoring improvements sort of
as you go.
So maybe I got a loop and I'm doing some kind of accumulation into a list and it could say,
you know what, that could just be a list comprehension.
Yeah, exactly.
Yeah.
Okay.
That sounds awesome.
It's easy to just get focused on writing code and then not really worrying about the quality.
It's also equally easy to get super obsessed with the quality and not just get the thing
done.
Right.
So I kind of see like these two bimodal distributions.
One's like, I don't give a crap.
I'm not going wrong as it works.
I'm just going to write, write, write.
Yeah.
And, you know, that's one group of people's philosophy.
The other is they're like really slow and super meticulous to get it just right because
they want to write it the best way.
And it sounds to me like the tool would let people kind of find a middle ground, right?
Like write it a little more loose and free.
But then it says, oh, by the way, that thing you just wrote, actually, we could make that
way better if you just let us.
Yeah, that's kind of the idea.
And that's some of the feedback we've been getting.
That's people write codes a bit more quickly without worrying so much about the quality and
it'll kind of tidy up for them.
Okay, cool.
And the other aspect of it is some people don't actually know what good quality code
is.
So if you're starting out in Python, you may be able to write the solution, but you won't
necessarily know how to write it well.
So you may not even know about this comprehensions yet.
Right, right, right.
And the benefit of using Sorcery in that case is it can teach you the Pythonic way of writing
code.
This portion of Talk Python to Me is brought to you by Datadog.
Are you having trouble visualizing bottlenecks and latency in your apps and you're not sure
where the issue is coming from or how to solve it?
With Datadog's end-to-end monitoring platform, you can use their customizable built-in dashboard
to collect metrics and visualize app performance in real time.
Datadog automatically correlates logs and traces at the level of individual requests, allowing
you to quickly troubleshoot your Python application.
Plus, their service map automatically plots the flow of requests across your application
architecture so you understand dependencies and can proactively monitor the performance
of your apps.
Be the hero that got your app back on track at your company.
Get started today with a free trial at talkpython.fm/datadog.
So maybe you don't necessarily know the idioms.
I think one of the challenges of Python in particular, I mean, all languages have this problem, but
Python suffers more than others from it.
One is that it's so easy to learn that people feel like they can learn it in a weekend and
then they just go write code in it, right?
Because like, look, it's a simple little language.
There's not a whole lot to it.
Which I think is actually not true, right?
I still feel like I'm learning Python every day.
I'm like, I didn't know, or I should have done this, right?
Like, there's all these things.
There's just, there's so much nuanced detail to it.
But it's easy for people to come from a language like C or Java or something else and just do
the Java style programming there or the C style programming there.
And not, like you said, not even know that there's this other component, right?
They say, oh, Python's crappy.
It doesn't have, you know, a four, a numerical for loop.
This is crummy, right?
When really a better way to do it would be to use enumerate collection where you get the
index and the item, right?
You don't have to go back.
If you didn't know, like that's multiple layers.
One, you have to know there's a four in loop.
And then two, you have to know that enumerate is a thing.
And then you've got to know about tuple and back.
And like, that's a pretty complex set of topics.
If you've, well, I spent the weekend now, I kind of know, let's, let's finish this project.
Yeah.
If you took that philosophy.
Absolutely.
I mean, like you said, the learning, the Python language, and it's so beautifully simple that
most people can pick it up in a week to a month's time.
But then you've got all of these items and they're not just the different bits of syntax
or the enumerate.
There's multiple libraries all over the place.
And they're the libraries that are built into the Python main library.
But there are also all of the libraries out there that you need to accomplish more complicated
tasks.
Yeah.
Just finding out what the best library is for the job is a difficult piece of research often.
It is.
Sometimes it's something built in, like inner tools or something like that.
Or other times it's something you've never heard of because there's 200,000 options and
they all have their, their behaviors and whatnot.
But if you would grab it like that would take 20 lines down to one, probably be faster at the
same time.
It's like, it's incredible.
Right.
That's kind of what I was thinking.
I'm like, I've never done learning Python because like, oh, there's this other standard
library module I discovered where I wasn't using counter in this way or, you know, like
whatever it is, right?
There's just all these options.
Yeah.
We've definitely come across that.
Sorry.
Come on.
Yeah.
I can imagine.
And coming from an academic space, I'm sure you see that a lot there as well.
Cause there's probably a lot of people who don't see themselves as a developer, but they're
still touching Python, writing code.
It's full of code you kind of write once and then don't, don't look at again.
So it's kind of, I guess done with a different aim in mind.
You're not, they're not so worried about reuse by other people often.
Right.
It's actually been quite a challenge for us because we learn Python together effectively
at the same time.
And we've never written Python in a large code base apart from our own.
So we've had to learn all of these things ourselves.
So over time, we've rewritten bits of the system where we found out, oh, we can use this,
this feature of Python that makes it so much better.
And, at one point we integrated my pie into our code base and that was a big improvement.
And you say, it's slowly learn about these options.
And I think a lot of people out there who are learning Python probably fall into that bracket
as well.
Then they're learning it as their first language.
They're not learning it with people who can guide them how best to use it.
And it's really hard on your own.
I mean, we're experienced developers and I think we're finding it really hard ourselves.
Yeah.
To do it right and to take full advantage of it, it is.
So it's cool to have like extensions for the IDE that will sort of not quite be a paired
program partner, but someone to sort of sit there.
But you know what, that actually is not the right way to do it, but it's super easy to fix
and I can take care of that for you, right?
Exactly.
Yeah.
That's exactly what we...
That's awesome.
So we talked about code quality and, you know, that's a little bit in the eye of the
beholder.
It's also a little bit in the trade-off.
Like Nick, you talked about this concept of I'm going to write a script and get an answer
and never run it again, right?
That has a different threshold for code quality than, you know, the core trading engine at a bank,
right?
Like it would probably be improper to put that much energy into that script that's going
to be run once, right?
You should just write the thing and get it to work and not worry too much about it.
But at the same time, if you're building something to be reused and is important, it's going
to be run by lots of people.
You really want to get it right.
And so I think there's this spectrum and people got to like figure out where they live on it.
But no matter where you live, you would like to have better code quality rather than less
good code quality by like whatever applies to your situation.
Right.
Oh, definitely.
And it is kind of hard to quantify what code quality is.
Sometimes, you know, when you see it, I guess it was reading the book Clean Codes.
I think it's by Bob Martin.
Robert C. Martin.
Yeah.
I really crystallized it for me.
And I was always trying to get the graduates at the old company I worked at to read it.
And eventually one of them stole it.
So I think, I guess that means they liked it.
You won.
Yeah.
It worked.
You converted and they stole your book.
I guess the real core of it is just high quality code is easy to read and understand.
It just reads like a sort of story.
Does this and this and this.
Yeah.
I really love this idea of clean code and the stuff that Bob Martin talks about.
He's got some really good ideas.
I don't totally agree with everything he says, but I think there's a lot of good lessons
to take from what he's doing.
Yeah.
Sure.
Yeah.
One of the really interesting ideas that comes from one of his contemporaries, Martin Fowler,
way back at the origins of refactoring.
I remember reading the book called Refactoring in 1999 or something like that, just going,
my mind is blown, right?
I've had this problem of bad code quality and I've had this problem of trying to write it
well or to fix it.
And then I realized, you know, reading what he was talking about, like, oh, there's this
way to take the bad stuff you've already put down as like sediment in the software.
It's crystallized and like turn that into something that can be improved and grown over
time.
And I just, I remember it really changed my way of thinking about programming, like digging
into refactoring.
So I'm just such a huge fan.
How do you guys come across it?
It's an interesting one because the code base we worked on our old company was so huge and
difficult to change.
Often we didn't even try refactoring it.
It's just kind of, you did a surgical approach.
You went in and tried to understand it and made the smaller changes you could.
Please don't break.
Just take the new feature without breaking.
Or I guess, Brendan, you actually sort of took the other approach and said like, okay, I'll
just rewrite this whole bit.
Yeah.
Well, I mean, it was a risky approach given that the system was not under test at all.
So I would only do that on front end components.
So the UI, but yeah, sometimes it was, it was just the case of, I don't understand what's
going on here.
I can see what the functionality currently is.
So I'm just going to re-implement it from scratch.
You know, if people are in that space, like the whole area of what Michael Feathers talked
about with legacy code and like how to take these, these huge systems that are hard to
change that you don't necessarily know.
They don't have tests and how to like break off little bits that are maintainable.
That's such a cool book working effectively with legacy code.
I really do enjoy sort of tinkering with code, tracking down bugs, sort of improving it, making
it a little better.
I think when I see a blank sheet of paper or blank screen, I kind of sometimes find it difficult
to start.
I think Brendan's a bit better at that.
So I, yeah, I super enjoy refactoring.
And I guess it's one of the things we're trying to kind of achieve with Sorcery is sort of if
there's a machine that can do the refactoring for you, you can be less worried about it
being under test because, you know, it's done proper analysis.
Because whenever I do refactoring, I break something.
And I, I would lean heavily on the tests.
It's almost never seems like a good idea to me to do a refactoring manually.
If there's some sort of like tool-based way in which it will happen, right?
It's just, you never know what little thing you're going to, you're like, oh, there's that
one, that cron job thing that we had that was also using that.
Now, apparently it's not going to take it anymore.
And with, with Python, you don't have compiling, right?
So you're not going to catch the obvious stuff.
Like I move this function over here.
It's just like, well, don't run that part.
It's going to crash.
That is one of the challenges with Python.
And I mean, the way we've approached it with our Sorcery code base is testing and
mypy type annotations.
Yeah.
I think they give enormous confidence when you're refactoring the code.
So nowadays I don't do crazy refactors, crazy rewrites like that.
I incrementally improve through small changes.
But yeah, I've experimented with that once and realized it's not the way to go.
But yeah, the real key is having those tests and the type annotations.
You can move something anywhere in the system and you'll get told about all of the errors
that you now have.
And then you can go and fix all of those.
And then you can do the next refactoring and build it up through there.
Most of our bugs are in the bits we hadn't added mypy to.
So I totally expected to hear automation from you guys and applying Sorcery back unto itself
and things like this.
And we'll dig into the features in a second.
But I didn't expect to hear typing in mypy.
I'm personally a huge fan of the type annotations in Python.
I think they make working with Python code so much easier.
You don't annotate everything, but certain places like this function returns one of these,
just knowing like actually it expects to return one of these.
It's super helpful and it'll light up the editors as well, right?
They can all of a sudden give you autocomplete for what they weren't sure before.
But now they know, oh, here's the five things you can do with what you got back.
Perfect.
How does mypy fit into your world?
I think that's pretty interesting.
What are you doing with that?
We really only use it internally because most code out there doesn't actually have mypy
or type annotations.
Right.
Even if it has some type annotations, there's like sort of a chain of annotations that have
to be consistent.
Like mypy is a stronger level than just saying, oh, this function happens to return a list.
Yeah.
I mean, one of the great things about type annotations is you can just look at a function
at the definition of it and understand the interface.
You don't need to read the code.
Without those type annotations, you have to actually read the code and say, oh, actually
this is a string and this is an integer and it returns a list of integers or something
like that.
Yeah.
With the type annotations.
Yeah.
That's a really good point.
I like that.
So I think type annotations are a form of documentation.
They're really, really powerful just from a readability point of view.
But then you get all of the security as well of knowing when you've broken the code or when
you've not called something correctly.
One of the things that is really important to us with Sorcery is that we never break other
people's code.
So we have to have extremely strong gar...
Yeah.
So let's take a step back and why don't you tell people about what Sorcery is?
How do people use it?
So you mentioned that it's a plugin for IDEs, but give us a little bit more detail and then
we can talk about how you keep from breaking people's code, which is probably...
People probably appreciate that.
If you...
It acts as a plugin to your IDE.
So we've got plugins for VS Code and PyCharm.
And as you're coding away, it sits there reading your code and analyzing it.
And if it identifies a change to a function that you're working on that will improve the
code quality, it'll suggest it to you.
And you can review that suggestion.
And if you like it, you can accept it.
And it'll apply that change in line.
And you carry on coding it.
It works almost seamlessly in your workflow.
Okay.
That sounds awesome.
Does it change the way the editors work in other ways?
Like, for example, does it change the autocomplete or things like that?
Or is it really more like the code intentions, like the little light bulb in PyCharm?
Yeah, it's kind of exactly like a code intention in PyCharm.
That's kind of the thing we've gone with.
Yeah.
So yeah, it does a little underline.
Cool.
So as you're going along, you're watching...
Oh, there's a little pop-up.
I should go see what this is about.
Yeah, yeah, exactly.
And it runs locally on your machine.
I guess that was quite a sort of concern for people.
They didn't want their code being sent to the cloud.
I can't imagine why.
So when we first started it, we were going to do it kind of as a service in the cloud.
And we kind of had to do a pivot and get it running locally on the machine.
Okay.
So somehow behind this, how does it make decisions about refactorings?
Is it like an AI-based thing?
Is it pattern matching?
Like, what is it doing inside?
So at the moment, there's no machine learning or AI in it.
The way it works is it is essentially pattern matching.
So it's looking for...
Well, it works at the level of the abstract syntax tree.
So it takes the code and it parses it into a data structure.
And that data structure will have, for instance, an if node.
And within that if node, it will have a function call.
And the if node will also have a list of statements.
It looks at those nodes and it looks for the patterns, like you say.
So, for instance, it might look for a for loop that has an if statement within it that appends to a list.
And then it says, okay.
Like, that's a list comprehension waiting to be made right there.
Yeah.
Exactly.
And I guess the clever bit is it kind of has these little, lots of little tiny little patterns of improvements it can do.
But it can compose those together into like a bigger refactoring.
And it's guided by a load of code metrics we've kind of incorporated into it.
So we can get into those later.
Like cyclomatic complexity and some of those types of things?
Yeah.
So we don't use cyclomatic.
We do use cognitive complexity, which I think is like a trademark of SonarCube or something.
But it's a different metric.
And we use a few we've written ourselves.
And so it can kind of chain together little refactorings to do something bigger.
So, for example, on the, I don't know if you've seen the Gilded Rose refactoring CARTA.
No, tell people about it.
So it's kind of this big fantasy.
Wait, let's take a step back.
What's a CARTA?
So a coding CARTA is a coding exercise to improve your programming.
Right, right.
And there are various ones of these floating around the internet.
And Gilded Rose is like this big, complicated set of nested ifs, basically.
And it's sort of about this fantasy in, I think.
And it takes maybe an hour to kind of manually sort the code out and refactor it.
And it's sort of an exercise people do.
And our aim when we started Sorcery was to, this was like our initial target problem.
So it can kind of do all that work at once by chaining together lots of little refactorings.
So it can take the sort of complex mess of spaghetti code and then turn it into something understandable.
Right.
Instead of having to say, okay, here's a little if statement that could be improved and then apply it again and say, well, now that we have this code, there's another thing we can improve than apply it again.
It'll like chain those all together and go, actually, we could roll this all up.
Exactly.
Yeah.
Because when you're doing manual refactoring, that's kind of what often happens.
You sort of do a little thing and then you realize, oh, now I can do this.
And you might have an aim in mind or you might not.
And then you start chaining these things together.
And in the end, you're like, oh, now it's an understandable code base.
Yeah.
Very cool.
Well, that sounds super, super useful.
I know that some refactorings build into certain tools like PyCharm has certain refactorings, but they don't seem to take this more holistic approach.
Right.
They're like, oh, this list, this list comprehension could be expanded to a for loop if you need it or something like that.
But that's kind of as far as it goes.
Yeah.
And they're kind of very developer driven.
You have to know you want to do them.
Yeah.
And you have to know where you can do them and then you have to do them.
So they're very useful if you know you want to do something because it'll do it for you.
And like, it won't make mistakes, but they don't sort of, they don't tell you if it's a good idea or not.
So our idea is we're kind of suggesting things that we think are good ideas to actually change.
This portion of Talk Python to Me is brought to you by Linode.
Whether you're working on a personal project or managing your enterprise's infrastructure, Linode has the pricing, support, and scale that you need to take your project to the next level.
With 11 data centers worldwide, including their newest data center in Sydney, Australia, enterprise grade hardware, S3 compatible storage, and the next generation network.
Linode delivers the performance that you expect at a price that you don't.
Get started on Linode today with a $20 credit and you get access to native SSD storage, a 40 gigabit network, industry-leading processors, their revamped cloud manager at cloud.linode.com, root access to your server, along with their newest API and a Python CLI.
Just visit talkpython.fm/Linode when creating a new Linode account and you'll automatically get $20 credit for your next project.
Oh, and one last thing.
They're hiring.
Go to linode.com slash careers to find out more.
Let them know that we sent you.
Let me share one of my favorite concepts from refactoring and then ask you about some of your favorite refactorings.
So there's all these different refactorings, even in the early days that Mark Fowler talked about, like, okay, there's a God object and here's how you break it down.
Or there's a function that's too long.
Here's what you do and so on.
But those are all kind of interesting.
Like, the most interesting concept around all that stuff to me was the concept of a code smell.
Right?
Just like, there's something wrong with this.
Like, it works, but your nose kind of turns up when you look at it.
You're like, there's something wrong with this part of my code or code probably I inherited from somebody else.
Right?
And the other thing was he would talk about comments and say, often comments have value, but a lot of times they're really just deodorant for these code smells.
Like, this is really hard to understand because it's written badly.
So let me write a comment that tells people what it really means.
Yeah.
And then just leave the bad stuff there.
Right?
It kind of deodorizes the code smell a little bit.
And that's this idea of, like, if you have those comments, it's like this underlying thing of, like, you should start thinking of applying these different refactorings.
So my question to you with, like, sort of putting that out there is, what are some of the favorite refactorings that you guys are seeing possible with, like, this deeper integration, right?
Like, obviously, for loop to Liz comprehension, Liz comprehension to for loop.
Like, those are pretty straightforward, but it sounds like there might be either things you just really love or there might be, like, some more interesting, larger refactorings.
The main code smell that I think Sorcery does really well with is eliminating duplicate code within a function.
And in particular, within different branches of a complicated set of if expressions.
So you may have the same body of code in two different places.
And Sorcery can restructure the code until there's just a single condition that applies for that block of code.
That's cool.
Yeah, I think that lets me delete lines of code is always very pleasing.
So, yeah.
Yeah, yeah.
And delete conditionals as well, right, if possible, or simplify them.
You know, I feel like the more the word legacy gets applied to a code base, the less you want to do those kinds of things.
You're like, I'm pretty sure these three things are the same, but I don't want to be responsible for what happens if I misunderstand that these are actually slightly different.
And try to refactor it to a cleaner version.
So as much as you can get software, does you go, actually, no, this is totally safe.
We got you.
It's actually quite interesting because one of the things that has happened is we've suggested refactorings to people and they've gone, this is incorrect.
And there's been these examples of eliminating duplicate code and simplifying if expression.
It's kind of once you work through the logic.
Yeah.
We've discovered actually sorcery is correct here.
And it turns out your code was either very confusing or possibly had a bug in it, which you have now identified.
Right, right.
You thought these two things were doing different stuff, but in fact, it has no...
The effect of this is not what you had in your mind its actual effect being, right?
Yeah.
You misunderstood what was actually happening and your mental model didn't match the refactoring result, but that's because it wasn't actually doing that.
Exactly.
And so actually, once it's done that refactoring, you can say, oh, actually, there is a bug in my code.
I can fix it.
But you have to have the trust in sorcery to know that it's correct before you're willing to take that step.
So it takes a little bit of usage to build up that trust.
So how do you guys check that your refactorings are valid?
So like we said, we have a library of smaller refactorings, and then we have a search engine that composes those together.
So the important thing is making sure each of those individual refactorings in the library is correct.
Right, right.
Because composing a bunch of things that are correct is not going to break anything.
Exactly.
Yeah.
So the challenge is to try and make sure that those individual ones are correct.
So we have lots and lots of tests, and those tests are of the form.
Here is a piece of source code, and here is the expected refactored source code.
And for each refactoring, there's a multitude of those.
But there's also a multitude of tests of the form.
Here's a piece of source code that looks similar to these other bits that you have refactored,
but you shouldn't refactor it because if you do refactor it, or if you do make the change, you'll break the code.
There's got some gotcha in there, yeah.
It's not a true refactoring.
And it will tend to be things like you're calling a function, and so you can't swap these statements
because one of them is actually a global variable.
So we have an awful lot of analysis, which determines what statements in a function depend on the other statements.
Yeah, it sounds interesting.
It turns out to be the hardest problem that we've tried to solve.
I can imagine.
Have you guys looked at using things like hypothesis or other property-based testing,
where it's like, here's a block of code, apply refactoring to it,
feed it a bunch of inputs to both versions, and see if as long as you get the same outputs,
or things like that?
That's a future plan that we have.
We do have a second form of testing that we do at the moment,
which is, as part of our build process,
we run Sorcery over a whole bunch of popular open source libraries and refactor them.
And then once they've been refactored, we run their tests over themselves
to check that we haven't broken any of their code.
Because people have already written a ton of tests for SQLAlchemy, for requests or whatever, right?
Exactly.
So, I mean, that has identified, I mean, when we first introduced it, identified lots of issues.
And since then, it's stopped us releasing any new bugs as far as we're aware.
That sounds like a pretty good way to just hit it with enough information that it's going to get caught in the issues.
Have you found errors in other libraries because of this and gotten back to them?
Like, you know what?
This is actually, we thought our stuff was broken, but actually your stuff is broken.
I mean, that'd be cool.
I mean, sometimes the tests are failing in master.
So after a while, we just decided to pick a tag and stick with it and be done with it.
But actually, we found that there's just a lot of backcode to review.
So we tend not to review it at the moment.
Yeah, it's not your job to check all open source libraries for correctness, right?
Exactly.
And some of the libraries like SQLAlchemy that we do run it on are absolutely enormous.
There's hundreds of files and multiple drivers for different database backends.
So it takes a long time to run those tests.
Yeah.
We just rely on the test to tell us whether we're good to release or not.
Yeah, that's cool.
But yeah, the hypothesis-based testing is a very interesting idea that we have talked about.
So the way we were considering doing it was exactly how you talked about.
You write a piece of code and then you put some inputs and you check what the output is.
And then you run Sorcery over it.
And the way we were going to do it is actually also write a generator for source code that takes
maybe an initial piece of code and does random mutations to it to start off with.
So here's a piece of code that Sorcery should refactor.
Let's apply a bunch of random mutations to it and then run Sorcery over it.
Check the inputs and outputs are the same again.
So yeah, the generator for that would have been quite interesting to write.
And it is something that we're considering in the future.
Okay.
Yeah, it sounds cool.
It sounds like it would take forever to run, but it sounds like a cool project.
Maybe don't run it on every save.
All right.
Well, let's see.
There's a bunch of things I want to ask you about, but I don't want to go over too much
on time.
So I guess one area that looks interesting to me is we've talked about this being a plugin
for an editor that's interactive.
You also talk about just applying it to open source libraries.
And on the Sorcery homepage, I see that there's a get instant quality of your Python code base,
like just point it at your repo and it'll give you some answers.
So is there more stuff that it does than just be a plugin?
Yeah.
Or is there like a CLI way to use it?
So it is also available as a GitHub bot.
So you install the Sorcery bot into your GitHub repo.
And every time you do a pull request, Sorcery will review that pull request.
And if it finds any improvements to any of the files that have been touched by the pull request,
then it will create a pull request on top of that saying, here's the changes that you can make.
An improvement to it.
Yeah.
Then you can just merge that pull request in straight away.
When you first install it, also it can refactor the whole library kind of all at once.
Very cool.
All right.
So let's talk about pricing.
So this is something that is free for some people, but it's not free for some other people.
What's the story with the whole business model, open source side of things?
Like what is, what are you guys offering here?
Because it sounds really useful to a lot of people, but at the same time, you are charging some folks for it.
So that might, you know, that might influence people's opinion on how they feel about it.
Yeah.
So the plugins are free at the moment.
We think in the future, we'll probably introduce a premium version and still have a free version.
I see.
So if I'm sitting here and I want to write code on my MacBook on PyCharm, I can just go get it for free.
I don't have to pay anything.
Yeah.
You can just go get it for free right now and not pay anything.
And that's whether you're open source or closed source or anything.
I could be working for a bank even, huh?
You could.
We have had users from banks working on it or using it.
So yeah.
Right on.
But there is some business model where you guys charge money for something.
So what are you, what is that side of the things?
Yes.
So for the code review, it's free for open source again.
But if you want to use it on a closed source repository, there's a small charge per developer
per month, basically.
And that's something we've only just released in the last few days, basically.
All right.
Cool.
So basically, if I'm going to apply it to my code base as a autonomous bot type of thing
and I'm open source, it's 100% free.
Right.
So if I was taking care of requests or SQLAlchemy or Flask or whatever, I could just plug it
into the Flask repo on GitHub.
And all of a sudden it would solve those problems of like, you didn't seriously just give me
a four loop that depends to a list, did you?
Right.
Yeah, exactly.
So yeah.
Particularly useful for people maintaining large open source libraries because they'll get a
lot of pull requests and they may come in at various standards.
So it does the initial code review for the maintainer of the project.
There's another way of trying it out.
If you have a GitHub account and that is simply to star our public repo and the GitHub bot,
our GitHub bot will find your most popular Python repository and send you a pull request
to refactor your code base.
So it's as simple as click a star and you'll get a pull request.
Oh wow.
Okay, cool.
Yeah.
I mean, it seems to me really useful to have it just built into GitHub automatically looking
over the code because I don't know.
You all have worked with different groups of people at different companies, different languages.
My experience has been that people that care about code quality and refactoring and testing
and maintainability and patterns and all that kind of stuff.
There's a massive spectrum on any given team.
Some people, it really matters to them and others.
Those failing tests and that failing build is just a nuisance.
And how do I turn off the build so I don't have to hear about it again?
Right?
And so having it as part of the repo means that I get kind of applies to everyone.
At least it suggests for everyone.
Whereas if it's just in the editor, there's going to be the people who love it and the people who just
like, how do I uninstall this or disable this?
So it doesn't, because it's just, I wrote my code and I don't want it to, you know what I mean?
Like there's just, it doesn't matter how much advocacy there is.
There's going to be that.
And so having that kind of external is pretty cool.
Yeah.
Like we started doing in the editor because we thought that was kind of the way to really
make you write code faster and kind of you hack it a bit and it straight away does the change.
But like you're saying, definitely that's why we introduced the GitHub plugin because the
code review, because not everyone's at the same level.
It kind of brings things up to a level.
Yeah.
So it's got, it gives you the benefit as a beginner programmer in the team.
So you get those code reviews, but also as the experienced developer, it saves you time
doing the code review because there's already a tool doing the simple steps.
It's not dealing with the architectural elements of it, but it's making sure each function is
nicely written.
Yeah.
I think that makes a lot of sense because it doesn't matter how good you are.
You don't want to have to go think of the implications throughout the whole code base.
Absolutely.
And we're just looking to say the tool set is good.
So it's good.
Press merge.
Yeah.
Beautiful.
But I do think it's really important that it's an editor because it teaches you, maybe it
could teach you.
It teaches you the idiomatic, the Pythonic ways of writing things.
You're like, I had no idea that I could create a for loop with enum that had tuple unpacking
instead of like trying to do a for over range and then pulling out the item and things like
that.
So it seems like a really cool combination.
Yeah.
I think definitely that educational thing is something we want to focus on more, like improving
our documentation.
So I wrote a blog post recently with a few little refactorings.
We're doing like why we think they're a good idea as opposed to just what we've done.
Is it the one that is called Python refactorings part one?
Yeah.
I looked through that.
So maybe you could give us a couple of the refactorings out of there that you like.
Code hoisting is like one of the best things because anytime you've got duplicate code,
you've got a way you can introduce mistakes really easily.
So that would be like maybe you have the same code in an if and an else statement.
Yes.
And it's just duplicated.
So often people will write sort of a bit at the end of the same thing in the if and the
else or in loads of elifs even maybe because it happens in every branch.
It means it always happens.
So it doesn't actually need to be in the condition at all.
And if you take it out.
Exactly.
Just put it at the end.
It also becomes kind of more clear what the conditional is doing, what it's controlling,
because it hasn't got this extraneous thing in it.
Another one you have in there is converting from a for loop, which does a yield to a yield
from that collection directly, which is pretty nice.
I mean, it might even apply to code that was written long ago before yield from was introduced
to the language, but yield was there.
And you could say, hey, look.
Yeah, for sure.
Nightgash, you've got quite a few comments.
This old way could be gone.
Quite a few comments that they didn't realize you could do that.
So it's like a lot of people aren't reading every kind of pep.
What?
Really?
And seeing everything they can do.
Strangely, it may seem to us.
I know.
It's so bizarre.
I think I'm sure there's peps that I don't read as well.
You know, if I had to pick a single most favorite absolute love it refactoring, it has to be
convert like a deeply nested set of code to something with guarding clauses.
So it's like flat, right?
Instead of going, if this is true, then if this is true, while this is true, if this is true,
and you end up like writing, starting on column 40 to write your code, if you negate them all
and like return early or break out early or something, it's just so much cleaner.
Yeah, so avoiding nesting is...
Okay, that case is out.
That case is out.
Now I focus on the essence.
Avoiding nesting is one of our course kind of code metrics.
Some of the other things I think we didn't touch on is how you get the computer to realize
that there's a code smell.
It's like writing good code metrics is quite...
How do you get a computer to know?
It's quite difficult.
So there's these metrics like cyclomatic complexity, which...
What's that about?
It's about avoiding conditionals, basically.
Number of decisions.
Yeah.
How many branches would you potentially go down, right?
There's kind of this enhanced version of it we've looked at called cognitive complexity,
which is trying to get to an idea of how hard something is to hold in your head.
And that really penalizes nesting.
How many variables are at play?
How many other things like that as well, right?
Probably.
Yeah.
So like that sort of penalizes nesting most of all.
So that's kind of like how sorcery knows not to...
Like nesting is a bad idea.
And then we've written metrics about...
Oh, I never really had thought about it that way.
But that's exactly the problem is like the reason it sucks so much is like that next test
is piled on as a and, and, and, and, and this.
And that all the stuff that you've nested yourself into, you've got to think of like all
those at the same time while I'm in here.
Yeah.
Cause the number of things you have to hold in your head, you know, a human can only hold
six or seven things in their head at once.
Yeah.
Maybe if you're exceptional, you can do eight.
So like some of our metrics is sort of focusing on how many variables you have to be thinking
of and how many conditionals you have to be thinking of when you're sort of halfway
down the function and it's gone off to the right somewhere.
So we actually call that the working memory metric.
Yeah.
Oh, cool.
That specifically measures the number of variables that are in scope at the current
time.
So we think you have, if you're reading the code from top to bottom, by the time you've
got to the 10th line of code, if you've got seven variables in your head, then you don't
understand the, you don't understand the function anymore.
You can't keep all that in your head and understand the next page.
So we keep having to scroll back and forth instead of just reading it.
Yeah.
There's this really interesting saying from a friend of mine that talked about, it went
something like when you write code, I guess debugging code is harder than writing code.
So if you write code at the very limit of what you're kind of able to write and do and like
the most complicated stuff you can do, you probably can't debug it because trying to think through
it actually is like a more complex than you kind of just pushed it over your limit.
And so there's anytime you can kind of dial that back a bit through refactorings or other
stuff, like, you know what, that should be three functions.
Then you won't have to think about so hard.
Sure.
And like the most three things in this part, so much about it.
The most interesting figure we found in a scientific paper that analyzed developers was sort of,
they spend 70% or we spend 70% of our time trying to understand the codes and only 5% of the
code time actually typing.
So it's that 70% of the time you really need to cut down on by making it more readable and
refactoring.
And you have the, that's kind of the whole Zen of Python, right?
I think that's why it's a popular language is because it's like clean to read presents itself
well.
Right.
So don't undo that by writing bad code, I guess.
Definitely.
Yeah.
All right.
Well, I think this is probably a good place to leave it.
You guys, it looks like a really cool project.
If people are using PyCharm or they're using VS Code, they could just go get the plugin and
give it a try.
Right.
Yeah, for sure.
Just search for sorcery in the marketplace of the ID.
Yeah.
Okay.
So you get it like as a, you go to the plugin marketplace in PyCharm or you do the extensions
in VS Code and it'll just be in there.
Sorcery with a U.
As in computer source.
Yeah.
Not as in Gandalf.
Yeah.
And also if you go to our website, it has full instructions for installing both the plugins
and using it on GitHub.
And it also has.
Right.
And people have open source GitHub repo, they should just drop it in there and it'll give
them some ideas, huh?
Yeah.
Give it a try.
Absolutely.
And links to our documentation as well.
Very cool.
All right.
Now, before I let you out of here, I've got the two questions I always ask at the end of
the show.
So we'll just be quick since there's two of you.
Brendan, how about you go first?
You're going to write some Python code.
What editor do you use?
I use Vim nowadays.
I ended up with wrist injuries from refactoring code using control and shift and the arrow keys
too much.
So I decided to learn Vim.
I've ended up with that as well a long time ago and had to like rejuggle a lot of interesting
stuff.
Have like funky, curvy keyboards and all sorts of stuff.
And yeah.
Try to use hotkeys rather than mouse a lot.
Yeah.
It was turning into a real issue.
So I had to learn Vim, which slowed me down by about 10 times for 10 weeks.
But now I feel as though it's magic under my fingertips.
That's awesome.
Nick, how about you?
Write Python code?
I use PyCharm at the moment.
So I'm a bit visually impaired and the high contrast mode is just really, really good.
Dabble with VS Code a bit.
I really like how it starts up super quick, but it's a little difficult to see.
So I've made the switch.
I can imagine that'll definitely push you over the edge.
All right.
Then notable PyPI package, maybe not something that everyone necessarily knows, but is like,
oh, cool.
I found this the other day and you should check it out.
Any ideas, recommendations?
Amazing one that we've used is this package called NUTKA, which is spelled N-U-I-T-K-A.
And it takes your Python code, cross-compiles it into C, and then compiles the C code and creates an executable.
And without that package, Sorcery just wouldn't exist as a locally running project because you'd have all sorts of deploy issues.
We'd just be delivering all of our source codes with the plugins and the extensions.
Interesting.
So you're packaging it up.
You're packing up Sorcery with NUTKA, huh?
Yeah.
Exactly.
Yeah.
How interesting.
Okay.
Yeah.
It's fantastic.
It builds in the version of Python that you're using and it reads all the imports to work out which bits of the code it needs to compile.
It compiles the whole thing down and it works on Mac Windows and Linux.
That's awesome.
It's magnificent.
Very cool.
I had Kay Hayen from the NU-U-K-A project on for episode 174, which is like a year and a half,
two years ago.
I don't know.
Quite a long while ago.
But yeah, that's super cool.
I didn't realize that it was so flexible in packaging up apps.
I thought of more as like Cython, like this little bit we can make faster.
So that's good to hear.
Very nice.
All right.
Final call to action.
People are interested in sorcery.
They're interested in refactoring.
What do you tell them?
Try it out now.
If you have a GitHub account, you can star our repo and try it out in five seconds.
Or you can install it and get all of your pull requests refactored.
If you're using VS Code or PyCharm, go and install it right now.
Try it out.
Get your code refactored as you work.
And let us know.
I mean, we're really keen to get feedback from people and keep on making it better and better,
basically.
Awesome.
Do you guys have like a GitHub repo?
Or how should they give you feedback or say, you know, my favorite refactoring is whatever
you guys don't do.
How do they make that happen?
Yeah, we've got the Sorcery AI repo where you can raise issues.
Or just email us.
And our GitHub repo is sorcery-iai slash sorcery.
Very cool.
Okay, there.
Awesome.
Well, Brendan and Nick, thank you both for being here and creating this cool project.
Looks awesome.
Thanks very much.
Thank you very much for having us, Michael.
Yep, you bet.
Bye-bye.
Bye.
This has been another episode of Talk Python to Me.
Our guests on this episode were Brendan McGinnis and Nick Thappen.
And it's been brought to you by Datadog and Linode.
Datadog gives you visibility into the whole system running your code.
Visit talkpython.fm/datadog and see what you've been missing.
Go throw in a free t-shirt with your free trial.
Start your next Python project on Linode's state-of-the-art cloud service.
Just visit talkpython.fm/Linode, L-I-N-O-D-E.
You'll automatically get a $20 credit when you create a new account.
Want to level up your Python?
If you're just getting started, try my Python Jumpstart by Building 10 Apps course.
Or if you're looking for something more advanced, check out our new async course that digs into
all the different types of async programming you can do in Python.
And of course, if you're interested in more than one of these, be sure to check out our
Everything Bundle.
It's like a subscription that never expires.
Be sure to subscribe to the show.
Open your favorite podcatcher and search for Python.
We should be right at the top.
You can also find the iTunes feed at /itunes, the Google Play feed at /play,
and the direct RSS feed at /rss on talkpython.fm.
This is your host, Michael Kennedy.
Thanks so much for listening.
I really appreciate it.
Now get out there and write some Python code.
I'll see you next time.
Thank you.