#251: Building and UX Testing Azure's Python SDK Transcript

00:00 Michael Kennedy: What does it take to build a Python library that will be used by a large number of developers? This happens all the time in open source. Projects take off and become wildly successful. What if you could sit down with the developers using your library and see how they take to it? Well, over on the Azure Team, Kate Olszewska and Johan Stenburg do incredible user testing for Azure's Python SDK. Just to give you a taste, imagine a room with a computer, a couple of developers unfamiliar with the API, a one-way mirror and lots of feedback. This is Talk Python to Me, Episode 251, recorded Monday, January 27th, 2020. Welcome to Talk Python to Me, a weekly podcast on Python, the language, the libraries, the ecosystem and the personalities. This is your host, Michael Kennedy. Follow me on Twitter where I'm @mkennedy, keep up with the show and listen to past episodes at talkpython.fm and follow the show on Twitter via @talkpython. This episode is brought to you by Springboard and Clubhouse, please check out what they're offering during their segments, it really helps support the show. Kate, Johan, welcome to Talk Python to Me.

01:19 Panelists: Hi. Thank you, it's great to be here.

01:21 Michael Kennedy: Yeah, it's great to have you both on the show. I'm really excited to talk about creating Python APIs and how you communicate those and document them for Azure. Azure is so huge and I haven't even done anything with Azure for a couple of weeks probably, maybe even a couple of months, counting some vacation time in there, and there's just so many things. So these libraries are super far-reaching, so it's going to be fun to dig into all those, I think. Now before we do though, let's start with your stories, how'd you all get into programming in Python? Kate, let's start with you first.

01:54 Panelists: Me first, okay sure. So my story is relatively short. I started with Python as the first language that I picked up, in my first intro to programming class and it kind of stuck with me throughout.

02:06 Michael Kennedy: Yeah cool, and did you start out in computer science just planning to go down that path or did you study something else?

02:12 Panelists: I was initially studying economics and kind of in a lot of data science that economists do. There is a lot of existing software but at some point I figured, there is more flexibility if you do some things on your own and that's when I started to pick up more data science through Python and that's what led me into it.

02:28 Michael Kennedy: Yeah, I never really thought of economics as having this super strong data science side to it but the more that I talk to data scientists, a lot of them come from economics background, it's pretty interesting.

02:40 Panelists: Yeah, there's a lot of analysis that's done on really large scale and there's a huge need and mostly what happens is that a lot of economists just work with data scientists to help them provide the data and the analysis, while they just provide, kind of contextual information but in the end, it's most efficient if you can do some stuff on your own.

03:02 Michael Kennedy: Yeah absolutely, it's definitely great if you can solve your own problems and not go asking for help, right?

03:07 Panelists: Yup.

03:07 Michael Kennedy: Johan, how about you? How did you get in programming in Python?

03:10 Panelists: Actually, I did not start until five years ago in Python specifically. At the time we were looking at rebuilding the client libraries and specifically in that case, the command line interface for communicating with Azure. We had one that was written in JavaScript and we decided that, that needed some overhaul so we looked into doing that in Python. So that's how I got started with Python.

03:35 Michael Kennedy: Right and what you were talking about, this is not just the libraries that maybe Python people might use, this is the command line interface for working with Azure as a Java developer or as whatever, right? It doesn't matter what language you are, this is the language you guys decided implemented in or move to at least.

03:52 Panelists: Correct, so it gave me my first introduction to the both, Python language but also the client libraries that we had to use, in order to implement the CLI but you're absolutely correct. That was an application that we wrote in Python.

04:04 Michael Kennedy: Right, where were you coming from before then? What language?

04:09 Panelists: Not surprising, lot of people at Microsoft write code in C# and C++, although I hadn't done much C++ in the last decade or so but I've also written code in Visual Basic. I'm probably one of the few people that have written Visual Basic .NET code that ships as part of Visual Studio so, but C# primarily.

04:30 Michael Kennedy: Yeah. What's the story with Visual Basic .NET these days? Is that still a thing? I mean, I know technically it's supported but it feels like the main story that Microsoft's telling around their own internal languages is C#.

04:42 Panelists: Most users, I'm actually not up to speed, haven't been on the Visual Basic .NET team. I left that in 2005 or something like that, so I've kind of lost contact with my friends over in VB .NET land.

04:54 Michael Kennedy: Sure, no worries. How did you get into programming in the first place?

04:58 Panelists: I'm from Sweden originally. We had a toy store in Stockholm, the capital of Sweden and they had a wall full of home computers, is what they were called at the time. Commodore 64s, VIC-20 then...

05:09 Michael Kennedy: To pose to the machine sized ones?

05:13 Panelists: I suppose the PCs that became popular later on and all of that. And there was always something fascinating about being able to tell a machine what to do. In all though, a kind of archaic way that you have to tell it what to do. Wrote things in BASIC at the time but, just playing with it and getting it to print your name over and over again. I think that was my first program as, it wasn't, "Hello world," at the time, it was, "Johan was here."

05:39 Michael Kennedy: It was, "Hello Johan."

05:40 Panelists: But, yes. I don't know, from that point on I realized that that's what I'm going to do when I grow up. That's what I did when grew up.

05:47 Michael Kennedy: Yeah, that's super cool. Well, both of those are really great ways to get into programming and cool stories. I guess, another follow-up question before we move off this intro bit here. So Johan, you had said that your first exposure to Python was when you guys rewrote the CLI in Python, which is good, but then you also, to implement that had to use all the Python libraries that other people who wanted to automate Azure or talk to parts of Azure with Python, had to use as well. What was that experience like? Was it like, "Oh, these are great libraries, they're really well put together." Or was like, "Wait a minute, this seems different than all the other stuff I see online. It looks more like C# in Python, .py files." I have no idea, I'm asking. What was it like?

06:36 Panelists: It was a little bit all over the place. One thing that we do here at Microsoft is quite a few of, what we consider to be the control plane, that is how you create resources in Azure, those are auto-generated from Swagger or OpenAPI. Which means that they are auto-generated libraries with all of the good and bad that comes with that. Other libraries or what we consider to be the data plane that is actually using the service after you've provisioned that uploading blobs, things like that, they were hand-authored. So there was a wide variety of what did client libraries look like, what patterns they used in there so it was a little bit all over the place.

07:18 Michael Kennedy: Yeah, so maybe just define Swagger and OpenAPI for folks listening, not everyone's API developer.

07:24 Panelists: Ah, sorry, sometimes I forget. That's part of my life as well, is reviewing the REST API so.

07:28 Michael Kennedy: Yeah, no worries. You're deep in it.

07:30 Panelists: Yes, deep in it. It's a way of describing a REST API, so basically, OpenAPI is an open standard that allows you to describe your endpoints, what kind of data that they can accept and what kind of data that they will return if you call them.

07:47 Michael Kennedy: So it's something like you guys maybe implemented your REST endpoints and, I don't know, ASP.NET Web API or whatever you guys chose and then you added Swagger to it and then everyone else is like. Hey, we need libraries for these, we can auto-generate some Python libraries or JavaScript libraries or, or whatever it is that you can auto-generate off of Swagger. Is that sort of how it went?

08:09 Panelists: Yes, so it's both that and we still do that. We have, there's a requirement for all auto-services to describe REST APIs using OpenAPI and it has a toolchain both to generate libraries as well as reference documentation and tooling for trying to ensure consistency among the REST APIs.

08:28 Michael Kennedy: Yeah, oh it sounds really fun. Now, next thing related to this. I want to ask you what you do day-to-day. I got a sense of what you do quite a bit there Johan but maybe give us the quick flyover and then Kate can tell us what she's up to as well.

08:45 Panelists: Yeah, so the two main areas of responsibility that I have is, I'm the architect for the Azure client libraries for Python, so I'm responsible for the overall design guidelines that we have, that are publicly available. I'm trying to help the people on my team figure out what the common pattern should be, what error handling should look like, what authentication should look like. So that's one half of my job. The other half of the job is to look at Azure's service APIs and try and ensure that we have as much consistency as we can on that side as well.

09:18 Michael Kennedy: Is that across different services or across different languages for a given service?

09:23 Panelists: It's across different services on the service API side of things, the cross-languages is a really interesting question, 'cause we have, I'm the architect for Python, I have counterparts for JavaScript, TypeScript, Java, for C# .NET obviously so, what we are looking at, is we're looking at these guidelines, the capabilities have to be roughly the same for all languages, if a service exposes a feature all languages need to be able to expose that feature. If a service has some specific failure modes or some quirks to it, everybody has to deal with that. So we're having discussions both how to best implement it in Python but also making sure that for all languages that we provide client libraries, that they all have same capabilities as well.

10:06 Michael Kennedy: Right yeah, that's got to be a interesting tension. Kate, how about you? What do you do day-to-day?

10:11 Panelists: I'm a program manager on the same team but on my side of things, we like to call ourselves just Azure Developer Experience because one aspect of having the great experience with the library is just having great libraries but another is also making sure that the experience you get when you come to Azure as a developer, is that everything is straightforward, if you go to the portal will you find the information that you need to get started and everything else. So we like to work with other teams across Microsoft and across Azure to ensure that we bring that consistency in other places as well. My main job is to bring the voice of the developers using Azure into the designs that we have, in all of these places.

10:53 Michael Kennedy: Cool, that sounds really fun and, you guys have some really neat stuff that we're going to dig into, that you're using over there. The scale of Azure, kind of like I was saying before, it's a little bit, I don't mean, I guess, if people use AWS, they probably have a similar feeling although I feel like Azure is even more so these days. You just go there and it's like, this huge menu of so many different choices, it's a lot to keep track and help people discover, right?

11:21 Panelists: Yeah, that's actually one of the main issues that we see for people who are just trying to play around with these platforms and I think that applies across both Azure and AWS and everywhere else. There's just so many different services to begin with that it might be kind of overwhelming to know where you need to go. So I think eventually our goal is to make sure that that choice becomes easier over time.

11:44 Michael Kennedy: Have you guys thought of things, like a beginner view or something for lack of a better word, something that hides a lot of the stuff that just says, "I'm a web developer in this language, just show me, the 10 things that I might care about."

11:59 Panelists: Yeah, I think we were trying to create some sort of track space on what we believe people might be developing in our documentation, so that it's easier to consume but I think, maybe that's a good idea to just have a very simple beginner view.

12:12 Michael Kennedy: Yeah, like show me everything, I don't know. My experience is there has been, it's like, "Wow, I'm on a bit of a treasure hunt." When I get through things here. Yeah, very cool. Now, you guys are also, Kate you're also involved in some of the documentation and tutorials and things like that. Johan, are you as well?

12:30 Panelists: Yes, yes I am. Yeah, actually on our team we're trying hard to have a greater push for creating better documentation overall. I think one of the problems with the auto-generated libraries is that, that sometimes ends up of being not the best documented library in the end and that's why all of our developers really focus and spend time on writing better comments and including more instructions for people to get started with.

12:56 Michael Kennedy: Yeah, some of those auto-generated documentation things, they can be really tricky because they could say like, the function could be login. The documentation might be, logged in user or upload, uploads file, right? It's just like one or two words where it just takes the name of the function and breaks it out into English-like words instead of compound words or whatever. Yeah, it's just like, yeah, you can tell that, "Okay this documentation, this is not going to be helpful for me," right? So it's good to have something really custom there, right?

13:26 Panelists: Yeah, exactly.

13:27 Michael Kennedy: Yeah, one of the challenges, I guess let me ask you Kate. How much are you focused on Python specifically and how much are you focused sort of cross-language, working on this stuff?

13:36 Panelists: I am focused on all languages to an extent because if we're creating some standards for documentation that should apply across all of them. Python shouldn't be getting some extra treatment just because it's Python. I don't see why not. Maybe that's just me. Sure let's talk about just Python, nothing else.

13:54 Michael Kennedy: I was more asking like, are you specifically on the team that does the Python parts and there's other people doing the same thing for JavaScript or are you overseeing it for all the languages.

14:04 Panelists: Our split, in terms of who cares about what is pretty complex. On the PM team, my focus was for Python in the sense of helping with user studies and helping with some telemetry searches but in terms of all of these kind of overarching themes, I work with all of the languages.

14:21 Michael Kennedy: Okay, yeah cool. This portion of Talk Python to Me is brought to you by Springboard. Springboard offers online data science and machine learning education, through their machine learning education career track program. This is similar to an online machine learning bootcamp, with the difference that this career track follows a project based learning methodology where students work towards creating their own portfolio of machine learning models. And every student is paired with a machine learning expert, who provides unlimited one-on-one mentorship throughout the program via video conferencing. This program was built for software engineers, so to be eligible, you must have at least one year of experience. You'll get a machine learning job, guaranteed. Springboard is the first company and still the only company in the US to offer a machine learning engineer career guarantee for students interested in changing careers. That means either you get a job in the industry or your tuition is reimbursed. So to get started today, just visit talkpython.fm/springboard or click the link in your podcast player. One of the challenges, I can see here right away and your joke Johan, sort of touches on it a little bit. I mean, I know you're joking but you've got some endpoints, some pod stuff that you want to control and do things and every language has its own idiomatic way of working, right? Python has Pythonic code, other languages have, whatever they call it, right? But there's always an idiomatic way of doing things and sometimes that really lends itself to doing something quite different than other languages. How do you trade off trying to make it, so that somebody who's a C# developer working with a Python developer, is not like two different worlds but also it doesn't feel like somebody put C# or JavaScript into a .py file?

16:17 Panelists: Yeah, you're absolutely right. That's a constant tension and we've had plenty of discussions among the architects for the various languages here, on how to accomplish exactly that and the way that you iterate through things is different a little bit in Python than it is in C#. For Java we have builder patterns for things.

16:38 Michael Kennedy: A lot of interfaces, lot of builders, lot of patterns over in the Java world for sure.

16:42 Panelists: Yes, so it is challenge. We're trying to make sure that we can talk, when we talk about the abstract concepts for the most part, that you can understand. You can basically say if you talk about a specific thing, a specific noun, be that a blob or something like that in one language, that you can map it down to what that means if I look at code in a different language. Exactly what it looks like is going to be slightly different because otherwise, you're stuck with something that is going to be looking like a specific programming language. We're certainly had that problem when I started here that, we had some jokes, some of the usability studies that we had, the participants were arguing about what language the original library was written in or the developer, what they language they preferred.

17:25 Michael Kennedy: Could they see the abstraction leaking through and go, "I know, I think I know what this is,"?

17:30 Panelists: Yes, there was one of the early Python library usability studies that we had where they argued whether or not it was a C# or Java developer that had written the Python library and it made me feel sad.

17:43 Michael Kennedy: Neither of those are the right answer. I don't know what the answer is but that's not right.

17:47 Panelists: Right. So.

17:49 Michael Kennedy: Yeah and Kate you're back there laughing. Do you have some stories like that, that you experienced as well?

17:54 Panelists: Yeah, I mean as Johan mentioned being kind of blamed of over C#-ing all of our libraries was one of the, I think initial kind of triggers that also made us want to explore the space we're in and launch our team to begin with to make sure people no longer have these doubts anymore.

18:13 Michael Kennedy: You know it's interesting, right? I'd like to hear what the user testing sort of uncovered for this, but to me when I use a library, I'm trying to write some program and then I'm like, "Oh, I might bring this library into the program." and it's clearly not following the idioms of the language. You can just tell, you're just like, "Ah, there's something..." You just get this feeling like, "This wasn't really made for me. It was kind of thrown together so I could still work with it, but it's not for me." You know what I mean? As a, whatever developer, right? If you're Python and your consuming something that looks like C# or you're a C# developer and you're consuming something that looks like C++ or whatever it is, it's not a great feeling like, "Oh, this is the place they want me to be." right?

18:58 Panelists: Yeah, no, absolutely. I do think, so something that I've learned working with the other architects for the different languages that, each programming language community has it's own personality as well. And I found the Python users are generally extremely pragmatic, which is something that I very much appreciate.

19:17 Michael Kennedy: Yeah, I would say that's a pretty good classification, right? They don't bring all these design patterns and these extra structures and whatnot in, just for the sake of saying it's well designed or well factored. It's like, "We'll do that if need to, but let's just put a function in a file and try that first." Something like that, right?

19:34 Panelists: Yep. So there are lots of other discussions on what is the best way of doing React in some of the other language communities. I have not seen too many of those things show up in Python.

19:48 Michael Kennedy: Yeah, that's a tough challenge. How do you guys, either of you, or both of you handle this because you have every language has its culture and its community but you can only immerse yourself so deeply in so many different cultures, right? And really get the zen of it. So how do you keep track of the other ones or is that just you work with the other folks that are more those language experts or whatever?

20:11 Panelists: The way our team is structured is that we have four separate teams with each of them has an assigned PM, an assigned architect and engineering lead and the entire engineering, kind of team. We're really trying to become experts in that particular language, so it becomes easier to track this as we have kind of separate entities that are caring about each language.

20:33 Michael Kennedy: Yeah, they have that expertise or whatever, right? And they could just express that, they sort of speak for their community at the meetings.

20:40 Panelists: Yeah, that is my role on the architect board that we have for the Azure SDKs. I try to bring the Python voice into the discussion. I'm telling them where we are going to do something different and what the reason for that is. Yeah for example, not from the Python universe of things but from JavaScript, we were wondering what would be the best way to present JavaScript documentation, should it be Node, should it be JavaScript, so we just brought in, I have no idea about anything JavaScript related, so I just brought in our engineering lead and our architect and kind of let them let me know better about what the community thinks about that particular space.

21:19 Michael Kennedy: Yeah, interesting. Well, there's just so many APIs and then you have to sort of cross-product that with how many different languages are consuming those APIs. It's really interesting. So one thing I think it'd be fun to talk about is the test setup that you guys have, Kate, for user studies and observations.

21:41 Panelists: Basically the way we just approach trying to create a library to begin with is we have to learn better about the service itself and we start out by writing initial preview versions of the library with guidance from the architects, as well as closer work with the service team. Once we have some sort of a preview to work off, we set up some user studies. We have a couple of labs in our building where we invite a lot of developers to come for, I think, two hours at a time. Where we set up some basic tasks for them to complete using the preview library that we have.

22:21 Michael Kennedy: By giving the library to Blob Storage and say, "Okay, use Python to upload this file, to Azure Blob Storage." So close them like that, right? And then just let them go?

22:32 Panelists: Yeah, exactly. So we try to start with some simple tasks to see how these initial steps are going and then try to proceed to more complicated tasks later on and then basically see how they proceed with that. We have kind of, two different separate types of studies that we do, one is focusing on the design of the API itself, where we often do not provide much documentation and see kind of, intuitively if the naming makes sense and where do people end up searching for them.

23:02 Michael Kennedy: Right, just auto-complete driven discovery, go with that and see what they do?

23:07 Panelists: Right, so in that case, it kind of helps us to better understand if our conventions of naming are discoverable and easy versus the other type of studies we have is to go more end to end and once we have some sort of, set of docs to go off, see how people actually go about discovering it and seeing if there are any gaps that we have.

23:27 Michael Kennedy: Yeah, quite interesting. So you're both checking the libraries and the API but you're also checking maybe the documentation and the tutorials in that second case?

23:37 Panelists: Yeah exactly, 'cause there is a lot of existing documentation for things that existed before us and making sure that all the places where people actually go to find it are updated or not misleading is a big challenge overall.

23:50 Michael Kennedy: Yeah, for sure. I mean, I can't speak for Azure in general. I know Azure's changed a lot and so that old tutorials don't make any sense anymore but in the Python space that is a challenge. You'll see tutorials written for Python 2 that'll tell people like, "Well go pip install virtualenv so that you can create a virtual environment." It's like, "No, no, no, that's already built into Python, it's a module, env. We don't need these things." Like, "There's these other steps." even there's new ways to install Python into Windows, right? Like from the Windows Store and whatnot. So it's got to be a constant challenge of like, that stuff was put there to help people and yet it's kind of in the way now. It'd be better if you didn't put anything and they might just have to go search for it and just maybe discover the right way.

24:36 Panelists: Yeah, I think for Python specifically, there has been an effort on our side to also try to consolidate all sorts of guidance exactly like you mentioned with the virtual environment, that is consolidated into guidelines on how we are supposed to write tutorials. I think it's still at a draft stage but it's something we're definitely looking more into to make sure that the guidance we provide for the tutorials and similar texts is also makes sense for the set of developers and for the language.

25:04 Michael Kennedy: Yeah, this question for either of you. How do you all deal with telling people to get started in Python? Because on Windows until recently they didn't have the python3 command, right? Steve Dower got the new version of Python into the Windows Store and it comes with python3 so that's one thing that you can just say, "Always type python3 to get started." but I've always found it very tricky to say, "If you're on Windows you do this, if you're on Mac or Linux you do that." And you kind of have to replicate that either or bits through everything, how do you all deal with that?

25:35 Panelists: Today it is, either or kind of for your operating system go here, kind of thing. We're doing less of now, when I think we used to do more of in the past, was that every getting started document had all of the instructions on how to set up Python first, which meant it was not until page three or something like that that you got to what we're actually trying to tell you. So now most of the getting started documents are that, "Assuming that you have python installed and by the way, if you don't know how to do that go here." So you just have that one little blurb at the top of the document.

26:08 Michael Kennedy: Yeah, that makes a lot of sense. Now I do want to come back to this lab that you have Kate 'cause I think it's really interesting from couple of angles. But let me try to describe what I think the lab looks like because I've seen some pictures and whatnot of it in action but this might not be the same thing. So the one that I saw, there was basically a room with some computers, some developers can go sit down there either like group program or do separate programming then there's also a one-way mirror and you could sort of screen share, you could watch what they're doing on the screen and from behind the mirror or somewhere else and sort of, critique how well your library is holding up to them poking at it and things like that. Is that basically what it is?

26:52 Panelists: It is pretty much how it is, so we do try to bring two developers at a time to do pair programming. We've kind of noticed that it's much easier for people to be more open about what they feel about the libraries and the experience overall. It's kind of more natural to speak out and comment and see how it feels. It's also usually easier to unblock the other person if one of them doesn't have experience in something.

27:19 Michael Kennedy: Yeah, I can imagine for a user testing angle that if there's more than one person that are working on it, they're going to just communicate verbally with each other because it's weird to just sit there and work with not sharing what you're up to but that's exactly what you would like them to do for you to understand what they're doing, right?

27:37 Panelists: Exactly, so even some more random comments are always great and we've had some people be very open about what they thought at the time of like, "Oh, this is crap, this needs to change. I don't understand why I'm even looking at this." And kind of having another person there I think, makes it less awkward for them 'cause you can imagine sitting in the tiny, tiny room, just you and a computer in front of you, knowing that you're being observed by x amount of eyes. It's a pretty stressful situation if you're on your own.

28:08 Michael Kennedy: Yeah, the other person will help you forget that there's all those eyes or thoughts thinking about what you're doing, pretty interesting.

28:14 Panelists: On our side of things, we also tried to bring in a lot of people so all of the devs that are actually working on the library try to always come to the study to see how their libraries are performing, so that it's also easier for them to kind of understand how users interact with the libraries and to understand where the struggles are. It makes it, kind of easier for them to understand how important it is to make it really good.

28:40 Michael Kennedy: Yeah, you'd think, "Oh, that's no big deal. People will figure that out, we don't need to do this." And then you see four people stuck for 15 minutes trying to do it, you're like, "Ah, actually okay, I see."

28:50 Panelists: Exactly.

28:50 Michael Kennedy: How often do you run these studies and how many people go through them?

28:53 Panelists: We're pretty much booked all the time. I think we have user studies, if not every week then every other week, depending on what we're currently working on. So we have usually, we manage to do I think, two groups of studies per day, which is four people then per day. Sometimes if there is something in particular that we're trying to test out we can use two, so that would mean most eight people a day.

29:19 Michael Kennedy: Yeah okay, well the reason I think this is so interesting is because there's so many people creating packages and libraries in Python for others to use and many of them are extremely, extremely popular, right? Over on PyPI we have 215,000 libraries at the time of the recording. You can go over to a place PyPI Stat, think, and you can check that out and you can see the most downloaded libraries, right? It's pretty cool, there's a bunch that are well over a million downloads a day. So there's these libraries and they're just being used by so many people, so many places and yet my feeling is that most people who build these libraries have very little visibility into how people feel about 'em, until they raised to the level of a GitHub request or a bug sort of complaint, or something like that, right? And having this discoverability and the ability to observe people working on your library, it must be really interesting.

30:19 Panelists: Yes, absolutely. I mean it's very gratifying when you hear someone talking about the library that you built in a positive way. It can be very humbling if you thought that you had a great idea that turns out that no one understands how to use or may even have other choice words for the experience that they feel like they were exposed to.

30:40 Michael Kennedy: Yeah, sure, "I had a rough day at work, people hated what I built, I thought it was great." That's pretty harsh but it does allow things to get pretty polished I suspect. So Johan, do you have some examples of things that you've changed or realizations you've had through some of Kate's studies?

30:56 Panelists: One of the experiences is, you touched on the tab completion kind of, of discovery of things but the Python community is extremely diverse in how they use things. I think about half of the people that we've had in the usability study did not use any form of IDE at all when they did things.

31:19 Michael Kennedy: Yeah, so you let them pick their IDE, it's not like, "Hey, we're all using Visual Studio Code 'cause it'll work with all the languages and it's consistent." or whatever you said. "If you want to use Emacs, knock yourself out. If you want to use PyCharm go for that." Is that how it went?

31:33 Panelists: The last couple of studies I think we set it up using VS Code but we've certainly had PyCharm, we've had people that even if you have VS Code in there, they only use the, well they use VS Code as an editor at most and then they drop into the terminal when they want run everything interactively or run things to the REPL immediately. The use of a debugger or similarly thing there I think, there was one quote that one of the participants had that, "Yeah I use a debugger for simple things but if things get complicated, then it's Pretty Print all the way."

32:06 Michael Kennedy: That's so funny and then I'm laughing because to me it's like the opposite. Right I'll print it out somewhere, this is getting hard man, I have to use the debugger and go figure this thing out.

32:13 Panelists: Yeah, but it's an insight into how people feel what people use and how important it is to make sure that things work well that you have a good, that you implement dunder str and repr so you can print out an object and you get some useful information and in fact, that we use it that way. So understanding the importance of that is something that I've come to appreciate more.

32:36 Michael Kennedy: Right, 'cause if you just print it out especially if you're in the non-IDE situation and you don't know what the properties are and maybe you've got something like a Blob Response object it would just say, "Blob response at memory address such and such," right? As opposed to here's the URL or the bucket and whatever, right? I think that actually is a good point in general for libraries to consider the dunder repr and str stuff so that not just if you actually just want to print it out but those appear all over the place, right? Those appear in debuggers, they appear potentially in log files, there's all kinds of stuff where having that it's just delightful. You're like, "Oh, I can just see what this is right here, I don't have to do extra work because somebody put in like a nice little repr that's good enough to know what's happening."

33:25 Panelists: Right, and so there are many of those small things that are there. I'm trying to reduce the number of types that we have in our libraries now, 'cause in order to call an API, if you want to call that from the REPL window, it's much easier to just use scalar types for example. And we have the wonderful keyword only arguments in Python that makes life easier, whereas in JavaScript I would create option objects with lots of properties on, we don't need to do that.

33:49 Michael Kennedy: Right, because in the option object you could have the default value but in Python, that's just part of the function.

33:55 Panelists: Right, so these kind of things that have changed a little bit and how I look at libraries.

34:02 Michael Kennedy: This is interesting about these sort of compare and contrast libraries trying to do exactly the same thing with people that know the system really well, I think that's super interesting. And I feel like a lot of that pragmatism that Python has is because there's a lot of flexibility in the lower level stuff, right? Like keyword only arguments with default values and stuff, allow you to avoid certain patterns or a global variable in a module, it's kind of like a singleton, that's what you need. You don't need a whole class for that and so on. What was your experience coming from the different languages and seeing that across teams? Is that about right or how's it been?

34:40 Panelists: Absolutely right. I still review Python code from other teams that come in and you can see where they're coming from, when they have extremely deep class hierarchies and things like that that, it makes sense if you will code in C# or Java but type-safety is not the same thing in Python as it is in C#, where you will get the compiler error if you try to call something that doesn't exist. There are other patterns that you can introspect things in Python that makes it more powerful.

35:09 Michael Kennedy: This portion of Talk Python to Me is sponsored by Clubhouse. Clubhouse is a fast and enjoyable project management platform that breaks down silos and brings teams together to ship value, not features. Great teams choose Clubhouse because they get flexible workflows where they can easily customize workflow state for teams or projects of any size, advanced filtering, quickly filtering by project or team to see how everything is progressing and effective sprint planning. Setting their weekly priorities with iterations and then letting Clubhouse run the schedule. All of the core features are completely free for teams with up to 10 users and as Talk Python listeners, you'll get two free months on any paid plan with unlimited users and access to the premium features. Go get started today. Just click the Clubhouse link in your podcast player show notes or on the episode page. Kate, one thing I want to come back to 'cause I do think it's really interesting and it's a challenge to this testing that you talked about is this not using an IDE or something that has some kind of completion story for working with the code that Johan talked about. If you're going to set up a study and say, "We're not going to give you much of the documentation, just here's the API." What do you do if somebody wants to use one of those non-assisting editors or you're working with a language that doesn't provide enough information to actually get that even in tooling that would otherwise show it, like some parts of JavaScript or something like that?

36:34 Panelists: So as I mentioned before, we have couple of different types of studies that we have. Some cases where we can't rely on people discovering things on their own, we provide them with documentation. We set up a couple of experimental documentation sites, which we can iterate on pretty quickly, where we just direct people to a site and they can look at reference documentation.

36:57 Michael Kennedy: I see, "So here's the SDK functions and types and whatnot, go look at that if you don't actually want to say dot and see the list." or maybe an addition. Yeah, what about code that you all put into documentation, right? You've got documentation for Python, you've got documentation for JavaScript, you both have these examples, these libraries are changing over time, the endpoints are changing potentially and extending or having better ways to do things. How do you deal with that kind of stuff and do you have somebody go through and just go, "Alright, time to retry all the tutorials." or what do you do there?

37:33 Panelists: Okay, so there is a couple of different things we were trying to do. So all of our packages live on our GitHub and one thing that we have already implemented for a couple of languages, but not all, is exactly as you mentioned. It's kind of challenging to keep control of all of the code snippets and samples, we have all across docs, across all of other places. So one thing that we're trying to do is to write tasks that we test all the time on an ongoing basis and basically be able to pull out snippets from these sources into our readmes and then eventually into docs. So that's something we have been putting a lot of work in. Already most of our stuff in our GitHub should be fresh because of this and we're now trying to ensure that the same thing happens when you're going to docs as we cross kind of reference our repositories.

38:30 Michael Kennedy: Yeah, that's really cool. You know the challenges that the little code snippet might need other stuff set up all around it, right?

38:37 Panelists: Right, the good thing is that usually in terms of the setup that surrounded that doesn't change as often because it's most having an instance of a service running or some sort of setting up the environment, which shouldn't change that often.

38:50 Michael Kennedy: Yeah, I'm thinking of a tutorial that says, "First import this thing." then a little bit later it says, "Oh, make sure you set this connection string." A little bit later it says, "Here's how you download a file." or whatever, right? And you want to test the, "Here's how you download the file." bit, the stuff that comes before it, still has to happen but it's not part of that snippet. You guys have examples like that you run into?

39:10 Panelists: Johan? Yes, so we do have that. We're using syncs for generating a lot of this, so using the literal includes, so you can basically have a comment in the code that says, "This snippet or this part of my function or method, corresponds to this example in there." So when you run through the whole thing it will do the setup and you can spread.

39:31 Michael Kennedy: I see, it'll chain them together or something like that. Yeah, okay.

39:35 Panelists: Yeah, the challenge is to keep everything up to date as you alluded to. If there's new functionality, it's relatively easy or not impossible to make sure that things continue to work, to make sure that things are using the best practices, if new best practices have come to light, it's much more challenging.

39:54 Michael Kennedy: Right, there's an old way of doing things, of course it still works because you wouldn't break it for old software but you don't want them that anymore, right?

40:01 Panelists: Right. So writing these examples, it's a little bit like giving someone a puppy. It's going to be work forever. That is true.

40:09 Michael Kennedy: Yeah, you got to walk it every day, even in the rain. Yeah, and how about testing these libraries, what do you all do for making sure that your library up to Azure or whatever service it's talking to, is still good? That's challenging and we were talking at the Python meetup here in Portland about this and it's like, "What if I wasn't connected and I wanted to run my test, how do I even get that to work?" Or, how do you test against real cloud environments but you don't actually want to create a whole bunch of cloud environments, just want to see if the API is still hanging together or something like that. It seems like a challenging thing to mock out all of Azure.

40:50 Panelists: Yes, so for what we're using is, that we're using a recording framework so basically record all of the requests and responses. Because it's very challenging and especially since we want to be able to take contributions from the public. We want them to be able to write tests and run tests obviously but as with all clouds, it costs money to run things in the cloud and we don't want to charge you.

41:17 Michael Kennedy: Yeah, some are cheap but some of them are really quite expensive to just do a little bit with, right? If you then provision, I don't know some type of VM with a GPU 'cause you need to test the GPU bit, I don't know, right? That could be expensive.

41:29 Panelists: Yes, so what we have is that we make these recordings available, so that you can run tests against these recordings. It's still challenging to come up with new tests. If you want to contribute the new test, honestly, we don't have a good solution for that today. But we're using these recordings as much as we can and then we run, we call live tests on a nightly basis to make sure that things still work when actually using the services.

41:53 Michael Kennedy: The recording is not lying. The recording is not out of date, right? What library are you using for this? Is this a Python library?

41:59 Panelists: Yes, it's vcrpy that we're using now.

42:02 Michael Kennedy: vcrpy that sounds very cool.

42:04 Panelists: Or pyvcr. Yeah, if you go our GitHub repository, github.com/azure/azure-sdk-for-python short name I know, but you can see that...

42:16 Michael Kennedy: Rolls of the tongue doesn't it?

42:18 Panelists: Yes, it does.

42:20 Michael Kennedy: Yeah, that's cool. You can see how you guys are using it there, yeah, this is really cool. I hadn't heard about this but I'm sure that'll useful for people out there in general, right? Not just for people working with Azure tests.

42:29 Panelists: Yes, and this is not the library that we have produced, this is an open source component so.

42:36 Michael Kennedy: Yeah, absolutely. So, how do you deal with, it's still hard for me to get my mind around, like how many challenges you have for trying to keep all the stuff consistent. So you've got all these different endpoints, you've got these different languages, how do you collaborate at the documentation level for, "Okay, we're going to create this new thing that helps people use this, the JavaScript team's got to do it, other teams got to do it and they got to be consistent." What guidelines and advice do you have for that?

43:03 Panelists: For the documentation specifically or for the designs?

43:05 Michael Kennedy: Yeah, more for the documentation I guess, is what I was asking. Although both are interesting but there's a lot of people or companies out there that have multiple technology stacks talking to the same thing. Like what advice do you have for them to help all the different consumers evenly?

43:21 Panelists: One thing that we are trying to do is to first of all better understand in terms of these different technology stacks, what are the most kind of, often used patterns and that kind of informs how we write the quickstarts and the tutorials. On their side, the process the way it looks is that our developers writing the libraries provide the reference documentation and then includes some sort of basic guidance of what they see fit for the readmes that we have on our GitHub. And then for each service we have a content development team, that kind of tries to take it and write a more, some sort of a tutorial that's more digestible by all sorts of different audiences and they have better understanding of what exactly they need to provide for these particular use cases in these different scenarios.

44:10 Michael Kennedy: Okay, so there's a like a canonical example and this is what we want it for here and then make a JavaScript version, a Python version and so on.

44:18 Panelists: Right exactly.

44:20 Michael Kennedy: Okay, that's quite cool. So this whole user testing of people consuming libraries. It's really interesting and like I said with the PyPI examples and those popular libraries, know I suspect very few people are doing that, if you exclude like the Azure libraries and other big companies but open source teams they have these sprints and they have other feedback mechanisms, all the stuff we talked about. Do you have any advice where people might be able to recreate a little bit of this? Maybe at the end of PyCon have a sprint where instead of trying to add new features, it's like, "Let's do some user testing, for people who are attending the conference and are very passionate about our library." or maybe try to grab somebody who's used the other library that's like Django versus Flask or something and have them come over and work in our world for a little bit, without having one of these labs and whatnot but maybe using things like Live Share or something. What advice do you have for bringing some of these cool practices that you guys are doing and applying to your libraries to maybe much smaller groups or open source projects?

45:29 Panelists: I can start and Johan can nod in but in terms of the conferences, I think that's an amazing idea and that's exactly what we're doing. In November we were attending a Microsoft conference, Microsoft Ignite and that's exactly what we were doing. We had a couple of stands setup, allowing people to kind of try things out on their own as well as take surveys based on the feedback that they would've gotten by completing some basic tasks and kind of taking that down and kind of informing for their designs. So, yeah I think it's an absolutely a great way to easily be able to work with people directly if you don't usually have that in person.

46:06 Michael Kennedy: Yeah, it seems like it would be a cool thing for conferences to set up right? I mean at Ignite, you guys set that up and that's really cool 'cause you're doing that sort of stuff but I'm thinking at PyCon or some other major JavaScript conference or something, right? The conference organizers could have a section where you could sign up, "Hey, I'm this library developer, I'm on this team, I want to have people go through my stuff." And I just think that'll be great. I mean, I have no ability to set that up, right? I can throw ideas out there, maybe people will run with it but it seems like it'd be really cool.

46:36 Panelists: Yeah, that seems like a really good idea. One challenge, I think that we've seen with that is that very often, people are happy to provide some feedback if this is something they really care about but if that's not the case, then usually they would expect some sort of swag or something in return, which is fair.

46:54 Michael Kennedy: It's fair and as companies it's fine, right? It's easy for Microsoft to put together a certain amount but as a solo developer on a open source project, it's much harder. But maybe like a cool little badge that's got the picture, like a little logo and like a, "I helped improve this at PyCon." or something or, "I helped improve Flask." or whatever. I think people can make it work without too much involved.

47:18 Panelists: Yeah, that sounds like a good idea.

47:19 Michael Kennedy: Yeah, Johan what do you think?

47:20 Panelists: You can look at it as a contribution. It may not be in the form of source code as well, so having the contributions somewhere and who contributed to the last release, usability feedback from people.

47:31 Michael Kennedy: That's a pretty good piece of currency to work with, that they do have something that they could hand out easily, right? You don't have little robots that you can hand out for everyone who helps as an open source developer but you do have, like you said, credit. That's cool.

47:44 Panelists: Yeah, and it's important as well to be clear on the fact that we have seen a little bit of bias that people want to be nice to you, when they give feedback but what you really need is the harsh feedback as well. So that's one thing that I think we're getting better at making clearer that the participants that we want to hear the good and the bad and not just the good.

48:06 Michael Kennedy: Right, "Tell us the truth and not just what you think we want to hear." Even though it will make your day not as good I guess.

48:13 Panelists: Right .

48:14 Michael Kennedy: Yeah, nice. Alright you guys, we're getting short on time here, so let me just ask you the final two questions and wrap it up. So, Kate we'll go with you first. If you're going to write some Python code what editor do you use?

48:28 Panelists: I recently started using VS Code and have been enjoying it.

48:32 Michael Kennedy: Yeah, you guys are doing a lot of cool stuff with VS Code, it's definitely growing quickly. Johan?

48:37 Panelists: VS Code is probably my go to editor right now but I try to switch between them. Just to get a feel for what the experience is for using our libraries in the different editors.

48:47 Michael Kennedy: Yeah sure, you definitely want to know, like, "What's the experience in Sublime?" Or, "what's it like in VI?" Or whatever, right?

48:53 Panelists: Right, I switched between them, basically every week I use a new editor.

48:58 Michael Kennedy: That's got to be a little hard. I mean you get used it and all but...

49:02 Panelists: Plenty of mistakes, just yes.

49:05 Michael Kennedy: Yeah, I bet. And then I'll throw this out to both of you, you either can throw something in, a notable PyPI package, some library out there that you're like, "Oh, this thing is super cool, use this." Maybe people haven't heard of it and if you want it, you're welcome to use vcrpy. If you got another one, throw it out there.

49:23 Panelists: That's a good question. That one I would have to think about a little bit.

49:26 Michael Kennedy: Alright, how about vcrpy? That's a good one, is it pyvcr or vcrpy?

49:31 Panelists: Yep.

49:33 Michael Kennedy: There is a vcrpy and it sounds like it's the right one but I don't want to recommend swapping it to, the wrong one to people.

49:40 Panelists: Yeah, it's vcrpy.

49:42 Michael Kennedy: vcrpy, okay cool. Yeah, that's if you have things that are really hard to mock out and there's a complicated exchange, that seems like that's a pretty cool way to do it.

49:51 Panelists: Yep.

49:51 Michael Kennedy: Nice, alright. Final call to action, people are out there, maybe they work at a big company, they've got a set of APIs they got to document, across languages or they work at some other company that exposes their API to the world and they've got to doc, what advice do you have for them? Take some of these things you've learned and apply it in their world.

50:10 Panelists: It's not free to provide a good developer experience for people. So understand that you need to, if you want to have a good developer experience, you need to invest time and effort into doing it so. If you think that you can get away with having the same documentation for every language, auto-generate things, you will get part of the way but it will not be great.

50:33 Michael Kennedy: Yeah, absolutely. It'll definitely show. Kate, what do you think?

50:36 Panelists: I think another important aspect is ensuring that whatever you provide is really well documented. Doesn't matter how much work we put into making sure that things are consistent and idiomatic in the end, people rely heavily on using documentations so making sure you create an easy and kind of consistent story around that is super important.

50:58 Michael Kennedy: Awesome. Good advice and thank you both for being here and sharing what you're up to. It's really cool.

51:03 Panelists: Thanks for having us. Thank you.

51:05 Michael Kennedy: Yep, bye. This has been another episode of Talk Python to Me. Our guests in this episode were Kate Olszewska and Johan Stenburg. It has been brought to you by Springboard and Clubhouse. Become a machine learning professional with Springboard's online bootcamp and get a job guaranteed. Visit talkpython.fm/springboard to apply today. Clubhouse is fast and enjoyable project management platform that breaks down silos and brings teams together to ship value, not features. Fall in love with project planning, visit talkpython.fm/clubhouse. Want to level up your Python? If you're just getting started try my Python Jumpstart by Building 10 Apps course or if you're looking for something more advanced check out our new Async course that digs into all the different types of async programming you can do in Python. And of course, if you're interested in more than one of these, be sure to check out our Everything Bundle. It's like a subscription that never expires. Be sure to subscribe to the show. Open your favorite podcatcher and search for Python, we should be right at the top. You can also find the iTunes feed at /itunes, the Google Play feed at /play and the direct RSS feed at /rss on talkpython.fm. This is your host, Michael Kennedy. Thanks so much for listening, I really appreciate it. Now get out there and write some Python code.