James Pogran is on a mission to simplify the developer experience for Puppet practitioners. He's here to talk about the Puppet VS Code Extension.
Puppet's developer experience has been through a lot of growing pains. Many of us learned Puppet before the PDK existed, when the state of the art was SSHing into a Puppet master, manually making a module's directory structure, and cracking open Vim to edit our manifests. Some of us still do that; old habits die hard. But the point is that the barrier to entry used to be really high, and it didn't have to be. James Pogran is a Puppet developer on a mission to change that and simplify the experience for practitioners of all levels. He's here to talk with us today about his work on the Puppet VS Code Extension.
Ben Ford [00:00:20] Hey everyone, I'm Ben Ford, a software engineer and developer advocate here at Puppet. Like you probably are, we're all working from home now and this is our first time recording a fully remote podcast. But we'll have fun figuring out the tech. Anyways, I've been with the company for quite a while now and I've been through a lot of our growing pains. I learned Puppet before the PDK existed -- that's our Puppet Developer Kit. That was back when the state of the art was SSHing into a Puppet master and manually making changes, you know, editing files, busting them open. I mean, I still do that sometimes. Old habits die hard. But what I'm saying is that the barrier to entry was really, really high and it didn't have to be. So that's why I'm so excited to introduce James Pogran here. He's a senior software engineer at Puppet, and he wrote the book on learning Powershell DSC before he came to Puppet. Now he's working on a brilliant Visual Studio code integration.
James Pogran [00:01:07] Hi, it's James Pogran. As Ben said, I'm a senior software engineer here at Puppet and I've been working on the Visual Studio code extension for Puppet for about three years now.
Ben Ford [00:01:18] I suppose that means I should be a little bit more careful with my use of the word integration instead of extension.
James Pogran [00:01:25] Yeah, they've focused on using extension. Sometimes they use plugin, but it's really extending the IDE so that it can support different languages. It's a core concept with Visual Studio code is that it's a text editor that gets out of your way and presents the language which knows best about what grammar and syntax and errors and linting and stuff, instead of trying to force a certain way of using a language,.
Ben Ford [00:01:51] That's a really interesting kind of distinction there. So what exactly does the Puppet VS code extension do?
James Pogran [00:01:58] Well, when you open up any kind of Puppet module or Puppet manifest or Bolt plan or a task, it will determine that that's something that's a Puppet language and it will start loading a language server in the background. And what that means is that there is a Puppet process that will then parse your manifest files as you type, and that will provide syntax highlighting intellisense of Puppet types, functions, classes and other methods. It will give you code navigation so you can bounce between files. You can find where a type is defined. It will give you symbol support so that you can find where variables are looted or where functions are called and will give you fact resolution so that you can see for which operating system this particular fact is going to resolve to. It also give you a linting and error checking as you type the file in front of you. So instead of having to hit Puppet apply or run Puppet parse and validate, as you type, those errors will pop in front of you, which really reduces the cycle of feedback of compiling code and seeing if it's syntactically correct.
Ben Ford [00:03:06] That's pretty cool. Something that I'm interested in is how it knows what facts to use since those are generated on the agent side before the compiler ever takes over. Where do you get the source of those facts?
James Pogran [00:03:19] So we're using Puppet itself and it uses--depending on how you started the code, it'll use your operating system, or if you've done something like starting in a container using the remote development extensions that VS code has. It'll start it as your container or your VM or the server, your SSH. And so it's whatever local process is bringing the language server back in, which opens up some pretty cool opportunities for you to test your code without actually running your code per se. It'll be in the context. So for example, you're sitting on your windows laptop, but you're writing Apache Puppet configuration and you're using the Microsoft remote extensions to SSH over to your server and edit the code over there. The language server, our Puppet extension rather, is remotely copied over to the server and run in a separate process. Thinks it's sitting on the language server, but really you're writing the code sitting on your desktop. So your facts resolution, your process, so access to all the modules on the server, all of that stuff is remote, but locally you have your desktop, so it opens up some different areas or ways of editing Puppet code without actually having to have your local environment have everything set up for you.
Ben Ford [00:04:37] That's actually really cool. And I want to dive into that even a little bit deeper because I think that it might have skimmed over something that may not be quite as obvious to people listening. If I'm understanding you right, you could theoretically start up three or four or five BMs on your your system or have easy two instances or something like that. They represent the different types of nodes that you have in your infrastructure, and do like real live validation of your code as you write it on the types of machines you're going to have in your infrastructure. Am I understanding that correct?
James Pogran [00:05:11] Yes. And it can be even more flexible than that. It can be SSA change into VMs that represent your servers. It could be local Docker containers. We have the PDK Docker image you can build from or reference or even use your own containers with your tooling installed and it will use that instead. And then you can start VS code inside that container and so your local workspace is mounted in that container and you can treat it as if you're sitting in that interactively. If you're on Windows, you can use WSL or the Windows subsystem for Linux with PDK installed in your distribution of choice and not have to touch Windows at all and you can write your Puppet code that way. It really opens up many different avenues for your development so you can choose how you want to or how you'd best perform and go that route. So there's less "do it our way" and do it more your way.
Ben Ford [00:06:06] That's pretty cool. I think I finally understand what the big deal about Microsoft's remote editing capability is.
James Pogran [00:06:12] Yeah. We've been trying to write some documentation, some best practices and using it. We're almost done with that.
Ben Ford [00:06:19] Right on. So you mentioned the language server, does that -- how does that relate to VS code, to the extension and to maybe even theoretically something else?
James Pogran [00:06:30] So the nice thing about Visual Studio code is it adopted a protocol-based system. So each language speaks a protocol back to the client frontend VS code and that abstracts the determination of syntax or linting or error checking to the language itself, so a separate process. That communication back and forth means that we don't necessarily have to have VS code be your frontend. Your frontend could be VM or neo-VM and you could still get the same benefit of the code that we wrote that does all of this magic stuff in the background. So we do have users out there who are using our language server except in neo-VM and you'll get the same error checking intellisense and autocomplete that you would get if you're using VS code, except in your favorite editor.
Ben Ford [00:07:19] That's pretty cool. You've mentioned the PDK a couple of times. How does the extension integrate with the PDK?
James Pogran [00:07:25] So it's a symbiotic relationship. Really, it both needs it and utilizes it, and extends access to it. We need the PDK because Puppet development is slightly complex in the sense it needs a Ruby version. It needs a Puppet version. It has some gems depending on which kind of solutions you're doing and which modules. You may have several different versions of all of that together. So the PDK makes that simple in terms of setting up your environment to access those kinds of gems and versions. The language server is Ruby process, so we need to be able to load Puppet and load your particular module in all of its gems and versions in order to cheat and not have to bundle our own versions of this, we use the PDK. So that's why that's our requirement when we start our Puppet extension is we use the PDK to get access to that stuff. But beyond that, we provide commands inside of VS code that you can, from the IDE, run PDK commands or run Puppet commands when you're inside the editor. So then it goes and does things like create new modules or create new classes. So if you're comfortable with the terminal, you can still type PDK commands from the command line, but you are going to be able to use VS code accelerators like the command pallette and be able to type as you're editing code, creating new files, but you're adding new classes rather or types or test files. But you don't have to drop down to the terminal. You can use the commands that keep you in the editor, keep you close to the code.
Ben Ford [00:08:54] So it sounds like VS code compatible, for lack of a better word, or modules that you've generated with VS Code are compatible with the PDK and vice versa. Modules you've generated with PDK are also compatible with VS code and you can use, kind of mix and match, use either one of them.
James Pogran [00:09:10] Yes, the VS code is using PDK to generate the modules, if you go that route or if you're creating classes inside VS code, choosing the PDK to do so. So everything is compatible both ways.
Ben Ford [00:09:23] That's really cool. And it sounds like as a PDK advances and does things like add Litmus support, etc., you'll get some of that just kind of built in as it grows.
James Pogran [00:09:33] Yep. You might see some modules out there that actually use a function of VS code called tasks, modules, tasks. Everybody uses the same terms over and over again. So the VS code tasks as a JSON file that you can put inside your project, and you can define simplistic or somewhat complex commands that can be run inside VS code as background processes or things that kick off based on events. Some of the projects out there, including some at Puppet, have utilized that to run Litmus tests. So instead of having to type the complex Litmus command when you're going through development, we'll hit a five and it'll go and run that Litmus command for you. So it'll run your tests or whichever keyboard combo you decide to set for that.
Ben Ford [00:10:17] That is really, really cool. I have a couple of questions, kind of like about how you got into this and how we can get other people involved, other interest in the project. So how long have you been working with Puppet code?
James Pogran [00:10:30] Well, with Puppet code in general, I've only been working with that since I started at Puppet about, almost five years ago. I didn't come to Puppet knowing Puppet code from a developer standpoint or even from a user standpoint. I came to Puppet from knowing DSC standpoint and helping Puppet to write the DSC modules at the time, which was very new. But in terms of the reasons why I started with the code extension is that I found editor support was spotty for new people like myself who are just trying to learn. I have a documentation window open at the same time I had my code window open. I had to look up things before I type. If I had an error, I had to run several commands before I could figure out if I was wrong. So that was really my motivation in starting this. I wanted something that helped a little bit more. Coming from a .net C-sharp background, I was used to editors that gave you intellisense and auto completion inline help, things like that that make it so that you don't have to leave the editor so much in order to be able to accomplish your job. So when VS code came out and they started trying to push people to make extensions for it, I really saw it as an opportunity both for myself, I wanted to learn typescript and I wanted to provide a better editing experience for Puppet. So the first versions just use the Puppet command lines. It was basically just a wrap around that. The phrase "great minds think alike" comes to mind. Glenn Sarti, another developer here at Puppet, he was also doing the same thing, except he was approaching it from the language server side, which was trying to run a Ruby process that loaded Puppet and that would compile your code and provide all of the effort. So we combined our efforts and made an extension that used the language server, but also the approaches that I was doing. Fast forward three years later, we've had almost twenty-five releases, a couple of hundred features, things like that. And I'm really proud of how far we've come in terms as the extension and what it could do.
Ben Ford [00:12:29] I would be proud too. That's a really big accomplishment. And I think it kind of highlights something that we've been trying to say to people for a long time and that working with Puppet really isn't all about like knowing the language, and where to tease about how Puppet works, and more about kind of understanding the concepts of configuration management itself and letting the tooling kind of help you guide you along. So if somebody wanted to get involved with this project, what would you say is the most straightforward way to get started? How can somebody find out about it? How can somebody find, like start helping with small things before they get into the bigger, more interesting thing?
James Pogran [00:13:07] Well, I'm sure we'll have a bunch of links at the end, but if you go to our GitHub repo page or use VS code and open up the extension pane and see our extension there, it'll link you over to our GitHub repository. We recently moved it over to the Puppet Labs namespace on GitHub, and that's the best place to get started in terms of finding maybe issues that we already have open or something that you want to raise with us. In terms of contributing, we have a contributing doc that says part of our repo. Since almost the first day we've done this, we've tried to make sure that all of our development is done out in the open. We'll have conversations and issues. All of our project work is done using GitHub projects and we've been doing that for about two years now. So if you want to participate, we accept PRs and we have a very low friction process for merging them in. The, it's like most open source projects; open an issue, discuss general direction and then submit a pull request. Glenn and I will review it. We've merged everything that has been contributed so far.
Ben Ford [00:14:13] That's a pretty impressive stat. What does your roadmap look like?
James Pogran [00:14:16] Well, right now we just completed a transition to move the extension from the personal namespace that I had it on before into the Puppet official namespace. We just shipped a rewritten node graph visualizer. That's a very big, wordy sentence for something that will take your currently open Puppet manifest and visualize it in a directed graph so you can see exactly how the relationships are happening inside your classes or your types. That was just released and is using a new data visualization library. And most of our clan features for the next month or two is based on taking advantage of that library is used for scientific data analysis, so it has some pretty cool features for adding nodes dynamically. That means you could type something in your file by adding a new type or new class in and they'll show up as you type different visualization methods. So the data library for scientists, there is all of those nice mathematical algorithms for figuring out dependencies. So we have a lot of those things planned for the next two months.
Ben Ford [00:15:32] That's really cool. I don't mean to interrupt, but it sounds like you're saying that you can pull this open like maybe in a second window, or side by side or something, and like have a live visual representation of the code that you're typing; as you're typing it nodes will pop into view and you can see the relationships being built.
James Pogran [00:15:50] Exactly. That's a common concept in the VS code to have, one of its actually first features was markdown preview to the side, and that was one of their first big demo things where you could type markdown and it automatically parses and displays it right next to you as you're typing the code, or the text, and it gets converted straight to markdown. So that same concept is what we wanted with the node visualizer. You're adding new types to your manifest and creating a class and it'll automatically change that representation to the right or the left hand side, whichever you had it displayed at. And so it will help you both visualize and also make sure that it's syntactically correct at the same time.
Ben Ford [00:16:30] That's really cool. We'll have to make sure to include like a screen capture or something at the bottom of the podcast, as I'm sure that people are going to want to see what that looks like. What's the name of the library that you're using?
James Pogran [00:16:42] It's cytoscape.js so it has a couple of different ports for different other languages, but we're using the street JavaScript port because that gives us a nice web panel inside of VS code. VS code itself is a node js process. Most extensions are written in typescript, so it gives us a nice non-barrier language so we can directly interact with the DOM and pass messages back and forth between our language server and the JavaScript process without having to have an intermediary. The rewritten part that I was referencing earlier is we used to have a dot graph representation pulled from Puppet. And now we send the actual objects as JSON objects. And since this library accepts JSON objects, there's a lot less juggling that we have to do to make the data representations happen so we can use more of the power of the library and less code on our side.
Ben Ford [00:17:39] That sounds really useful. I might have to look into it for a couple of projects of my own. So what do people need to get started using VS code in the plugin?
James Pogran [00:17:47] Well, that's the nice thing about VS code is that if you opened up a Puppet file, it'll prompt you and let you know that the marketplace may have an extension for you. So if you opened up a Puppet manifest file or a Bolt project, it'll prompt you and say, go search the marketplace and it'll open up the built-in extension gallery and it'll show the Puppet extension right there. If you click install, it'll install the extension within a couple of seconds, depending on how fast your Internet connection is. And then it'll actually, nowadays it doesn't even need to reload the window. Our only requirement on your local system is having the PDK installed.
Ben Ford [00:18:22] And you should have that installed before going through this process?
James Pogran [00:18:25] You can install it after if you install the extension. Our website has a QuickStart of what I just said too, so you can reference that as you go through too. But if you don't have the PDK installed before you went and installed our extension, our extension will prompt you and say, hey, we can't find it. Maybe go over to here. Here's a link to go get it. You can go click and it'll bring you to the PDK download page and Puppet.com. Once you have the PDK installed, it'll ask you, can you restart, reload and reload the extension and it will immediately notice your environment and start being available for you to use. If you used a container or a WSL environment, just have the PDK container or PDK installed in your WSL distro.
Ben Ford [00:19:07] How difficult is it to set up with the PDK in a container like that? Like do you have to install visual studio code itself in the container or do you just tell it to use that container somehow?
James Pogran [00:19:18] So part of the Microsoft remote extension pack has a container extension in a SSH extension and a WSL extension. And those are responsible for loading the environment for VS Code to use those particular environments. So in terms of the container, it will reload VS code using the workspace that you are currently in. So in our case, it's a Puppet module and it will reload the window and present the Docker container as your interactive environment. So you'll get if like you're using Alpine or something like that, you'll have your Alpine bash shell in front of you. File system shows up as an alpine container, but it's from your local system, so that has PDK in it and you can use your commands inside the bash shell, inside the VS code window, but your host system didn't have to have that. Not sure if I'm making sense there.
Ben Ford [00:20:10] Well, I'll tell you one thing. I expected it to be an intensely convoluted process. So even if this itself doesn't sound like it makes a lot of sense, that's far, far easier than what I thought it was going to be.
James Pogran [00:20:22] The VS code website has a walkthrough of what I'm talking about, and to make it even easier, what I forgot to mention is if you open up a brand new workspace, but you do have the Microsoft remote container extension installed. There's a command pallette command. Again, reusing these words that are overloaded so many times, the command.
Ben Ford [00:20:45] Puppet doesn't know anything about overloaded words, you know?
James Pogran [00:20:49] No, what are you talking about? Haha. The command's called adding a dev container file. I forget the exact syntax of the name here, but what it does, it adds a JSON and a Docker file to your workspace. It'll prompt you first though, do you want to set up in a particular environment? And so it'll list all of these environments, everything from .net to Azure to Ruby to AWS, all of these different pre-canned environments. But they accept pull requests. So when that first came out, I submitted a pull request to add Puppet as one of those. So if you start typing Puppet in that dropdown, you'll get a pre-canned Docker environment set up for you and that will pick a Docker container that is compatible with PDK, build that for you real time in front of you and have it set up so you wouldn't need to have anything installed in your local system except Docker VS code. A Puppet in a box, almost.
Ben Ford [00:21:43] Seems pretty easy to do to me. I think we already covered this, but do you have anything else like exciting that you want to talk about or anything you want to sort of tease anybody with?
James Pogran [00:21:52] Glenn has been working on a -- very experimental right now -- but he's been working on some type inferencers. And what that means is that since Ruby is a dynamic language and Puppet is based on Ruby for certain parts of its language, there are some things that we can't do with figuring out what it is in open in front of us without compiling the code. You were referencing catalogs before. Some of the language is just too dynamic to figure out from a flat text file. Glenn has been working on a piece of code that will make guesses based on Puppet types and other conventions inside of Puppet code. So we can do something like for instance, a Bolt plan where we can't get intellisense for some of the objects that a Bolt plan has. But because of Glenn's inferencer, you can actually type the variable inside your plan and get a listing of all of the properties and methods for the target object inside of it.
Ben Ford [00:22:53] Oh wowl.
James Pogran [00:22:54] You couldn't get that before because that's compile time. That's something based on the Puppet language and the Bolt internals. But Glenn is able to do with his library based on his code and that's something that's he's active developing on.
Ben Ford [00:23:08] So it kind of takes a guess as what it might be. So it's not static analysis. It's kind of guessing what you could be looking for.
James Pogran [00:23:16] Correct. A simpler example is variables. Variable could be a string, could be an integer. It all depends on what's resolved in compile time. This one will make guesses based on what the variable's set to inside the text in front of you. And it'll tell you, this is a boolean. This is an integer. This is a string. And it'll give you the appropriate methods or the auto completion for that particular variable.
Ben Ford [00:23:38] That'll go a long way in helping resolve some errors that people have with Puppet code, especially since the language has been kind of so loose about what a boolean means, or what things are coerced into strings, when I think that as we get stricter about the, about these things, this will be really helpful to help people come along.
James Pogran [00:23:57] Yep, that's where Glenn was trying to solve. Speaking of errors, something that's a little less experimental that we dark-shipped, kind of, was a Puppet file resolver. If you open up a Puppet file in any other kind of editor, you'll just get a text file representation with maybe some syntax highlighting. If you open up in VS code, you'll get a language. The server will actually resolve the references that you have in your Puppet file and it'll determine if those references are up to date. And then I'll go one step further and say that you are specifying, let's say, for instance, Powershell. But it has a dependency on making something up now on standard lib.. But that version dependency that you list in your Puppet file is invalid. If you had sent that over to the Puppetmaster, it would have failed compilation and blown up your environment. You got that instead on your desktop, in your coding, and as you were making your commit.
Ben Ford [00:24:52] You're saying that as you're editing the Puppet file, it will actually go look up the things that you're pointing to, and tell you if you're pointing at the latest version, like call out to the Forge or to GitHub or something.
James Pogran [00:25:03] Yeah, right now it only calls up to the Forge because we have the API access, but we're working on other methods for validating versions. But core of that is, is you're right. It's going to, as you type, determine is that the latest version for you? And then it will resolve the dependencies and make sure that they'll actually compile correctly.
Ben Ford [00:25:23] So if a module, like if a module has dependencies that are not listed in your Puppet file, it'll warn you and say, hey, you also need to include this in your Puppet file.
James Pogran [00:25:31] Correct.
Ben Ford [00:25:32] That's huge.
James Pogran [00:25:33] Yep. We shipped it in the last release, but we've been making sure that it's stable enough to be completely talked about and we're at that point now. So much so that it's a gem. So it's usable by processes outside of VS code now. So some of the Vox Pupuli members are working on trying to integrate that into their release processes.
Ben Ford [00:25:55] That's pretty cool. And that seems like something that we should investigate using in our 10K itself so that the deploy's a little bit less fragile.
James Pogran [00:26:02] I believe the people responsible for that and Bolt project are also looking at ways of using it too.
Ben Ford [00:26:10] I love hearing about this kind of synergy of different groups working together and kind of helping each other along with with their different goals.
James Pogran [00:26:16] Same here.
Ben Ford [00:26:17] Well, this is really exciting. I'm really enjoying watching the developer ecosystem grow and seeing all the tooling get easier and more intuitive to use. This would have really, really helped me when I was first starting to learn Puppet code, because just like you, I wrote code with a VM window and a Docks window and a terminal window to try things out, and having code that would just auto complete would have been like a difference between night and day. I'm really excited about this ability to kind of like lower the on-board friction for new users, and just all the way up to making it faster for the more experienced users like streamlining the development workflow. So I'm really excited about what's going on. I'm really excited to see the new work and like the different Puppet file resolver and the inference you're talking about.
James Pogran [00:27:03] Same here. We're excited to work on this stuff.
Ben Ford [00:27:06] Well, that's great. Just before we close up, I want to make sure that everybody knows how to get in touch with you or to find the extension. Could you maybe tell us what your website is or Twitter handle or anything else you want to cover?
James Pogran [00:27:17] Sure. From a Web browser, you can go to a short link. So that's pup.pt/vscode. That'll bring you to the extension gallery page for our extension. There is also the documentation website that we have that's puppet-vscode.github.io. And that'll show you all the information that we have for each of the features that we have. And my Twitter handle is @ender2025. And I'm also pretty active in our Slack community over there talking office hours and Puppet dev chats.
Ben Ford [00:27:54] All right. Well, thanks for sharing with us, James. This is real exciting to see this grow.
James Pogran [00:27:58] It was a pleasure being here. Thank you for having me on.