In this episode of the Puppet Podcast, we talk with Corey Osman about Puppet Debugger, the command line REPL debugging tool he built.
We're kicking off 2019 with a bang and a series of spotlights on notable community members and their achievements. We've got some great shows lined up for you, from useful development tools, to community experiences, to lessons we could learn from the history of computing. Let's get started today with Corey Osman of Portland, Oregon who's built a command line REPL.
This quick screencast shows some of the capabilities of the tool:
A REPL, or command-line shell, allows you to interactively work within Puppet and see what effect each line of code you write will have. Rather than trying to understand an entire system at once, you can drop into the shell and try out a language construct or see how a variable will be interpolated. For even greater understanding, you can add breakpoints to your Puppet code so you can stop compilation at a known spot and inspect your variables or see what classes have been declared.
Today's musical intro was recorded by Michelle on our SRE team and her band, Great Niece.
Learn more
• https://www.puppet-debugger.com
• https://github.com/nwops/puppet-debugger
• http://logicminds.github.io/categories/debugger
*Ben Ford is a developer advocate at Puppet.*
Ben: [00:00:10] Hi everybody. My name is Ben Ford. I'm the developer advocate here at Puppet. I'm here today with Corey Osman. Could you tell us a little bit about yourself?
Corey: [00:00:18] Hi. So I am a DevOps consultant. I've been working in the Puppet space for, man, just over I would say nine years now and I write tools for the DevOps community.
Ben: [00:00:35] That's pretty awesome. That's a very long time. We're here to talk about Corey's Puppet Debugger project. Could you tell us a little bit about what that means?
Corey: [00:00:41] So the Puppet Debugger project, what it initially started out was just a way to run Puppet code without having to write a manifest and it's a Ruby-based tool that you install as a gem and it is designed to help the user learn and understand the Puppet language.
Ben: [00:01:04] And how does it do that? Do you, can you work with the language interactively or...?
Corey: [00:01:10] You know first back up a second and we can talk about what a REPL is because underneath the covers the Puppet Debugger is a REPL in every language that I can think of except for maybe one or two has a REPL and REPL stands for read evaluate print loop. And if you just break those down first it's going to read code, then it's going to evaluate the code, then it's going to print the output, and then it's going to loop back around and do the same thing over and over again. So if we take those foundations and we bring that into a tool, you have a way into the Puppet compiler so that you can write code and then it evaluates it and prints it back out and shows you wat would become about if it was actually applied to a node. The debugger portion of that you know the debugger in general is just a tool that helps you find the problems in the code base itself, so in a lot of this is you know logic errors and if you're writing unit tests with rspec-puppet then that would also help. But there's quite a bit of setup involved in writing tests so this is a tool that will at least help you understand your code and help you find problems much quicker than than doing all of the unit tests set up.
Ben: [00:02:43] So a REPL is kind of a, I don't know, maybe an esoteric name for something that's really really common. I think the one thing that we all know that we've all used is just running bash interactively. Like when you open up a command prompt and you start typing commands into bash that's a REPL right there. So what you're what you're saying what you've built for us is basically the way to interact with Puppet code more or less the same way you would interact with a with a bash shell.
Corey: [00:03:10] Correct, yeah. So you know in bash you're typing something in and if you mistype it tells you that you mistyped it. And that's exactly what the debugger does is if you tried to declare a variable twice, well you can't do that in Puppet and it's going to tell you what you did wrong. But the cool thing is is that it doesn't take 10 or 20 seconds to compile that code and spit out the problem. It's immediate.
Ben: [00:03:39] Could you tell us just like a little bit of like how it works internally? Like how were you able to compile code using Puppet?
Corey: [00:03:46] Underneath the covers it takes your input and we'll just say from a while loop what it's doing is it's setting up the environment. It knows where your Puppet modules are. It grabs the facts from the node that you're running Puppet apply on and it brings all of that together for the parser to eval it. So one of the differentiating factors between say puppet apply and running the puppet debugger is that I just provide like a common set of facts for you so that you don't have to pull them. But I also give you the option to get real facts from your system as well.
Ben: [00:04:30] Could you tell us a little bit about like how this came to be? Like is there like an inception story of where this tool came from, why you built it?
Corey: [00:04:40] Once I understood what was going on in the covers it just sort of came to me and I was actually eating breakfast at a place right up the street from me. And as I was like oh my god like we have not had a REPL in the public community for, well, ever. And I think I just discovered a way to how to do this. And so I think that day I had like a small proof of concept. I mean it wasn't great but it actually read the code that I entered in and spit out the same thing that puppet apply would do, except it did it immediately.
Ben: [00:05:22] So you're saying you can stop in the middle of a catalog compilation and you can actually print out the value of a variable.
Corey: [00:05:30] Yeah that's been something that has been on my want to do list for some time. I think a lot of problems with unit testing is we all want to try to test the intermediate state and to this day you can only test the the end state like what what resources have have been compiled in the catalog and whether or not those things are in the catalog. With the debugger you get the end state but you get everything in between all the intermediate states all of the things inside ephemeral loops. So if you want to know the value of the sixth iteration of an array you can do that. And when I finally discovered why I need a puppet functions I was like oh because I need to manipulate this incoming data. But now it's like OK well there's over 100 functions available to me. What do they do? And there isn't really a good way to do it. So the Puppet Debugger it allows you to use the function and if it's using facts or you have to supply data to it, it allows you to run those functions over and over again and really understand what they're doing.
Ben: [00:06:59] Just kind of interactively play with them a little bit rather than having to write code and run the code. You can just try it and see what happens.
Corey: [00:07:06] Facter DB is another tool that is essentially a database of a pre-canned facts.
Ben: [00:07:14] So what you're saying is you can compile and poke at this code as if it were a CentOS system or as if it were an Ubuntu system or anything like that. Even different versions like CentOS 6 or CentOS 7.
Corey: [00:07:26] Even Windows.
Ben: [00:07:27] Even Windows. So can you do things like run hybrid data lookups or testing functions or maybe like iteration or anything can you look like validate this code you're writing while you're writing it.
Corey: [00:07:41] Absolutely. You know I used to think Hiera was just magic and then I realized it was just a core function that was included with Puppet and I'm like "He,y I can just call the Hiera function and pass in the values and I can have it look up values within my Hiera database. So if you understand that then now you can set up a whole Hiera lookup all the data files and now you can start to test out your Hiera restructure to see if that works correctly and you can swap Facter sets in and out to to validate that your Hiera setup is now so.
Ben: [00:08:27] So can you change facts while you're running? So you can you can test a look up against Red Hat and test that same look up immediately against Debian?
Corey: [00:08:35] Yes you can. There's a reset command that you can run that basically wipes out the environment and you can reset the the Facter query that selects effects.
Ben: [00:08:48] That's even faster. Wow, that's cool. One thing that I know that some people have struggled with quite a bit is understanding Puppet's data type system it can be it's relatively straightforward but it can be pretty complex if you have like different validations or if you have complex types. Would this be useful in trying to validate making sure that you are using the data type properly?
Corey: [00:09:14] Yeah, that is one of my use cases as well. And I have a whole, I think I wrote a whole blog article on validating your data types.
Corey: [00:09:24] There is a couple of things that you can do and you might never discover these if you if you're not using a debugger tool where you can interactively play with the code, but there's a function I think it's called type of - basically there's an operator that I want to say the bacon canon. But that's not the operator name. Yeah it's the equal tilde tilde equal. And that operator will tell you true or false if the data type matches on both sides.
Ben: [00:10:00] OK. So what's your favorite feature that this thing offers?
Corey: [00:10:03] By far my favorite feature in the Puppet Debugger is the ability to set a breakpoint anywhere in your code. This includes Puppet functions. This includes anywhere in the manifests and you can set a breakpoint and the breakpoint is really just another Puppet function and it's just debug colon colon break and when Puppet compiles that catalog with whatever means that it normally does through puppet apply or maybe you're running a unit test or maybe you're just feeding the Puppet Debugger a file through standard in it's going to run that brake function and just stop right there, right in the middle of the compiler compiling the code, and that allows you to see everything that the compiler can see so that allows you to inspect the ephemeral loops the values that are around you at the time the top scope variables the facts that are fed in the output of functions that will run either before or after that break command.
Ben: [00:11:21] So at that point all you really had to do is put in a breakpoint right after the look up call and then see that the variable that should have been set by the lookup didn't have anything set and then the rest was just a little bit of playing with the values and the variables and running the look up command before you noticed that what you were passing in was the wrong case.
Corey: [00:11:43] Yeah correct. So it made it just really simple to determine what my fault was. And after that I just remove the break command and off I went. I finished the code I was working on. So that that is my favorite feature and I'm going to say I totally ripped it off Pry. Pry is an amazing tool for Ruby debugging and I just wanted that same feature in Puppet code because it is such a valuable thing to have to be able to stop inside the code. And I did whatever I could to make it work. And I believe that I did I even inspired heavily from Pry. So if you've never used Pry for really debugging, go check it out because it's awesome.
Ben: [00:12:38] Now I think maybe my last question for you is, it would be really really useful if I were able to impersonate a specific node while it was validating code so can I like load up a set of facts for that specific machine in my environment?
Corey: [00:12:55] So there's two things you can do. What I've been meaning to write this Facter DB article for a couple of weeks now but if I wanted to spoof a system I could capture the Facter set of a real system and use that as a external Facter set for Facter DB. And because the Puppet Debugger uses Facter DB I can now load that externalize Facter set of some of that real node in the real data that it has inside the Puppet Debugger and get to the bottom of why my code does not work the way that I think it should.
Ben: [00:13:40] So how would somebody get started using your project? What's the first steps that they ought to do in order to try it out?
Corey: [00:13:47] So if you just want to try it out, I have a website: www.puppet-debugger.com. There is a demo there and essentially it's a live application where you can just you know test the waters out and see see if you want to download it and try it on your system. But it's an interactive it's real it's running on Heroku it opens up a shell to a remote system using web sockets and it looks like a real system and it loads up the debugger.
Ben: [00:14:25] And then installing it is just as easy as gem install and running Puppet Debugger?
Corey: [00:14:30] Right. So to to install it, it's just gem install puppet dash debugger. It does require you to have Facter and Puppet but I think Puppet is a dependency so she'll come along for the ride. If you're using PDK I think that you should I think it I'm not sure if it ships with it. I think at some point I was trying to get it in there but it may already come with it, it may not. But if you know how to use the gem command you could install it in the PDK environment and then you could install it on a remote node if you want to. I've used it that way before you can install it on your laptop. Yeah. So there's there's multiple ways to use it.
Ben: [00:15:25] Right on. Well, thanks for talking with us Corey and for sharing this project with the community. I think it can really help people out when they're trying to learn how Puppet parses their code base and figures out what to do. Thanks for your time today.