George Pandzik (@DevOpsFables) talks about the patterns from yesterday that we should continue applying towards the new ways of working.
It seems like every day is another paradigm shift anymore. SaaS, PaaS, configuration management, cloud native, immutable infrastructure, serverless, and so on. But one has to ask, what do we lose by always chasing the bleeding edge? Information hiding, API contracts, and other ways of abstracting away implementation details are all part of solid system architecture. Are there patterns from yesterday that we should continue applying towards the new ways of working today and tomorrow?
George Pandzik (@DevOpsFables) thinks so, and he's here to talk about patterns that he doesn't think should be lost to the sands of time.
*Ben Ford is a developer advocate at Puppet.*
Ben: [00:00:08] Hi everybody. My name is Ben Ford. I'm the developer advocate here at puppet. I'm here with George Pandzik. He's in the envious position of having formal computer science training and also many years of kind of boots on the ground sort of ops experience. And he's here to talk to us about how those two things relate to one another. Could you introduce yourself and tell us a little bit about what you will be talking about?
George: [00:00:33] Sure. My name is George Pandzik. I have a Masters of Science in Computer Science with a focus in software engineering and also over a decade's experience in everything from performance analysis to handing out flashlights instead of fixing light bulbs. I've worked on z/OS mainframes, Windows systems, Linux systems and automating the same so I've seen a lot of things. Kind of what we're going to talk about today is a topic that's near and dear to my heart is I had a lot of good mentors when I was new who had been around since the '70s back in the days when they were still using punch cards. So I learned that there's a lot that we can pick up from people who've been there and done that before us. That's a big thing. We should ask questions like what worked for them and why did that work. Because if we get to the why we can look for patterns that we can incorporate today.
Ben: [00:01:38] Right. If we look at how past people solved problems when we see problems that are similar and maybe not the specifics of the same thing that they're working on but the same kind of patterns then we can learn from how they solved those problems and we can move forward with our own solutions that you know take from that. So how how do you think that that's something that will affect the way that people write their Puppet code today?
George: [00:02:02] Well so it's really easy to naively just dump thousands and thousands of lines into a site.pp file. That's really hard to maintain because you can make one change anywhere you might have to make a change everywhere. So Dr. Parnas back in 1972 wrote a paper on data encapsulation and information hiding that I think we can learn from and incorporate into our Puppet code.
Ben: [00:02:33] So this is something that's been around and it's been evolving for years and years and years. Could you tell us a little bit about that story.
George: [00:02:40] Sure. A big thing to remember is none of this is new. I mean Parnas wrote his paper 46 years ago. He used an example of what he called the quick algorithm for a keyword in context. Just at a high level. If you want the gory details you can go read the paper from the ACM. But it would take a sentence, print out the sentence in full, and then it would rotate the first word to the end, shift everything and then print out that next sentence. Shift print shift print shift print.
Ben: [00:03:18] Pretty simple algorithm. I'm sure it was a little bit more complex without all of the libraries we have today.
George: [00:03:23] Well there were no libraries back in the day he wrote pseudocode which smacks very strongly of like PL1 or assembler so he actually shows three different implementations of that program including a naive like top down writing nested loops to do the parsing of each sentence, figuring out what is space, what's a comma.
Ben: [00:03:48] So like the the spaghetti code in `site.pp` sort of example.
George: [00:03:51] Exactly. And he showed why that really doesn't work. He showed kind of intermediate solution but the final solution he presented he called encapsulation where you basically if you think about this program there's really three parts of it. There's the parser, there's the rotator, and there's the printer so in the naive solution those were all just kind of jumbled together and you're doing printing as you go. So any one part of the program could wind up messing with the output stream, which could wind up breaking the whole thing. And remember this was '72, so we're talking about entire address spaces. So what he showed was encapsulating the parser, which would only have access to the input stream, the rotator that would just take a intermediate sentence, some string of bytes, figure out what a word boundary was and manipulate the string. And then you had an output printer. That was the one thing that could that could touch output.
Ben: [00:05:06] So these are things that you might call like functions today or even maybe component modules.
George: [00:05:12] Right. And that's what's really beautiful about this concept is this is what gave rise to objects and methods. The interface concept which became contracts. These are all very similar ideas which come from pretty much the same place. The good interfaces make good code. The idea is you take all the parts that could change. So in "Quick's" example if we wanted to change what a word boundary was if you go with a naive solution and just go by spaces, commas, periods are themselves words and so you could wind up with a period beginning of a sentence. Well that's not useful.
Ben: [00:06:01] Right and so you'd want to be able to to iterate and fix what these things are without breaking other parts of the code.
George: [00:06:07] Right. And the output function if you will shouldn't have to worry about that. So what is important to get with encapsulation is it allows you to just focus on the couple of things that you need to worry about. I call it ignoring things for fun and profit. It lets you be good at one thing.
Ben: [00:06:33] Or like they say you know that good fences make good neighbors.
George: [00:06:36] Yeah exactly. And this is why I like metaphors because it's the same concept but verbal. We reuse stuff. So the other thing that encapsulation does for you is you stop thinking about the implementation. How does this thing that I'm trying to get done work and you just focus on what effect do we want to have. So just as with the function, it's an interface. If I feed it a certain set of parameters I should expect certain behaviors. And it's a black box. I don't know what kind of internal variables it uses. I don't know what syscalls it makes and I don't have to care. It can change them at whim.
Ben: [00:07:24] Coincidentally that's a lot about how like Puppet itself works. You declare a resource you don't have to care about `File.open`, `File.write` and all of that. It does it for you.
George: [00:07:33] Exactly. The way it works with Puppet is what Adrien Thebo called in SysAdvent "Configuration is Legos", which kind of led into Roles and Profiles. The idea is that if I'm defining a server as a role they should never have anything but include statements period. If you're doing that you're getting too deep in the weeds. Roles include profiles. Profiles should just define the module resources that they need from a purist standpoint. If you have a `file` or an `exec` in a profile you're doing it wrong.
Ben: [00:08:22] Definitely.
George: [00:08:23] I mean that's academically pure so we'll have deadlines I understand. But the modules then do all the nitty gritty of defining users, defining files, `exec`-ing different install commands, dealing with yum repos and all that stuff. The beautiful thing about that model is if - let's say I'm the manager and for whatever silly reason I am dealing with roles. Managers shouldn't always have their hands in code. But for those of us who do.
Ben: [00:08:55] It happens.
George: [00:08:56] Yeah. They should be able to find out, based on what profiles are included, this is what this role server will do.
Ben: [00:09:05] It's almost self documenting in a sense.
George: [00:09:07] Right and you don't have to discern that from all of the files and exec statements and all that stuff that gets hidden by the profiles and by the modules. It puts it right there in front of you. It's as you said, self documenting.
Ben: [00:09:27] Right, so that kind of gets people away from you know bike shedding about the implementation details.
George: [00:09:32] Exactly. And that really lets you free up your mind and your concentration to do the one thing you're supposed to do well. Like a good Unix programmer. So since you can not have to worry about the implementation you can focus on the effect of what you're trying to do. And focus on the desired outcome that lets you actually write tests and mocks. Because if I don't care what the result of your function is, I can pretend it can be whatever I want to see. This lets you write good tests. It requires that you put trust into other parts of the program. If you don't trust, you might as well write it yourself. And We're all people, we all need to trust each other, whether we're managers or the professionals.
Ben: [00:10:20] Have you read Martin Fowler?
George: [00:10:22] Yes, I will say I've been influenced quite heavily by some of his refactoring books. I highly recommend them in whatever language you want to go look them up. He gives examples of how to, for instance, extract method which is it takes a common chunk of code, you wrap some tests around it and then you actually pull it out and put it into a separate definition. It's a really good practice for simplifying and refactoring your code.
Ben: [00:10:52] What do you think you would say is kind of like the moral of the story here.
George: [00:10:58] So, I guess my DevOps fables moral for the day is don't ask questions you're not prepared to have answered and trust those who are doing work that you don't want to do yourself. We have all these complex systems that when you build up all this trust, none of us have time to re-implement all of standard C every time we read a program.
Ben: [00:11:23] Or even just `stdlib`.
George: [00:11:24] Right. Or rewrite our operating system to schedule processors. We have to trust that those things are done.
Ben: [00:11:34] For sure. You'd totally want to group everything together, group the modules you write into the component modules and write profiles of group those together. Nobody wants to write spaghetti code - that's nothing to be proud of. So what would you do - like what advice would you give Puppet module authors? How would you apply this directly to how people are working?
George: [00:11:54] So this is where we get the benefits of Adrian Thebo's Lego concept and Roles and Profiles is really harden those boundaries between things. If you are writing a role, only write the role. If you're writing a profile, only write the profile.
Ben: [00:12:12] Like would you would you suggest like totally separating yourselves away from that? So you you're not even involved with the implementation of that?
George: [00:12:19] In an ideal sense, yeah. I mean from if I were just spit balling an org chart the role authors would be more of your business analysts who are talking the business and figuring out what do we need to meet the need - your solutions architects, that nice high level where nobody actually does any work. I saw that as a solutions architect. Your profiles would be kind of your mid-level folks who are actually taking - OK, you want me to set up Apache or nginx or SQL Server. And here are the things I need to do to make those things functional. Finally you would have your most senior people, your real more locked down in the depths, writing the guts the modules to actually do the thing.
Ben: [00:13:13] But now in a kind of a more real world implementation where you don't have that luxury of being separated, where are the people writing the profiles often are also writing component modules. How do you keep yourself like in that mindset of right now I'm writing a profile, tomorrow I'll write a component module or the other way around?
George: [00:13:34] That's a really good question. It takes some self-discipline and a whole lot of self-deception. And when I figured it out I'll let you know.
Ben: [00:13:48] We'll have you back to to talk about that next time then.
George: [00:13:50] Sure. Kind of a real way to implement some of this is like I said harden the boundary around what a thing does. Define your interfaces and document. These are the methods that I expose. These are the options that I can accept, and if anybody tries to get cute with that throw an error. And it goes the other way too because if you have behavior that is either undocumented or the documentation says if you call me you call it this way it'll do this and it doesn't, or does something else entirely. That's a bug. I take a little broader definition of bug. I know it's still controversial, but any undocumented behavior or behavior that is different from the documentation is a bug.
Ben: [00:14:44] Do you use Puppet strings to make this documentation or are you talking about like writing manual user docs for it?
George: [00:14:50] That is really up to however you want to implement it. I'm going to use the old academic defense and say this is just the idea. Implementation is something else. Another really good way of kind of carving these things up is - I'm going to take a page from 1980s Lean here - is map your value flow. Identify each section of your process and group the things that affect or change the same things. Those become a capsule, a module. Now it's going to require you to be able to understand what a business process is and how to express that in code. None of this is new. This is 1980s Lean on top of 1980s ITIL.
Ben: [00:15:44] It's a new way of looking at the same old lessons that we've been learning.
George: [00:15:48] Exactly. And we're keeping the stuff that works. Adjusting it where it doesn't quite fit and we're throwing away the things that don't.
Ben: [00:15:55] Right on. Do you have like any closing thoughts or anything here?
George: [00:15:58] Yes and this is a big one. I encourage everybody to learn from the past. Nothing is ever new. Go find the gray hair in your office who has been doing this for 40 years. Talk to him about the good old days and try to understand why what they did back then worked, and what you can learn to do today.
Ben: [00:16:22] Someday we're all gonna be the greybeards.
George: [00:16:25] It happens. I'm going there myself.
Ben: [00:16:28] That sounds like a very good lesson to take from this. Thanks for sharing George.
George: [00:16:31] Yep and always remember if it doesn't work right, fix it.