A couple weeks ago at Networking Field Day 9, Brocade presented with their usual A-list of networking gurus. One of the presenters was Jon Hudson, a very engaging, visionary speaker. His talk, shown below, was about the state of network programmability.
During the conversation (which is well worth watching), discussion turned to the question of “will network engineers become programmers?” posed by John Herbert of MovingPackets.net. Jon Hudson’s response elicited applause from the room. He said:
“The trouble I have with that statement is, most network engineers I know, like myself, we know how how to code. We went to school for it, and we chose not to.” – Jon Hudson
The conversation went on to discuss the value of programmability for the sake of consistency in the management and configuration of large-scale network fabrics (which I don’t think anyone would really debate as a “Good Thing”), but Jon’s quote about being a programmer and some of the sidebar that flowed from it created a fair bit of activity in the Twitter stream. Following the presentation, my attention was called to a mailing list on which a question was asked about networking engineerings being “given a pass” on the need to “code” while other IT disciplines have been forced to embrace it. The remainder of this post is a slightly-edited (for context) copy of my reply with my own thoughts on the matter:
Jon’s quote resonated with me because, well, it is me. I have my CompSci degree from a (relatively) prestigious
engineering university, but along my journey to earn it, I discovered the distinction between what I consider “tools” or “scripts” and “software.” Scripts were the little programs I wrote for the first few years of my training. “Calculate the leap years for the next 3 centuries”, or “take this file, iterate through the entries on each line and munge them in some way.” That is fun, useful, mentally stimulating, and easy to replicate. They solve a specific problem and can easily be replicated and tweaked to solve a slightly different problem. Most of those programs are also fragile, inefficient, probably insecure, etc. My experience with writing real software was of the “Your function will be given as input a pointer to a vector containing values which must be validated against the established data dictionary. Any exceptions thrown during execution of your routine must be handled. Your routine must run in O(n*log(n)) and produce as output a pointer which shall be passed back to the calling function which references a new vector containing the calculated values available only to the namespace of the calling function. Document your code thoroughly and demonstrate that your solution was accomplished with the minimal number of memory allocation calls” variety. That’s about when my interest in “developing software” died.
The long and short is that I don’t think networking admins should be given a pass from having to work efficiently and build tools if necessary. However, I consider scripting and tool building to be very different from serious “coding” or “being a programmer.” And that, I think, is a point on which some people get lost. I realize that some understand that distinction, but I think a lot of the “Code or Die!” rhetoric is less specific about it. I also think a vocal group of people (majority? minority? I really don’t know) are trying to cram the concept that “you *must* be writing code if you want to have a future” down our throats. I disagree. We *must* work efficiently and with high quality/low error rates. If that is best done through building some tools to augment your toolbox — great! If you work in a cloud-scale environment or somewhere that has embraced DevOps automation to near-complete levels maybe that does mean everything has to be programmatic. But, you know, sometimes it’s just easier to go tweak the damned firewall rule yourself. And sometimes it’s faster to do that once a week than it is to write a script that does it for you which you have to conceive, POC, develop, debug, and then, most significantly, maintain.
source – xkcd.com
Also, as a consultant, I have seen the effects of systems automation taken to bad extremes. Wizards building intricate clockwork that nobody in their own organization understands, which is quite unfortunate when the wizard finally gets bored and resigns. Then, no one understands how the company’s custom network access control system based on identifying OUIs of MAC addresses specified in DHCP requests actually works. This happens because this stuff gets written like tools. Quick, dirty, often not well (or at all) documented. Those solutions differ significantly from production-level code which has to account for exception handling, input validation, secure coding practices, memory efficiency and garbage collection, QA, regression testing against test plans, controlled release cycles etc., etc.
source – xkcd.com
I’ve talked with some of my close college friends who earned their CompSci and CompSystems degrees right along-side me who went on to build careers now with 15+ years of experience in software engineering. I’ve told them of how the buzz is that “everything must be codified” and all IT people will have to become “programmers” or “developers.” They have a hearty laugh over that every time.
To be sure, in this new world the idea of the NetDev does exist. But honestly I think there are some folks in the Twittersphere who aren’t just embracing “code or die” as an IT admin, but are instead making a career transition to part- or full-time tool builder or even software developer and are confusing that move with being a network or systems admin who must, by definition, “automate all the things” using Python. By all means, build tools when the tools help you get something done faster, better, cheaper. And demand of everyone on the team that things get done better, faster, cheaper. But I don’t believe in “coding” or “developing” as an IT religion. If you need a saw, just use a saw. If you need a jig to saw better, build the jig. Be careful about becoming a jig-builder who forgets how and why to use a saw, unless you are looking for a career change.
I believe the tool-building will happen and has always happened in IT, but will become more prevalent in networking gear as the capability becomes more widespread. I expect development of SDN apps will be done by software development companies, whose product is SDN apps, and will be written by software engineers who know how to properly write applications. Or by open-source developers who are usually still “real” software engineers by trade. The rest of us practicing IT folks should be expected to work efficiently and accurately. Sometimes that suggests or dictates building a tool, and sometimes it just doesn’t.
That’s my two cents. It’s probably wrong.
I’m interested in your thoughts on the matter. Leave them in the comments, even if it’s to remind me how wrong I am.
Brocade was a sponsor of Networking Field Day 9, and as such indirectly helped to pay expenses associated with my attendance. At no time did they ask for, nor where they promised any kind of consideration in the writing of this post. The opinions and analysis provided within are my own and any errors or omissions are mine and mine alone.
Like this:
Like Loading...
Related
Herding Packets 
[…] Thoughts on Building Tools versus “Programming” […]
Bob – I don’t think you are wrong at all. I have been doing automation (server and network) for the better part of 20 years, both as the end-user and the vendor, and I think you are pretty much spot on.
As an end user I would write shell, tcl, perl, python or whatever I needed to work more efficiently. Sometimes the “tool” would be used by others to accomplish the same or similar task. I didn’t do it because I was ordered to, I did it because it was the right thing to do. There were times when I had to do support for something that I never really intended and it was a drain.
As a vendor, we tried to make software and/or frameworks that made it easier for engineers but in reality the software was built more towards solving a business problem and it just became too abstracted for engineers to use. Not that it wasn’t usable, but if you try to abstract away complexity you sometimes (most of the time) also wind up losing some of they dynamic of the direct system interaction and not everything translates into the abstraction.
So then you are stuck with the conundrum of trying to adhere to the business practice(s) of using the “tool” vs. waiting on the vendor to fix whatever the problem was/is. You usually just wind up circumventing the application so you can get your job done.
This is going to be a tough road.
I think you’re elevating programming too far from tool building. I also think that the real shift of all the “IT has to code” commentary is about just that tool building. We non-coders have to start picking up on some of the best practices of the coders because while that jig was a one off assistive tool, this thing I threw together over here is now critical to production staying up. And needs to be treated as production grade.
Great insight. I think we’re very close to agreeing. I guess my point was that there’s a distinction between the tool building and “becoming software developers” as part of our IT jobs. Thanks for reading, and taking the time to comment!
Some of this argument is truly just in the semantics. However I look at it like this: I wouldn’t hire a programmer to be my network engineer, just as I wouldn’t hire a network guy to develop my next code release of project X. Just because my app dev guy can install windows server and sql and get them running doesn’t mean he’s a sys admin – he’s probably forgotten twenty things important to actually running those systems long term – but just because I can create a “program” to automate ten drudgery tasks that need to be done regularly and with precision doesn’t mean I’m a programmer either.
If you are good system or network admin, you know how to do at least *some* level of coding, whether its Python, or just Powershell, you’ll be far better off knowing how to use those tools. And you ought to understand what your development teams do in terms of project lifecycles, etc….you’ll be better for it.
As an aside….I’ve always said the reverse corollary is true as well – if you’re a good programmer, *truly* knowing how the systems/networks function will make you a better programmer. It drives me nuts when my web site development team asks for DNS entries without having a clue how they work.
To put it in a different light….I’m a sysadmin. I install SQL server. When things break I often do some of the things a DBA would do – performance troubleshooting, creating/running a T-SQL script to fix something, or querying the database to “find all users that have XYZ fields mismatching between these two data sets.” But I wouldn’t come close to quantifying myself as a SQL programmer.
Your post makes other good points that aren’t solely in the domain of a network engineer. How many projects at company X have been implemented like your last xkcd image?
Hey Mike,
We definitely agree, and it’s absolutely a matter of semantics for the most part. The overarching point I tried to convey in the post was that tools and good, and building tools is necessary (especially these days), but I consider the fear mongering that net/sys admins will need to become “developers” in the world of Software-Defined Everything to be just good ol’ FUD and a few folks who are feeling high-and-mighty that they do it all in [language of choice].
Great comments, and I appreciate you taking the time to read and write. Hope all is well.
Bob