It seems like sooner or later everyone ends up writing about interviewing. “My favourite questions” are one of those topics that pop up again and again, but I am so often bemused by what those questions are.
Take a recent StackOverflow query; “What are your favourite questions for a senior Unix admin?”. The responses elicited were almost entirely of the “swallow the man pages” or “trick questions I know” or worse yet “simple stuff I can Google” variety.
Folks, whether or not I remember how to use ldd or strace, or remember that it’s truss on Solaris, not strace, does not make me a senior SA, and if you think it makes you one, I pray for the sake of your ego I never interview you. Those are questions I’d expect that a junior would either know already, or be able to find given a few minutes with the search engine of their choice. Even in the dim dark days before ubiquitous workplace Internet access and decent search engines, one could fairly quickly find the answers with a wall of documentation or, if all else fails, telling a colleague you’ve forgotten and letting them tell you.
What does a senior SA look like to me? A senior SA is someone who solves problems. S/he understands that the role of a senior is to be able to understand the relationship between complex application layers and the systems s/he’s responsible for, so that when someone rocks up to them and says, “This critical application that makes us lots of money isn’t working, and it appears to be a slowdown with this other application it depends on running slow, but we looked at the virtualized server and we can’t work out why it’s going slow”, s/he can get on with pinpointing the fault, not raise their hands and refuse to look at it until the app and network guys prove it isn’t their problem. That’s why I ask questions that are app-centric as well as server-centric. Can a candidate trace down the stack from high-level symptoms to nail the problem? Protip: if your answer at this stage is “not my problem”, we’re done.
A senior SA can sit down with a problem they’ve never encountered before, and even if they can’t solve it, they can demonstrate that they understand the principles of problem-solving, that they know where to look, that they’ll look to the most likely indicators based on the evidence they’ve unearthed. They’ll follow a logical train of thought, and they’ll investigate things in a controlled fashion. And if all else fails, they’ll call for help and beat their head against the problem until help solves it, or they do. That’s why I ask questions that start with a simple problem statement from real-world issues that have paged me at 2 am, and see if the person in front of me attacks it the right way. The result isn’t really relevant—they don’t know my environment in detail, after all—but the methodology is priceless.
(You get bonus points if you mention, “I’d check the change logs to see what changed today” as a starting point.)
Problem solving isn’t just at the break fix level. A senior SA should be able to plan, design, think ahead, solve the bigger problems. I want to know how you work out your patching strategy, how you balance critical security fixes with regular cyclical upgrades and the desire for stable systems, how you’re going to work in with project deliverables. I want to hear your views on the best way to manage authentication. Tell me how you like to automate server deployments, how you deal with guest sprawl in virtualized environments. What information is key to your understanding of capacity planning. And virtualisation? How do you feel about it, anyway? What problems does it solve, what does it create? I don’t care if you don’t know all the answers to these questions, hell, I don’t. Can you ask me good questions when you don’t know the anwers? Can you, in short, talk to me about them intelligently?
I do not fucking care if you remember how to delete a file called ‘-rf’ from the root filesystem off the top of your head.