Provide the candidate with a genuine programming test in the interview - a laptop hooked up to a projector with a broken project, with a mixture of basic to complex bugs, and ask them to add some functionality, again ranging from basic to moderately complex. This is something nobody can really study for and will actually provide a showcase of their skills. Have technical people there talking to the candidate to get a feel for their personality and how they approach the problems faced.

For the job interview for my current role, I had spent 2 weeks studying programming principles, design patterns, database stuff etc. I was able to answer all the theoretical questions well and think I came across well

I was then handed a laptop which was connected up to the projector and shown a solution which contained a broken website with a defined number of errors which I had to debug and add some simple functionality. While I wasn't very familiar with the front-end framework used by the company, I was able to fix about half the bugs and then give my best guess as to what the causes were for the remaining bugs based on what I could see, and this was good enough to get me in the door.

The whole thing, between the theoretical questions and the website debugging session lasted about 2 hours.

once you actually get the job, you'll most likely never run into any of those problems from the job interview book again

If their job doesn't involve solving those kinds of problems, why test them on them in the first place?

Why not give them a real problem they would encounter on the job instead?

That's my recommendation. Surely you will have many examples of real on-the-job problems to choose from to find a suitable one.

About a year ago I started to have discussions with candidates. There are many topics that are controversial in the industry and which do not have only one correct answer. Answers to questions about these topics usually start with "It depends…" – for example:

• What framework is the best?


• What database do you prefer and why?


• Is a certain design pattern actually useful?


• Focus on new features or fix bugs first?

First I ask the candidates about their opinion. Once they got their point across I simply pick the opposite and argue against it. Or I give an example in which their choice would not be a good fit answer and ask them if and how they want to change their answer with this new example/requirement? The following discussion will tell me a lot.

• Did they only memorize some facts for the interview or do they actually have experience with this area and good examples to prove their point? Actually, I was very surprised by how often their argument is: because I always did it that way
  


• How well are they able to bring their arguments and the complex technical details across? Are they good in teaching?


• How do they handle resistance? Do they try to convince, do they try to please or do they start to be aggressive or arrogant? Are they open to learning?

I am aware that this might be very stressful for the candidate. Therefore the has to be done very carefully.

• Have multiple interviewers. Your strongest engineers on the team should participate in the interview process - they definitely want to work with people better than the ones that are struggling, right? Let other engineers conduct their own 1 on 1 interview. And have another engineer do the initial phone screens.




• Ask an open ended design question. for a problem of medium complexity. Ask the candidate to write out the header file, class declarations, box diagrams, etc... Does the candidate ask clarifying questions? After he's done with a basic design, introduce a new requirement that would break his design. Is he able to pivot? Pick one part of his design and probe deeper.



• 
  

  Ask a question that probes if they have a deep understanding of the platform fundamentals. Some examples I've asked:

  
  
  
  
  
  • "How does the C++ compiler implement virtual methods?"
  
  
  • "How do closures work in JavaScript?"
  
  
  • "Explain when you would ever want to use UDP instead of TCP."
  
  
  • "After you type www.foo.com into a browser address bar, tell me about all the network activity and protocols involved to get the content on the screen."
  
  
  
  



• Probe for genuine interest in programming. This applies more for university candidates and junior engineers and less for experienced hires. An example questions is "Tell me about a program you wrote for fun that wasn't for work or school." A good candidate will tell you about an app, website, or hack that he did for his own personal pleasure. A weaker candidate will only tell you about the program he wrote to figure something out for work/school.

I've been most of my life as an electrical engineer, and back in the 90s many of the companies I'd interview with stopped asking me questions about electrical engineering. Between my schooling and my work resume, it was obvious to them that I knew enough about electrical engineering to meet their needs. What did they ask about?

My personality. My ability to work in (and lead) a team. My cross-discipline skills. My capacity to learn.

To be ruthlessly blunt, engineering skills (especially programming skills) are a dime a dozen. There are so many basically competent people out there that there is nothing about programming or engineering that's worth asking (unless you're interviewing a new graduate with no internship experience).

What's important is being sure the candidate is suitable for your environment. How well will they work with the existing personalities in your company? Do they bring more to the table than the skills needed for the basic job? How quickly can they come up to speed with changes in company direction? technology? or design standards? Are they willing to work beyond the basic job, providing (e.g.) scholarly articles, conference presentaions, patents, and other "bring honor to the company" activities? And can they show that they can do any of this?

I can sum this up with a comment I made to Philips Semiconductor recruiters long ago (as an employee). I can teach a 10-year-old to connect the dots when it comes to microelectronic design. Teaching them WHY you connect those dots is another matter. Therefore, you can't judge a new employee by how well they connect the dots. You have to find ways to discover how well they know WHY they're doing what they're doing. That's where teamwork and extra-curricular activities come in. They show depth of understanding.

The fundamental problem here is that you are not using coding problems correctly as an interviewing technique. It sounds like your interviewing technique looks like this:

• Pose a problem


• Did the candidate successfully solve the problem? HIRE. If not, NO HIRE

This interviewing technique is, as you've discovered, (1) very easy to "game" by the candidates, and (2) gives you low quality "signal" about their actual abilities that are relevant to the job.

Here's how I give a coding problem interview:

• Pose a problem. The problem should be one that could be easily solved by anyone in this job, it should be relevant to the job, and it should not rely on any "culture-specific" knowledge. In particular it must not require an "aha-moment" insight.

For example, when interviewing for a position where the candidate would be writing a standard library for a programming language, I'd ask how to implement one of the easier methods in that library. When the position is designing tools that spot defects, I'd put some legal but buggy code up and ask the candidate to find defects and describe why they are defects and what the severity is. When the position involves manipulating parse trees in a compiler, I'd ask about extracting facts about trees, like whether a balance property is violated. And so on.

• Make sure the candidate understands the problem by working some simple examples.

These simple examples are test cases; the candidate should recognize that when it comes time for them to justify their solution.

• Give the candidate an opportunity to ask clarifying questions.

This is particularly important if the problem is in any way underspecified. "Do we have to take daylight savings time into account in this date computation?" "Do we know for certain that integer arithmetic is twos complement 32 bit?" "Is the tree shallow enough that we can write a non-tail-recursive algorithm?" and so on.

This is valuable signal about the candidate's ability to analyze a problem, communicate about it, and resolve an ambiguous situation. Those are all valuable job skills. Use this signal.

• Once the candidate can begin to solve the problem, how do they begin? Do they use "test driven" style, where they write some test cases first? Do they write a clear function header that describes the desired inputs and outputs? Do they follow standard practices for naming? If using an OO language, does the candidate follow good OO design principles? In short, can they use the function abstraction effectively?




• If the function has error cases that are not caught by the type system, does the candidate use validity checks, assertions, or any other technique to ensure that preconditions are met?




• Are there "easy out" cases that can be handled after error checks? Does the candidate write code for the easy outs? Can they describe why they made that choice? Easy out cases give us a tradeoff between code complexity and performance; under what circumstances is that tradeoff justified?




• Once the entire method is written, is the method written a valid, correct implementation of the spec? Can the candidate convince you of this? If the code is not correct, then do they try hard to convince you that it is correct? Can they use test cases that they've already got to provide evidence of its correctness? Can they come up with new test cases? Are there assertions that verify postconditions? Work with the candidate until you are both convinced that the code is correct because it is correct.




• Now we have a correct implementation. We are not even close to done yet. The problem should have been so easy that coming up with a correct implementation takes nowhere close to all of the interview. Next question: analyze the performance characteristics of the algorithm you just wrote. What resources does the candidate identify? Do they understand that performance is about user requirements, not about raw speed? Can they make tradeoffs between space and time? Do they understand amortized performance vs worst case performance? Do they understand throughput vs time to first result? Can they describe to you exogenous effects on performance like collection pressure in gc'd languages? Can they correctly give the asymptotic order? What modifications might they make to the code to adjust its performance characteristics?




• Now we see how deep the candidate's toolbox is. Tweak the problem statement to make it harder. "What if these two dates could be in different time zones?" "What if we ran this code on a machine where pointers could be of different sizes?" "What if we knew the tree was likely to be highly imbalanced on the left, and we could not do any left recursion?" "What if we're in a memory-constrained environment and we cannot allocate arbitrarily many objects?" and so on. All of the above are situations I've encountered at work. Take examples from real-world situations you've had to deal with.
  Get good signal from how the candidate deals with them.

In short, I don't care if the candidate has seen my problem before because solving the problem is the least interesting thing they are going to do in the interview. I am not looking for ability to produce a standard solution to a standard problem; I assume I'm going to get that, and if I don't, it's an easy "no hire". I am looking for ability to design, specify, clarify, implement, test, justify, analyze and extend solutions.

Temporary paid trials.

Welcome new user, I've come to believe that the only way to find software engineers is, once you've found someone who seems to know what they're talking about in the specific technology at hand,

• Simply pay them for a one or two week freelance contract. Actually have them jump in and do a real project on your real product with the real team.

This is becoming more and more common.

I think that's it.

It's the only way to know.

Everything else is useless, as the OP points out.

The amount of time/money you'll spend on the sundry alternatives ... might as well just put them on the project for a week.

(Depending on your style and project, you might retain 'em for a time period (a week) or some "task" for $x. Either way.)

If you consistently take this approach, you'll find after a year or two that, the money you wasted doing this approach, was indeed far less than the sundry other costs of search and hiring.

"Because I am frankly tired of hiring promising software developers who answer each question perfectly in job interview questions, and then when you actually see them code in a real-world situation, you realize how little they actually know."

Temporary paid trials.

Some further thoughts, the tenor of this QA is

• For software engineers, recruitment processes fail - indeed they're often "hopeless". (One apex of this, and the particular one raised in frustration by the OP, is that hopeful programmers now just "game" the "technical questions" phase of things.)

• Note that even "google-style" absurdly detailed/extended recruitment processes ............ often completely fail. Programmers (in particular) often "just don't work out" no matter how careful the recruitment processes.

• By all means in any industry or mode of occupation, folks sometimes "don't work out" even after careful recruitment. However, this is especially a problem for software engineers - it is notorious.

{You may ask - why? How come it's particularly a problem with programmers? I know precisely why this is, and some people have their own theories on it, but no need to start another argument!}

Even more thoughts!

This answer has a big pile of downvotes and a big pile of upvotes.

Apparently

• for many folks this is a totally obvious idea




• but for other folks it is freaky crazy stuff.

After all, setting aside just programming per se, back in the 90s one of the keys to the staggering success of General Electric at the time was Jack Welch's scheme where each manager simply fired the bottom 10% of people each year! (His book is terrific BTW.)

And I cringe to type the phrase "the gig economy" but in "the gig economy" we are all on "a trial basis" - from day to day and forever.

Winnowing programmers is incredibly, notoriously, difficult.

Use Whiteboard interviews, and ask problems which are related to the skills and business domain that you're interested in. Look for people that you can work with, even if they aren't perfect matches for the position, because you can always train the right person if they have potential. Ask existing employees for recommendations (and offer a finders-fee)

After getting through enough basic questions to show that they have some minimal (by my needs) ability to write code, I move on to the tougher part of the interview, based on two subject areas, their work and my work.

Ask for information about a recent project at their current job. Even with respect to NDAs and IP restrictions, they should be able to explain the business problem, the complexity of the problems and how they overcame them. If they can't explain their own work to me so that I can understand and evaluate, then I worry about how successful they will be.

Ask them to white board a design/solution to one of your current problems. The expectation should not be a working solution but how they approach the problem. Provide enough information that they have a high level scope for the problem. They should ask good questions, and you should provide good answers so they can proceed. They should be able to sketch out some sort of solution. In the end their solution may not work out, but you will be able to see how they work on a problem and whether or not they should be able to work on the projects you will have.

Simply. You just grill the applicants on the implementation details and change requirements on them to see how they respond. Make them use what they've memorized as building blocks.

To be frank, problem memorizing isn't an issue. Memorization is actually one of the key techniques chess players use to learn the game. By memorizing algorithms your applicants are developing a library of solutions from which to work from. This means that they'll be able to solve problems better in the future, if the patterns and implementation of the patterns they're learning are actually useful.

What you need to test is that last part. By varying the problem in real time you get to see how well the applicant actually understands how to use what they've memorized. This means that you can ask them about better ways to solve the question and you can even ask them to compose their solution into a work flow to do something completely different.

What you want is someone who can solve problems. What you don't want is someone who solves problems the way you want to see them solved. If everyone's memorizing solutions, all you need to do is test their ability to use what they've memorized. When it comes down to it, everyone uses what they know to break down what they don't know. If your devs can use patterns they've already mastered to break down problems of increasing complexity, then they would have been able to reverse a linked list if they've never seen anyone do it before.

As a recruiter, this is as far as you can reasonably be expected to go without trying to get free labor out of your applicants.

In some ways, studying for the interview is a good thing. You probably already ask that people study about four years in a University to prepare for the interview.

Now, if your interviewing process is scripted, and the script lacks any kind of variation, then it is just a matter of time before the answers become memorized in the population. Solutions include:

1. As questions that are generated, with variable fields that require the work to be done specific to this asking of the question.


2. Change the question set from time to time, to minimize the time to build the pool of answers.


3. Change the entire approach, giving them a trivial project with specifically added bugs, asking them to fix it and add a known feature.

Also, consider the problem in the larger sense, if you are asking for information in an interview, and the information was memorized, then that should be an ideal candidate, unless you are asking for the wrong information.

You are deciding that the information isn't what is wanted, so you might be able to fix the problem by asking for the right kind of information.

Being able to come up with a solution on the spot isn't really that great a qualification, either, at least IMHO. I've certainly come up with a lot of on-the-spot solutions that in retrospect weren't all that great, and have sometimes spent weeks or months finding a decent solution to a really hard problem. (And sometimes the solution is just remembering that I saw something similar in a code example somewhere...)

You don't say whether you're trying to hire new grads or experienced developers, but in either case I think you really have to find some way to look at what they've done previously. Sometimes it's looking at published papers (even if the person isn't the principal author), sometimes it's a personal recommendation, sometimes just asking them to show you something they've done that they're proud of. FWIW, I don't think I've ever gotten a job that I interviewed for that didn't have at least one of those things going for it.

I believe you misunderstand the reason why big companies ask algorithm questions. Writing out the algorithm for quicksort is probably not a very useful skill for the majority of developers. Likewise figuring out the best way to parse a 1 million character string is not something people routinely do at work. However knowing how to solve these problems is a great way of filtering out the dumb candidates. If someone is too dumb for the job, they won't be able to coherently answer any of the interviewing puzzles commonly asked at job interviews. Even if they memorize all the answers, you could easily break their pattern by asking a slightly different variation of the question or adding a bonus question on the spot. In contrast, a good candidate would have no issues preparing for the "test" and would pass with flying colors.

Interviewing is about filtering out the bad candidates, rather than about finding the best candidates per se. There's no harm in having someone prepare in advance, as this wouldn't help someone who's not prepared to work on the right level.

The problem is that you're not testing for the skills you're looking for.

A process I've used both as an interviewer and interviewee that has worked well:

1. A quick 15 minute phone screening to find out if the person is competent and someone you could see yourself hiring.




2. A simple coding exercise for the applicant to complete on their own time. This is where you get a feel for actual problem solving and coding skills. The exercise should not take more than a few hours and you can optionally pay them for their time.




3. The actual technical interview, which involves reviewing the coding exercise and discussing the solution and the decision making process.

Most of the (currently) highly rated answers to this question suggest some sort of live coding as a part of the interview. I would highly discourage this as it is not at all representative of a real world scenario. It's a high pressure, unnatural situation where a candidate is likely to produce an inferior solution than they otherwise would, with no additional benefit.

As an applicant, there are a few things I noticed which can still be tested during an interview, even if you're not in the IT field yourself:

• Behaviour-based questions

The objective is not to nail the person you have in front of you, but actually see what kind of person they might be. This gives a short window of whether or not they might be suited for working in the company.

• Past experience questions

[Could you] Tell me about a situation when...

For this one, the objective is to invite the person into telling you more about their experience in a specific situation. This type of question can assert their skills as well as their self-being.

It can be useful if you know the IT teams has some situations which can be difficult to handle (having to handle a lot of tasks in short notice, having to handle tired clients, etc).

• Problem-solving based question

You mentioned you are looking for problem-solver.

I once had an interview with a company where the interviewers didn't ask any programming based question. In order to assert my problem-solving skills, they instead told me about a situation, and I was told to think aloud so they can witness my problem-solving skills in action. For example:

We have a plane with 50 seats, and 50 passengers awaiting to come in. The first two passengers lost their cards, so they will sit in a random seat. The next passenger will sit in their seat if they can (if it's not already occupied); otherwise, they will sit in a random seat. What is the probability for the last passenger to sit in his own seat?

The objective is not for them to find the solution, but to show you how they tackle the problem.

~~~~~~~~~~

For each of these questions, remember to take note of the applicant's answer. This way, you can better discuss about them with the IT department manager.

I don't think you need to change the questions, just how and what you ask.

It's gotten to the point where you pretty much can't ask a question without the applicant already having pre-studied at least a version of that same task and hence knows, roughly, what the solution is.

This isn't a zero sum thing, you can feed off the solution for an entire interview on it's own if you want, and get a deep understanding of the interviewee's skills (assuming as the interviewer you have your own technical skills).

You present the question, the interviewee creates a solution (maybe code, maybe whiteboard). This is the start:

• Take me through the way this works


• What factors led you to choose this solution?


• Are there any other ways of doing this?


• Why is your solution better than the others?


• Are there any potential advantages in the other solutions missing from your solution? what scenarios might that apply?


• Having just written this off the cuff, any refactorings you'd like to do now you've looked it over?


• If you didn't do this test driven, what tests would you be looking to write/do to ensure this is correct?


• Any bugs you've noticed?


• Do you think the code is production quality? Could another developer support/refactor it without knowledge transfer from you? What would you need to change to ensure it's intent is understood?

Someone who has memorized an algorithm or pattern without understanding it will not be able to drill down into detail, but a good developer should be able to have discussions on these topics, even if their raw solution wasn't perfect (and seeing someone discover a better solution while talking to you and be prepared to call it out shows a maturity over code/design reviews and refactoring which are highly desirable).

The best interview I ever gave was one of the simplest interviews I gave and was also the best my skills were ever measured. This is what its 5 rounds were:

1. Telephonic with the recruiter: who asked me to verbally go about explaining the answer to a simple coding problem. (The recruiter was not technical.) Answering him demonstrated that I can explain my solution to non-technical people, proving that there is a real connect between the skills mentioned on my resume and practicality.




2. On-site 1: One of the company's senior dev demonstrated their product to me. He paid attention to the questions I was asking during and after knowing their product. This helped them accurately gauge my interest in that domain and whether I was genuinely curious.




3. On-site 2: A tech-manager from the company brought one of their own pieces of code and asked me what I think of it. The code had basic issues - like not using try-catch, not closing resources, poorly named classes and variable etc. The code was working, but was not written responsibly. The corrections I suggested, demonstrated my design skills. If one is not a responsible programmer, this interview would end up being the hardest. Especially for solution muggers.




4. On-site 3: A senior developer discussed with me the design of Merge Sort and after correct design was reached, asked me to implement it in my language of choice. The design discussion was relaxing as I was sure that him and I were on the same page about what we were going to implement.




5. On-site 4: Another senior dev asked me to explain one of my own projects to him. This demonstrated that I actually understood things being done in my current job instead of doing short term and nonsensical code patching.

All the rounds were very relevant and nowhere did I feel that I was being asked to do something out of the blue especially for this interview. Every task was gauging how good a software developer I was, rather than how many answers had I memorized.

Not surprisingly, this company was very highly rated on Glassdoor and had very low attrition. Employees didn't write much about what they were doing but they all were really invested in their company and their work.

In my job, the right solution to difficult problems has come after discussion between teammates. That is what a good interview would try to simulate. Just asking difficult problems in an interview is generally not helpful.

A number of people have answered "The questions don't match the needs of the job" - and that the solution is to change the questions to something that more accurately reflects what's needed to know how to do the job.

I'll take a different angle: This probably has a lot to do with recruiters/interviewers not putting in the time to create their own questions.

In our group, we're currently interviewing .NET devs, and here are some of the questions from our interviewing process:

• We've got these two SQL tables with these columns. Write a query that gathers data X, filtering down based on Y, with a corner case of Z being handled in a specific way.


• Here's a badly-written 8-line C# function; Code review it.


• Write a C# function that takes an input of a filename, and outputs how many bytes there are before the first null byte in that file.

The answers to those probably aren't going to be found in a '140 questions to memorize before an interview' - because we came up with the questions from scratch. We didn't copy them from another company, or crib them off a 'Good Interview Questions' site. Yeah, it takes more work... but we don't have to worry about someone already having the answer memorized.

If your candidates are able to pass your interviews by studying, isn't that a strong indication of their ability to study and learn? If you're hiring fresh grads, that's really the most valuable skill to hire for.

If an experienced candidate is doing it, at the very minimum it means they're highly motivated to study for your interview. Which means they're strongly interested in coming to work for your company. And also, it's a strong signal that they haven't lost their ability to learn and grind after leaving college. You should still ask them questions about their work experience, achievements, and collaborations with co-workers to get a better sense of their overall ability.

It's problem-solvers that we want, not problem-memorizers.

Very few people can come up with quicksort, or Floyd's cycle detection algorithm, or finding the longest increasing subsequence in the timespan of an interview having never studied the solution. Those who can are unlikely to be interviewing at your company for regular dev jobs. No offense, I'm sure it's a great company, but statistically most of your candidates are not going to be of that caliber - because very few programmers in the world are.

On the other hand, if your candidates are able to apply solutions to problems they've seen before to variations of that problem or to related problems, it's a signal that they're good at pattern-matching and identifying what type of problem they're looking at. Conversely, if they've never seen that type of problem before you also have signal that they know how to research a solution and understand it - because after all, they studied for, and passed your interview.

I've had to recruit 5 developers of different skill levels over the course of the past year and it's not been an easy process. The best method we found was to make use of online programming tests to filter out the weaker candidates before we interviewed.

We picked a coding test website (CodinGame, though other alternatives are available) and asked candidates to sit a test before we interviewed them.

Pros:

• Filtered out the 'looks good on paper' candidates who didn't live up to their CV / resume.


• Allowed us to see the candidate's working - we offered interviews to people who didn't score highly on the test but by reviewing what they had done to solve the problem gave us a good idea of their analytical skills.


• We viewed candidates who weren't serious enough about the position to do a 50 minute online test as not worth interviewing, this filtered them out.


• We could set tests at different skill levels.

Cons:

• Some candidates, when faced with several interviews, would go for the path of least resistance and choose the the ones that didn't involve a coding test.


• You need a few candidates to take the test to get a benchmark of what is a decent score.


• There is a cost involved in taking the test.

Flip it: Make it a test for gamers. Pick those who fail...

Anytime you have a metric that is being intensely gamed, the metric becomes useless.

But it's worse. Excellent performance on the metric becomes a flag for a gamer.

And that makes it better :)

So you flip it. You optimize this testing to identify people who have learned to test well. Don't make your testing gamer-resistant -- make it even more gamer-vulnerable, so gamers will do exceedingly well on it, and people who have not "trained up on interview books" will not. Also include other interviewing or testing that the interview-training books cannot prepare you for.

Now you are going to see a pattern/signal. You will see candidates who score really well on the gamer test, but flounder on your other stuff. Those are gamers, don't hire them. Conversely you see candidates who show relatively poorly in the gamer test, say in the 25-50th percentile compared to other candidates. they're not incompetent, they're just getting trounced by the gamers... and they do just fine on the other stuff. Hire them.

You say

"Now, this would of course not be a problem if knowledge of those
problems is what we are looking for ... except, that isn't what we are
looking for."... "It's problem-solvers that we want, not
problem-memorizers." .

You just answered your own question.
If you want to test their problem solving skills, just ask them to explain their earlier projects and work in detail. Ask them questions like why they chose a particular platform/programming language. Ask them what were the requirements and how they were implemented etc. Questions like these will give you an idea about how much complexity the candidate has handled before.
Some have suggested to actually give them real debugging problems during the interview. Although a good approach, it can be time consuming. Also, every bug is different and even great programmers struggle if they don't know the history of the code.
I think you can have a mixture of all three approaches i.e some textbook questions + earlier projects + actual bugs.

Instead of asking specific pre-prepared questions, ask the candidate to bring samples of their work. For graduates that could be stuff they did on their course or as a hobby.

You can then discuss it with them, ask them to explain what it does and any particularly interesting parts of it. You will get a lot more from this process than you would simply asking technical question. You can see how critical they are of their own work, and how they think about solving problems.

It also takes the pressure off them. If you ask them to do coding tasks under time pressure and with you watching they may not perform well, and it's a very artificial environment and something they are likely not used to.

You can use this discussion as a jumping off point for talking about things more specific to the job, but the focus with graduates tends to be on the ability to think and learn.