Talent Development 16 Surveys That Work

November 8, 2020

Section 2.8 in the CPTD Certification program for ATD is Evaluating Impact. Section B reads, “Skill in creating data collection tools, for example questionnaires, surveys, and structured interviews.”

For about 5 years, I taught a graduate course called Experimental Design. The course was part of a curriculum leading to a Masters Degree in Organization Learning and Human Resources Development.

In preparation for writing this brief article, I went back and reviewed my slide deck for the course. It was 200 slides long, and many of the slides were heavy with statistical techniques. Obviously, I will need to skim the surface in this short summary article.

In the course, we studied how well-intended surveys often miss the mark and produce bogus results. We also studied the antidotes.

Why Surveys Fail

There are many reasons why surveys fail. I will list some of the more common reasons here and then describe the typical antidotes.

1. Survey too long and complex – one reason for poor data is because people get turned off by too burdensome and too many surveys. When people are angry about too many surveys, they give responses that do not reflect their true feelings.

2. Changes made are not evident – if leaders do not stress that a change is being made because of an employee survey, people will believe their input was ignored. The common misconception sounds like this, “They make us fill out all these ridiculous surveys, but nothing ever changes.”

3. Survey not valid – incorrectly designed surveys often do not measure the thing they are trying to measure. Surveys must be statistically validated to be useful.

4. Survey not reliable – If you would repeat the survey a second time, you would get a different result?

5. Questionable anonymity – If people believe there is some secret way management can find out who said what, then the instrument will not give accurate results. People will respond in ways they think management wants to hear.

6. No clear objective – When people are asked to fill out a survey, they need to know ahead of time why they are being asked to participate and what to expect.

7. Questions not clear – Often the wording of questions leaves people wondering what is really meant by the questions. In this case, you will get guesswork rather than valid data.

8. Leading Questions – Sometimes the way questions are worded leads to skewed data. For example, a political survey might ask, “Are you frustrated by the lies being spread by my opponent?”

Antidotes

I will list the antidotes to the problems in the same order.

1. Make sure your survey is user friendly. Take the survey yourself and ask if you would take the time to do this instrument justice on your most busy day. A good rule of thumb is to be able to fill out the information in less than 10 minutes.

2. Make sure you get back to everyone who responded with the results of the survey. Also, tie all changes made to the survey results, so people are aware of the connection.

3. Test if the survey is valid. The only way you can tell is a survey is measuring what you are trying to measure is to use a statistical analysis of the data. There are five different types of validity (Content, Construct, Concurrent, Criterion, and Predictive). Get help from someone qualified to measure validity. Don’t just wing it.

4. Test if the survey is reliable. This involves trial runs of the survey with different groups under different conditions. The survey needs to produce consistent information to be reliable. Another method is to use a split-half technique. Again, get help if you are not an expert in this area.

5. Insuring anonymity is tricky – The best method I have found to get people to really believe the survey is anonymous is to select a skeptical person from the population to help reduce the data into usable form. The skeptic will let others know that there was no secret means by which management knew who said what.

6. Clarity of Objective – This is a matter of good survey design. It is not just a simple matter of generating some questions and handing out the survey. It must be done with care and solid logic. The way the survey is introduced (typically with an email or letter) is critical. Otherwise you have garbage in garbage out.

7. Test your questions for understandability – This is usually done in the final design phase. You ask people how they interpreted the question. It is not uncommon for many people to be baffled by the wording. Check it out carefully.

8. Avoid leading questions – do not telegraph the requested answer by the way a question is worded. Like don’t ask “Would you like a yummy pizza?”

These areas are general, but they do show how generating a survey is not so simple as most people believe. If a survey is going to generate valuable information for the organization, it needs to be constructed well and administered correctly.

Bob Whipple, MBA, CPTD, is a consultant, trainer, speaker, and author in the areas of leadership and trust. He is the author of: The Trust Factor: Advanced Leadership for Professionals, Understanding E-Body Language: Building Trust Online, Leading with Trust is Like Sailing Downwind, and Trust in Transition: Navigating Organizational Change. Bob has many years as a senior executive with a Fortune 500 Company and with non-profit organizations.


Trust Seeds

April 22, 2012

We are all aware that interpersonal trust is precious. Trust is fragile; it is difficult to build, and easy to destroy. Most people believe it takes a very long time to build up trust with another person. There is an alternate view; if certain conditions are present when people first meet, a “seed” of trust is created upon which further trust will grow if both people continue to nurture it.

In his book “Blink,” Malcolm Gladwell describes the “Thin Slices” we humans use to size up other people within seconds of meeting them. We absorb an enormous amount of data instantly in the body language and the first words uttered by a new acquaintance.

I can recall meeting two influential men last year within seconds of each other. The first one gave me a solid handshake and a smile. He made great eye contact and asked me a question about my family. The second individual gave me a half-limp handshake while his eyes were scanning the room to see who else was there. He did smile, but it was forced and phony. Since that time, I have effortlessly developed a relationship of high trust with the first individual, and I have felt uncomfortable to be in the same room as the second one. The relationship with the first man took several months to develop, but the seed was planted in the first 5 seconds. With the second man, there was nothing for trust to grow on, so a relationship never kindled.

There are numerous things people instantly assess about us. Here are five conditions that allow you to plant a seed on which trust can grow.

Competence – People must be convinced that you know what you are doing to view you as being trustworthy. If they sense that you have the ability from a knowledge and skill set to deliver on your statements, then you pass the competence test. If they have doubts that you can deliver, then they will remain skeptical until there is enough time to test you.

Integrity – Do you have the character to do what is right? People need to feel that you are not duplicitous and that you will stand up for what you believe is right. It does not mean that you always need to agree with others on every point, but people need to see you as a person of high moral and ethical fiber before they are going to trust you.

Reliability – People need to be convinced that you will do what you say. This characteristic normally takes people a long time to test, but it actually can happen quickly. People can discern your reliability through the way you phrase intentions and even the body language you use to chat with them. The ability to follow through with intended actions or at least get back to the other person if conditions change is easy to spot, just as it is easy to observe a blowhard who says nice things but has no intention to actually do them.

Attitude – To gain trust, you need to project a positive attitude when another person is meeting you and ensure that it comes from the heart. Depending on the contextual background of the meeting, a smile is the usual way to show a positive attitude toward another person. Caveat: putting on a false smile is the kiss of death, because it pegs you as someone who cannot be trusted at all. In a different context, a look of concern or sympathy might be a more appropriate way to show a positive attitude toward the other person. Your attitude and demeanor must be heartfelt and congruent with the situation.

Care – It is vital to project that you really do care about the other person. People might say it takes years to know if someone else really does care about you. In reality, care can be displayed in hundreds of small gestures, just as selfishness can be easily spotted. Giving deference to the feelings of others is an important component of Emotional Intelligence. The interesting observation about this is that the people who have low Emotional Intelligence have the biggest blindspots, according to Daniel Goleman. Translated, if you come across as a phony in terms of really caring about other people, you will not have the ability to detect this in yourself, but others will see it instantly.

On the back of my business card, I have a picture of a pile of various seeds. The words say:

Seeds for Growing Leaders
Plant in an environment of trust,
Sprinkle daily with humility,
Weed out negativity,
Place in the light of truth,
Be patient,
Enjoy the fruits of great leadership.

It does take a long time of consistent performance for a very strong bond of trust to build, but the first seeds of trust can be established quickly upon meeting someone. Make sure when you meet a new person that you genuinely project the five conditions above, and you will be well on your way to a trusting relationship.


Few Employee Surveys Work

January 30, 2011

We have all been exposed to an employee satisfaction survey at some point in our working lives. For some of us, the idea of filling out yet another QWL (Quality of Work Life) survey is not nearly as appealing as having a root canal. I have witnessed a significant hit to morale in many groups as a result of these attempts by management to gather information. Let’s examine why employee surveys cause problems and suggest some antidotes that can make a big difference.

1. Questionable anonymity – Nearly all QWL surveys are advertised by managers as being anonymous. This is to encourage people to share information without fear of repercussion. Unfortunately, nearly all surveys these days are conducted electronically. Most people are aware that anything online can be traced, if a smart IT technician is assigned to the case. People simply do not believe the promise of anonymity, which lowers the validity of the data. One way to give slightly more confidence in the integrity of the process is to include a statement such as this at the end of the form, “You do not have to type your name on this survey because it is anonymous. We will pay no attention to who you are if you do not sign the document. However, some people do wish to be contacted personally to give further input that might not be adequately covered on the survey. For that reason, there is a line to give your name if you wish. Someone will get back to you.”

Of course, no statement by management is going to convince the die-hard skeptics, but this explanation will assist most people. Another action that can help is to have a shop floor person (perhaps a known skeptic) participate in the data analysis phase, so he or she can verify personally that the managers do not know who said what.

If you claim anonymity but are quietly keeping track on the side out of curiosity, then there is no hope for you, and you should get out of the leadership occupation immediately.

2. Poorly worded questions – many managers believe that putting together a survey is a simple matter of writing a few questions about how people feel. Actually, survey design is a rather complex and exacting science. There are numerous ways to present questions that will yield meaningless (or at least ambiguous) information. There are also some methods that will generally produce usable data. The cure for this problem is to have someone who is trained on survey questions actually construct the instrument. If you have not had at least one course in experimental research design, then it is best to leave the matter to someone who has.

3. Long and tedious survey – It is not uncommon for QWL surveys to contain over 100 detailed questions. It is amazing that the designers of these surveys do not realize the obvious fatigue factor involved in completing one of these burdensome questionnaires. A much more accurate reading can be obtained by keeping the number of questions to 20 or less. This can be accomplished by paying attention and only asking important questions. Leaving out the fluff can cut the time to take a survey by more than half.

4. Management interprets the information as they please – It is frustrating to witness how managers wave away complaints or gripes on surveys as simple whining. Or they might shrug their shoulders and say, “there is absolutely nothing we can do about this issue,” so they consider the input as moot. The antidote here is to not ignore input regardless of how painful it is or how frivolous it seems. All input needs to be considered valid and not assumed away with some convenient rationalization.

5. “Nothing ever changes” – This is a common theme on the shop floor. “We take these stupid surveys, but nothing ever changes.” The antidote to this habitual problem is to actually take concrete actions based on the survey, then (and this is the part most managers forget) advertise that the changes are being made as a result of the QWL survey. Rather than saying, “We are going to add a second brief afternoon break,” say “As a result of your input in the recent survey, we are changing the break rules to allow a short second break in the afternoon. As always, we appreciate your candid feedback.” If managers do not make a conscious effort to communicate that changes are the result of input, people will usually not make the connection. Once a change is made and it becomes habit, people forget that there was a change, so the perception of “nothing ever changes” is common.

6. Managers try to react but do the wrong thing – It is far better to let the shop floor people be involved in decisions of how to improve conditions based on survey results. It may take a little more time, but the quality of process changes will be far better if those impacted the most have a say in their invention. They will make the changes work rather than wonder and push back at the clueless inventions of upper management. It is better to have workers inside the tent piddling out, than outside the tent piddling in.

7. Managers reacting to the vocal minority rather than the silent majority – This problem is common when surveys give the opportunity for open-ended comments. People on the fringe can give strong input, and managers might mistakenly interpret this to be the will of the majority. The simple antidote to this problem is to verify that a strong message really does come from several individuals rather than one highly disgruntled outlier.

8. Survey not tested for validity – For a survey to be useful, it needs to measure the phenomena it purports to measure. There are statistical techniques for determining if an instrument has validity. You may not have the time or money to invest in a professional survey designer to test the validity of an instrument, but at least you should ask the question of whether you are actually getting valid information.

9. No thought to reliability – The reliability of a survey is different from validity. For a survey to be reliable, it should produce a similar result if repeated and there have been no changes in processes since the last survey was taken. If survey results are all over the map when nothing in the environment in changing, it is a sign that the instrument is not reliable (repeatable).

10. Poorly communicated – When surveys are sent out, the cover letter explaining the purpose and process is a critical document. Many managers have an administration person whip out a paragraph of “management speak” like this. “It is vital that we know what people in our operation think in order to continually improve working conditions. Please take the time to fill out this anonymous survey that will give us the information. Thank you.” Here is a different note where the manager took the time to set up the survey for success.

“We are going to re-do our strategic plan, and it is important to include your input before making changes. The attached survey will begin the process. Before you take this survey, please reflect on the following points:
• The survey really is anonymous – we will have shop floor people help with the analysis to verify no names are attached to the data.
• We will summarize the data for you as soon as it is received.
• We will use the information as the basis for a series of meetings (you are invited to participate) on how this business can be improved.
• We will be making changes based on the results of this survey.
• We are all part of making this organization a success.”

With an introduction like that, employees will know this survey will likely have some impact and their viewpoints matter. It is critical to not waste credibility, time, and energy on a poorly designed and administered QWL survey. If the above 10 points are considered when designing an employee survey, it will produce results that can be the basis of solid organizational progress.