Talent Development 16 Surveys That Work

November 8, 2020

Section 2.8 in the CPTD Certification program for ATD is Evaluating Impact. Section B reads, “Skill in creating data collection tools, for example questionnaires, surveys, and structured interviews.”

For about 5 years, I taught a graduate course called Experimental Design. The course was part of a curriculum leading to a Masters Degree in Organization Learning and Human Resources Development.

In preparation for writing this brief article, I went back and reviewed my slide deck for the course. It was 200 slides long, and many of the slides were heavy with statistical techniques. Obviously, I will need to skim the surface in this short summary article.

In the course, we studied how well-intended surveys often miss the mark and produce bogus results. We also studied the antidotes.

Why Surveys Fail

There are many reasons why surveys fail. I will list some of the more common reasons here and then describe the typical antidotes.

1. Survey too long and complex – one reason for poor data is because people get turned off by too burdensome and too many surveys. When people are angry about too many surveys, they give responses that do not reflect their true feelings.

2. Changes made are not evident – if leaders do not stress that a change is being made because of an employee survey, people will believe their input was ignored. The common misconception sounds like this, “They make us fill out all these ridiculous surveys, but nothing ever changes.”

3. Survey not valid – incorrectly designed surveys often do not measure the thing they are trying to measure. Surveys must be statistically validated to be useful.

4. Survey not reliable – If you would repeat the survey a second time, you would get a different result?

5. Questionable anonymity – If people believe there is some secret way management can find out who said what, then the instrument will not give accurate results. People will respond in ways they think management wants to hear.

6. No clear objective – When people are asked to fill out a survey, they need to know ahead of time why they are being asked to participate and what to expect.

7. Questions not clear – Often the wording of questions leaves people wondering what is really meant by the questions. In this case, you will get guesswork rather than valid data.

8. Leading Questions – Sometimes the way questions are worded leads to skewed data. For example, a political survey might ask, “Are you frustrated by the lies being spread by my opponent?”

Antidotes

I will list the antidotes to the problems in the same order.

1. Make sure your survey is user friendly. Take the survey yourself and ask if you would take the time to do this instrument justice on your most busy day. A good rule of thumb is to be able to fill out the information in less than 10 minutes.

2. Make sure you get back to everyone who responded with the results of the survey. Also, tie all changes made to the survey results, so people are aware of the connection.

3. Test if the survey is valid. The only way you can tell is a survey is measuring what you are trying to measure is to use a statistical analysis of the data. There are five different types of validity (Content, Construct, Concurrent, Criterion, and Predictive). Get help from someone qualified to measure validity. Don’t just wing it.

4. Test if the survey is reliable. This involves trial runs of the survey with different groups under different conditions. The survey needs to produce consistent information to be reliable. Another method is to use a split-half technique. Again, get help if you are not an expert in this area.

5. Insuring anonymity is tricky – The best method I have found to get people to really believe the survey is anonymous is to select a skeptical person from the population to help reduce the data into usable form. The skeptic will let others know that there was no secret means by which management knew who said what.

6. Clarity of Objective – This is a matter of good survey design. It is not just a simple matter of generating some questions and handing out the survey. It must be done with care and solid logic. The way the survey is introduced (typically with an email or letter) is critical. Otherwise you have garbage in garbage out.

7. Test your questions for understandability – This is usually done in the final design phase. You ask people how they interpreted the question. It is not uncommon for many people to be baffled by the wording. Check it out carefully.

8. Avoid leading questions – do not telegraph the requested answer by the way a question is worded. Like don’t ask “Would you like a yummy pizza?”

These areas are general, but they do show how generating a survey is not so simple as most people believe. If a survey is going to generate valuable information for the organization, it needs to be constructed well and administered correctly.

Bob Whipple, MBA, CPTD, is a consultant, trainer, speaker, and author in the areas of leadership and trust. He is the author of: The Trust Factor: Advanced Leadership for Professionals, Understanding E-Body Language: Building Trust Online, Leading with Trust is Like Sailing Downwind, and Trust in Transition: Navigating Organizational Change. Bob has many years as a senior executive with a Fortune 500 Company and with non-profit organizations.


Successful Supervisor 37 – Mastering Body Language

July 30, 2017

The topic of Body Language has fascinated me for decades. There is no way to cover the topic completely in a short blog article, but I can give some tips that may help supervisors know how to prevent misreading BL and how to control their own so the message they are sending with their body is consistent with their verbal message.

I will also share some resources at the end in case this brief introduction whets the appetite for further study.

Body language points us in the direction of what people are feeling, but it is not an exact science. A person could make a gesture that is a random act or something not indicative of the classical meaning.

For example, if a person touches the side of his nose when giving a response, the experts would say that he is lying or exaggerating. Alternatively, it could simply mean that he had an itchy nose at that moment.

Taking singular pieces of body language and assigning specific meaning can result in some wrong conclusions. So how do we know which things are the real meaning?

In interpreting body language, keep the following five “C’s” in mind, and you will improve your accuracy at reading people.

1. Context

Pay attention to what is going on around the person. Body language is contextual and can vary greatly due to ambient conditions.

Folded arms at a cocktail party might suggest the person is being defensive, but folded arms in a snow storm is more likely to be the result of the person being cold.

2. Congruence

Do the words and body language match? If we are faced with two signals, one from the words that are spoken and another from the body language, then we will likely believe the latter.

For example, if I ask a coworker if he is angry with me and he glares at me with a scowl and clenches his fists while he says “NO!,” I am most likely to believe he is angry with me, even though what he said denied it.

3. Clusters

If we see a single bit of body language, we might suspect that it is an indication of something, but it is hard to tell. However, if I see an individual exhibit several signals of a particular body language cue, then I can be very sure of the conclusion.

For example, if a person is wringing his hands and shuffling his feet while wrinkling his forehead and not maintaining eye contact, I can be pretty sure the person is feeling anxious.

4. Consistency

We all have habitual patterns of body language that we revert to when nothing special is going on. While you are sitting in a classroom or in church, you will habitually cross your legs in a certain way or touch your face in a certain spot.

It is best to not interpret body language that is habitual as some kind of signal, but if the habitual position changes, especially as a result of some comment or other stimulus, then the change in body language is probably a signal.

For example, if a person is listening to me and suddenly starts pulling on his ear lobe as I shift the conversation to the new employee, he is likely showing high interest in what I am saying.

5. Culture

Body language and proximity (which is a part of BL) vary greatly from one region of a country to another and even more so from one country to another.

If I am a USA-based business person and I am doing some work with a person from Saudi Arabia, I may find that he stands a little too close for my comfort level.

Likewise if I meet an Eskimo, I might interpret his head shaking side to side as a “no” response, when he is actually telling me “yes.”

Beyond these five general rules, there are thousands of facial expressions and other cues relative to body language. The more you know these cues the better off you will be at interpreting their meaning accurately.

A few of my favorite resources:

1. A very old book, but still available is “How to Read a Person Like a Book” by Nierenberg.
2. A great online test of your ability to read facial expressions accurately.
3. Online resource Body Language Dictionary
4. A DVD of Bill Acheson; a body language expert from University of Pittsburgh

This is a part in a series of articles on “Successful Supervision.” The entire series can be viewed on http://www.leadergrow.com/articles/supervision or on this blog.

Bob Whipple, MBA, CPLP, is a consultant, trainer, speaker, and author in the areas of leadership and trust. He is the author of four books: 1.The Trust Factor: Advanced Leadership for Professionals (2003), 2. Understanding E-Body Language: Building Trust Online (2006), 3. Leading with Trust is Like Sailing Downwind (2009), and 4. Trust in Transition: Navigating Organizational Change (2014). In addition, he has authored over 500 articles and videos on various topics in leadership and trust. Bob has many years as a senior executive with a Fortune 500 Company and with non-profit organizations. For more information, or to bring Bob in to speak at your next event, contact him at http://www.Leadergrow.com, bwhipple@leadergrow.com or 585.392.7763