Talent Development 16 Surveys That Work

November 8, 2020

Section 2.8 in the CPTD Certification program for ATD is Evaluating Impact. Section B reads, “Skill in creating data collection tools, for example questionnaires, surveys, and structured interviews.”

For about 5 years, I taught a graduate course called Experimental Design. The course was part of a curriculum leading to a Masters Degree in Organization Learning and Human Resources Development.

In preparation for writing this brief article, I went back and reviewed my slide deck for the course. It was 200 slides long, and many of the slides were heavy with statistical techniques. Obviously, I will need to skim the surface in this short summary article.

In the course, we studied how well-intended surveys often miss the mark and produce bogus results. We also studied the antidotes.

Why Surveys Fail

There are many reasons why surveys fail. I will list some of the more common reasons here and then describe the typical antidotes.

1. Survey too long and complex – one reason for poor data is because people get turned off by too burdensome and too many surveys. When people are angry about too many surveys, they give responses that do not reflect their true feelings.

2. Changes made are not evident – if leaders do not stress that a change is being made because of an employee survey, people will believe their input was ignored. The common misconception sounds like this, “They make us fill out all these ridiculous surveys, but nothing ever changes.”

3. Survey not valid – incorrectly designed surveys often do not measure the thing they are trying to measure. Surveys must be statistically validated to be useful.

4. Survey not reliable – If you would repeat the survey a second time, you would get a different result?

5. Questionable anonymity – If people believe there is some secret way management can find out who said what, then the instrument will not give accurate results. People will respond in ways they think management wants to hear.

6. No clear objective – When people are asked to fill out a survey, they need to know ahead of time why they are being asked to participate and what to expect.

7. Questions not clear – Often the wording of questions leaves people wondering what is really meant by the questions. In this case, you will get guesswork rather than valid data.

8. Leading Questions – Sometimes the way questions are worded leads to skewed data. For example, a political survey might ask, “Are you frustrated by the lies being spread by my opponent?”

Antidotes

I will list the antidotes to the problems in the same order.

1. Make sure your survey is user friendly. Take the survey yourself and ask if you would take the time to do this instrument justice on your most busy day. A good rule of thumb is to be able to fill out the information in less than 10 minutes.

2. Make sure you get back to everyone who responded with the results of the survey. Also, tie all changes made to the survey results, so people are aware of the connection.

3. Test if the survey is valid. The only way you can tell is a survey is measuring what you are trying to measure is to use a statistical analysis of the data. There are five different types of validity (Content, Construct, Concurrent, Criterion, and Predictive). Get help from someone qualified to measure validity. Don’t just wing it.

4. Test if the survey is reliable. This involves trial runs of the survey with different groups under different conditions. The survey needs to produce consistent information to be reliable. Another method is to use a split-half technique. Again, get help if you are not an expert in this area.

5. Insuring anonymity is tricky – The best method I have found to get people to really believe the survey is anonymous is to select a skeptical person from the population to help reduce the data into usable form. The skeptic will let others know that there was no secret means by which management knew who said what.

6. Clarity of Objective – This is a matter of good survey design. It is not just a simple matter of generating some questions and handing out the survey. It must be done with care and solid logic. The way the survey is introduced (typically with an email or letter) is critical. Otherwise you have garbage in garbage out.

7. Test your questions for understandability – This is usually done in the final design phase. You ask people how they interpreted the question. It is not uncommon for many people to be baffled by the wording. Check it out carefully.

8. Avoid leading questions – do not telegraph the requested answer by the way a question is worded. Like don’t ask “Would you like a yummy pizza?”

These areas are general, but they do show how generating a survey is not so simple as most people believe. If a survey is going to generate valuable information for the organization, it needs to be constructed well and administered correctly.

Bob Whipple, MBA, CPTD, is a consultant, trainer, speaker, and author in the areas of leadership and trust. He is the author of: The Trust Factor: Advanced Leadership for Professionals, Understanding E-Body Language: Building Trust Online, Leading with Trust is Like Sailing Downwind, and Trust in Transition: Navigating Organizational Change. Bob has many years as a senior executive with a Fortune 500 Company and with non-profit organizations.


Talent Development 7 Cultural Awareness and Inclusion

August 16, 2020

The topics of cultural awareness and inclusion are part of the ATD CPTD Certification model. Basically, this involves skill in integrating diversity and inclusion principles in talent development strategies and initiatives.

I had a recent wake up call on this topic because I had just finished a leadership course but failed to create enough discussion on the social unrest that occurred in the summer of 2020. I received a comment to that effect on a feedback report.

Since then, I have gone back and modified my course in several ways to elevate the topics of equity and inclusion. Here are six of the points I have added.

Point 1 – Diversity is an Asset

When you have a mixture of cultures and differing opinions, the team can come up with more creative solutions to problems. The ability to see issues from different angles enhances the quality of dialog as long as all individuals show respect and trust for each other.

At work, I made it a point to promote people so that my team was highly diverse. Of the (roughly 40) supervisors and managers reporting to me, they were 1) more women than men, 2) roughly 30% racially different from me 3) of different age groups and with diverse cultural upbringings. I always enjoyed the diversity of my teams because we were able to see things from different angles. We listened to each other and avoided a monoculture in my area.

In nature, a monoculture is a weakened state. If you plant the same crop on a plot of land year after year, it will become susceptible to disease and produce lower yields.

Point 2 – Silence is being Complicit

Discussions that include individual differences can become uncomfortable, so many leaders tend to avoid them. That is a mistake. If you try to ignore the topics of equity and inclusion, you actually become part of the problem rather than part of the solution.

Dialog is essential because it leads to higher levels of awareness. The most dangerous part of bias is unconscious bias, so it is essential to discuss differences, and be receptive when others point out how you are showing bias.

Point 3 – There is no Fence Anymore

You must take a stand and declare your posture on fairness and equity. It is not possible to sit on the fence and let others argue the fine points of racial injustice, or any other form of prejudice.

Point 4 – Do not say “I Understand”

There is no way that a person from a privileged class can understand what it is like to be from a disadvantaged group. The person from a disadvantaged segment will have endured far more pain and feelings of inadequacy every day of his or her life than you can possibly imagine.
Recognize the emotional load that others carry, but do not patronize by saying “I understand.” You don’t.

Point 5 – Get Comfortable with Being Uncomfortable

Many of the discussions on equity and inclusion will be challenging and difficult. Both sides of any issue will make false steps along the journey to understanding.

Recognize and factor in the difficulty of the challenge.

Point 6 – Don’t Hire with the Idea of Getting Someone to “Fit In.”

It is a mistake to bring in people who are just like the rest of us. Always seek to hire people with differing points of view and backgrounds. Note: that does not mean you should seek to hire people who will be disruptive or abrasive. Rather seek to diversify the points of view for various people on the team.

These are just six points out of thousands that could be discussed, but they do demonstrate that I am trying to address the issue of cultural awareness, equality, and inclusion more consciously in my leadership work.

Robert Whipple is also the author of The TRUST Factor: Advanced Leadership for Professionals, Leading with Trust is like Sailing Downwind, and Trust in Transition: Navigating Organizational Change. Bob consults and speaks on these and other leadership topics. He is CEO of Leadergrow Inc., a company dedicated to growing leaders.