Talent Development 16 Surveys That Work

November 8, 2020

Section 2.8 in the CPTD Certification program for ATD is Evaluating Impact. Section B reads, “Skill in creating data collection tools, for example questionnaires, surveys, and structured interviews.”

For about 5 years, I taught a graduate course called Experimental Design. The course was part of a curriculum leading to a Masters Degree in Organization Learning and Human Resources Development.

In preparation for writing this brief article, I went back and reviewed my slide deck for the course. It was 200 slides long, and many of the slides were heavy with statistical techniques. Obviously, I will need to skim the surface in this short summary article.

In the course, we studied how well-intended surveys often miss the mark and produce bogus results. We also studied the antidotes.

Why Surveys Fail

There are many reasons why surveys fail. I will list some of the more common reasons here and then describe the typical antidotes.

1. Survey too long and complex – one reason for poor data is because people get turned off by too burdensome and too many surveys. When people are angry about too many surveys, they give responses that do not reflect their true feelings.

2. Changes made are not evident – if leaders do not stress that a change is being made because of an employee survey, people will believe their input was ignored. The common misconception sounds like this, “They make us fill out all these ridiculous surveys, but nothing ever changes.”

3. Survey not valid – incorrectly designed surveys often do not measure the thing they are trying to measure. Surveys must be statistically validated to be useful.

4. Survey not reliable – If you would repeat the survey a second time, you would get a different result?

5. Questionable anonymity – If people believe there is some secret way management can find out who said what, then the instrument will not give accurate results. People will respond in ways they think management wants to hear.

6. No clear objective – When people are asked to fill out a survey, they need to know ahead of time why they are being asked to participate and what to expect.

7. Questions not clear – Often the wording of questions leaves people wondering what is really meant by the questions. In this case, you will get guesswork rather than valid data.

8. Leading Questions – Sometimes the way questions are worded leads to skewed data. For example, a political survey might ask, “Are you frustrated by the lies being spread by my opponent?”

Antidotes

I will list the antidotes to the problems in the same order.

1. Make sure your survey is user friendly. Take the survey yourself and ask if you would take the time to do this instrument justice on your most busy day. A good rule of thumb is to be able to fill out the information in less than 10 minutes.

2. Make sure you get back to everyone who responded with the results of the survey. Also, tie all changes made to the survey results, so people are aware of the connection.

3. Test if the survey is valid. The only way you can tell is a survey is measuring what you are trying to measure is to use a statistical analysis of the data. There are five different types of validity (Content, Construct, Concurrent, Criterion, and Predictive). Get help from someone qualified to measure validity. Don’t just wing it.

4. Test if the survey is reliable. This involves trial runs of the survey with different groups under different conditions. The survey needs to produce consistent information to be reliable. Another method is to use a split-half technique. Again, get help if you are not an expert in this area.

5. Insuring anonymity is tricky – The best method I have found to get people to really believe the survey is anonymous is to select a skeptical person from the population to help reduce the data into usable form. The skeptic will let others know that there was no secret means by which management knew who said what.

6. Clarity of Objective – This is a matter of good survey design. It is not just a simple matter of generating some questions and handing out the survey. It must be done with care and solid logic. The way the survey is introduced (typically with an email or letter) is critical. Otherwise you have garbage in garbage out.

7. Test your questions for understandability – This is usually done in the final design phase. You ask people how they interpreted the question. It is not uncommon for many people to be baffled by the wording. Check it out carefully.

8. Avoid leading questions – do not telegraph the requested answer by the way a question is worded. Like don’t ask “Would you like a yummy pizza?”

These areas are general, but they do show how generating a survey is not so simple as most people believe. If a survey is going to generate valuable information for the organization, it needs to be constructed well and administered correctly.

Bob Whipple, MBA, CPTD, is a consultant, trainer, speaker, and author in the areas of leadership and trust. He is the author of: The Trust Factor: Advanced Leadership for Professionals, Understanding E-Body Language: Building Trust Online, Leading with Trust is Like Sailing Downwind, and Trust in Transition: Navigating Organizational Change. Bob has many years as a senior executive with a Fortune 500 Company and with non-profit organizations.

1 Comment | Candor creates trust, Communications, Enabling Actions, HR, Interviews, Organization, Transparency, Trust, Trust and breach of trust | Tagged: , , , , , , , , , , , , , , , , , , , , , , , , , , , | Permalink
Posted by trustambassador

Clean Out Your Clutter

October 7, 2012

Most of us need a reminder once in a while to clean out our clutter. This article is about the topic of clutter in various parts of our lives and how we need to keep it from building up. If you have the personal discipline never to have a cluttered desk or workbench, stop reading and give yourself a medal for being so organized. The rest of us will pick apart the concept of clutter and find some coping mechanisms.

First, it would be good to identify exactly what clutter is. Clutter is that set of things (or ideas) that once served a useful purpose in our lives, but now are no longer useful. For example, if you look in your cupboard or pantry, you are likely to find some condiments or food items that expired over a year ago. If you think about it, these items are not safe to eat, and you will never use them. They remain on the shelf taking up valuable space, but they will not be consumed by you or anyone else. To throw them out would be the smart thing to do, but we continue to work around these artifacts and simply refuse to do what is obviously right.

Look in your closet. There are probably clothes in there that you intellectually know you will never wear again. Your body shape is not going to return to the size that would allow these items ever to be wearable by you, and you cannot legitimately give them to someone else. Yet, year after year, they remain in your closet taking up space and leaving the place a cluttered mess.

Keeping clutter is not just a bad habit for people; it is also a problem for organizations. In any organization, there are procedures and processes that have no current purpose, but we continue to do them out of momentum. They sap energy and time from our current operation, but we fail to stop them. An example might be a daily report that nobody pays any attention to anymore. It may be the ancient Mimeograph supplies in the stationery cabinet. They will sit there for decades in their unopened boxes, even though the Mimeograph machine was tossed out in 1975. You probably have ink cartridges or toner for printers that no longer exist in your office. The list goes on and on. Spare parts for machines we no longer own; old Christmas decorations; trade show posters collecting dust; a broken vase; these are all items that can be found in most office store rooms, and there are thousands of other examples if you think about it.

There is also mental clutter that clogs our brains with old ideas that do not apply in our current world, or maybe never did apply very well. For example, many managers still practice a “command and control” philosophy, clinging to the ancient belief that in order to get things done they need to scare people into compliance. Managers may believe that to “motivate” people, all they need to do is add some extrinsic goodies like t-shirts, pizza parties, or hat days. Those ideas went out with Herzberg’s Two Factor Theory over 60 years ago, yet every day I still see managers trying to motivate people with extrinsic rewards.

How can we get a handle on clutter and remove much of it from our lives? To start with, we need to be able to actually see the clutter in a different form than we usually do. I think one way is to do campaigns where we remove every single bottle of lotion or shampoo from a cupboard and then only replace those items we are likely to use in the future. You can do one cupboard or closet a day and have an entire room cleaned up in a week. You can set aside three consecutive days on your calendar to do the garage or attic. Just be sure to have a dumpster handy and a wheel barrow to carry the junk out to it.

With the office procedures, why not have a “clean out” day where we challenge all of the rituals and things that take up our time. There is a formal process for this called “work out.” The idea is to take the useless work out of our processes so we can spend our precious time only on the things that matter, thus de-cluttering our processes. The concepts of lean thinking and “5S” principles are particularly helpful for these clean out activities.

The benefits of cleaning out your clutter is that you make room to put the vital few things for your current existence front and center where they are readily available and not hidden among the piles of useless garbage that has built up over the years. In the event that you need to downsize your environment in the future (and we all eventually do) you will need to throw out the clutter anyway, why not start now and enjoy some more usable resources today.

6 Comments | Communications, Enabling Actions, leadership, Organization, Teamwork, Trust | Tagged: , , , , , , , , | Permalink
Posted by trustambassador