Designing surveys is really hard, but we need to do it anyway – By Jason Shriner

DSC_0792

In 2017, we conducted our first patron engagement survey based on a survey that the National Recreation and Parks Association conducted nationwide. You can review the reports on our website by clicking here.

At first, the process was fairly easy. We wanted to see how we compared to the rest of the country, so we asked nearly all of the same questions that NRPA did and asked a few additional questions that we were curious about too. The results have been really insightful.

What I didn’t realize was how difficult it is to design an effective survey. After all, the first year all of our questions were basically written for us. However, my team and I realized that we needed to reword questions, include additional answers, rewrite options, and ask more questions. I began to realize how much effort goes into designing a survey.

This year, we have been designing the survey with even more intent. We want to know more about the population we serve and to make sure we’re serving a diverse community that represents the City and the Greater Prince William region. We also want to make sure that we’re developing programs and training our staff to best serve populations that are visiting our facilities.

This year we’ve revamped some of our demographics questions. With gender identity becoming more of a discussion, we realized we need to be more cognizant of language and people who may identify outside of the cisgender binary. We’re also asking one simple question on sexual orientation. Why “one simple question” when sexual identity is extremely complex? I recently watched a webinar by the National LGBTQ Institute on IPV (Intimate Partner Violence) where they discussed collecting demographic information on gender identity and sexual orientation. The main focus was on collecting information for providing services, hospital in-take forms, and to demonstrate to funders their money is going where the organizations said it would – but the information was still invaluable for us. One question they asked us to consider was, “If you don’t plan to use the information, why are you asking for it?” They framed it as an ethics question as demographic questions can feel invasive for some people.

With that question in mind, I asked myself, “What information do we need and why do we need it?” With gender identity, we ask this question to make sure we aren’t seeing one gender disproportionately responding to the survey over another. That doesn’t necessarily mean 50% man and 50% woman since we might not necessarily see that kind of participation at our facilities anecdotally. We also don’t want to make the experience othering for people who do not identify as man or woman – so this year we’ve removed other as an option and replaced it with “something else” and we’ve also added gender nonconforming and transgender as options and made the gender identity question allow a “check all that apply” answer.

With the LGBTQ question, the concern about listing out specific identities became more about erasing identities rather than othering people. As I started to write out the various sexual identities I knew, I realized how complicated it was quickly becoming. So I asked myself, “What am I trying to ask and why do we need it?” Since sexual orientation specifically doesn’t have a strong impact on how we develop our programming, I’m really just trying to find out – are we serving anybody from the LGBTQ community? This would be different if I worked at an LGBTQ community center, where I would need to make sure that the organization was serving a broad range of the community and it would make sense to ask for specific identities. So this time we’re planning to ask the question, “Do you identify as LGBTQ (lesbian, gay, bisexual, transgender, queer, or questioning)?” and you all will be able to answer yes, no, or prefer not to say. Depending on the response, it may be worth asking for specifics next year – but, again, only if we plan to use that information in a meaningful way.

In this slide from the webinar by the National LGBTQ IPV, they show the complexities faced when attempting to ask people their sexual identity quantitatively or qualitatively. All identities, including ethnicity and disability, are extremely personal, so it’s not a surprise how broad the response can be with a qualitative question. (Credit: http://lgbtqipv.org/)

As you can already begin to see, demographics questions alone are so complicated. How do you ask the question? What are you trying to find out? How do you remain sensitive to all identities? How do you keep the data manageable for analysis? How do I avoid asking for information that we aren’t going to use?

These questions became especially difficult when writing the two questions asking about disability. As somebody who is not part of the disability community, I’m going to innately have an ableist approach to the question. The more I researched about the topic, the more complex it became. Do we ask about mental wellness? Do we ask about chronic illnesses? If we don’t specifically provide those options, would they choose another option? Would they feel erased? Othered? These questions are still being developed, but we’re putting a great deal of consideration into them because we need to know who we are serving – or not serving – and how we can do better to serve the disability community.

There are so many options when designing a survey, especially online, including order of questions, where page breaks occur, even details like color and fonts. How do we make sure people who begin the survey finish the survey? How do we set up the options so we aren’t accidentally encouraging people to pick one option over the other (or give the impression of a hierarchy especially when it comes to demographics questions)? How do we set up the options so the data visualization gets automatically generated properly? How do I make sure everybody taking the survey interprets the questions and answers the same way I do as the designer?

Is there any wonder why people who design surveys for a living can have doctorates?

And we haven’t even begun to discuss conducting the survey. And we won’t. That’s a blog for another day.

The point for this blog is that we are doing our best to become more and more diligent about creating a proper survey. None of us in the Department is an expert at surveys or statistics. But that doesn’t mean that we shouldn’t do everything we can to make the survey the best tool for our Department as possible. The answers provided in the survey could help us redirect funds for programs, research trainings for service gaps, identify where we need to build partnerships, reevaluate marketing, hire new staff, apply for and secure additional funding, and so much more. The survey report can be a powerful tool in helping us serve the community, but only if the community takes the survey. And you all will only take the survey if the survey is accessible, respectful, and trustworthy.

So this January, look for the 2018 Patron Engagement Survey and please take it. We’re counting on your participation!

Jason Shriner is the Marketing Manager for the City of Manassas Park, Department of Parks and Recreation. He can be reached at 703-335-8872 or via email at J.Shriner@manassasparkva.gov.

Leave a comment