Logistical Challenges of Sampling an Emerging Field

The field of dialogue and deliberation is growing, contested, and overlaps with other fields such as facilitation, organizational development, community organizing, conflict resolution, and many others. Due to the difficulty of defining a population of practitioners who work and volunteer in dialogue and deliberation, this survey used a non-random, self-selected sample of respondents who identified themselves as dialogue and deliberation practitioners, solicited from over twenty practitioner listservs, online discussion forums, and social networking groups focused on dialogue and deliberation methods. For the purposes of academic research, such a self-selected sample is inevitably biased toward frequent internet users and those centrally engaged in online discussions within the field, which makes precise generalizations about the larger field or error estimation on the basis of our survey data impossible.

Nevertheless, such a survey is useful inasmuch as there are substantial challenges in attempting to conduct a random sample of organizations in an emerging field, let alone of practitioners. Each method has advantages and disadvantages; we believe the extensive posting of our survey invitation to as many online communities where dialogue and deliberation practice is discussed as possible had the most advantages for our purpose, which was to learn more about the larger field of practitioners engaged in dialogue and deliberation facilitation in the United States.

Why not a random sample? Wouldn’t a random sample of the membership list of the National Coalition for Dialogue and Deliberation produce a representative sample of D&D practitioners?

It’s important to recognize that many online surveys are conducted by D&D organizations of their own memberships for research and development purposes. These organizations are generous in sharing their findings with their memberships and the public, so survey data on organization members is generally readily available– and an extremely useful point of comparison with the sample we did obtain.

For this project, constructing a random sample would have forced us to rely on non-random sampling frames such as organizational membership rolls, which to some extent would have duplicated existing organizational research. Relying on a random sample of the NCDD membership rolls would have allowed us to generalize to NCDD members– but not much further. Most organizations have membership fees, so such lists are biased toward those doing paid work in the field. While 94% of our sample of U.S. professionals belonged to at least one organization, less than 60% are members of NCDD.

We believe that the online listservs maintained by these organizations– typically much larger than their current membership rolls– better represent the larger field and, when combined, can give a better picture of the diversity of dialogue and deliberation as it is currently practiced in the U.S. Our non-random sampling method gathered information from those self-identified practitioners who would have been overlooked by the researchers, and also includes those who are less centrally engaged in professional associations in the field. Nearly 30% of our U.S. sample identified themselves as volunteers or non-professional practitioners.

Why sample dialogue and deliberation practitioners instead of dialogue and deliberation organizations?

This project sought to understand dialogue and deliberation from the unique perspective of practitioners. Existing survey research sampled the field of organizations sponsoring deliberation in 1998, based on a mailing list compiled by Jacobs, Cook, and Delli Carpini from media coverage, the internet, foundation lists, and the International Association of Public Participation membership rolls. Our survey distinguishes between organizations sponsoring dialogue and deliberation processes (such as local cause-based non-profits, called “sponsors” or “clients” in the survey) and D&D-focused organizations whose “cause” is promotion of a D&D method or facilitation of that method on behalf of sponsors or clients. Rather than sampling D&D organizations, our sample includes both practitioners who work within D&D organizations as staff members or employees, in addition to practitioners who provide facilitation services to sponsors as independent consultants.

Developing a list of well-known dialogue and deliberation organizations would be relatively straightforward, but a sample using representatives from these organizations would not provide a comprehensive perspective on the full range of practice identified by Caroline Lee in her ethnographic work. To wit, many professional practitioners also provide volunteer facilitation services to organizations with which they are not affiliated (over 55% of the U.S. professionals in our sample), many practitioners have independent consultancies without an internet presence, many sole practitioners use multiple organizations to conduct different aspects of their work, and large deliberation organizations have a role in diverse projects and processes all over the country that typically use subcontractors and volunteers. A single respondent in the communications department of these organizations may be able to give basic information about these projects but can not give ground-level observations about these processes. As such, by sampling practitioners who facilitate processes for a variety of clients, in a variety of capacities, and across a variety of organizational settings, we can understand better the full range of dialogue and deliberation processes facilitated in the U.S.– beyond those that are formally organized by prominent organizations and receive press attention.

Accounting for Sampling Bias

A number of measures used in the survey design help us to judge the effect of potential sources of bias, including survey questions about respondents’ organizational affiliations, how they received an invitation (which allows us to calculate response rates for individual lists), and their relative engagement level with the survey (which allows us to gauge whether the sample we got overrepresents those who are deeply invested in the field or highly motivated). Using demographic questions, we can also compare the demographics of our survey to the demographics of dialogue and deliberation conference attendees, and to pre-existing organizational surveys, in order to identify particular groups that may be over- or under-represented in our survey data. Survey respondents also provided feedback in open-ended questions on areas where they believed our survey was biased towards facilitators, or against those who practice dialogue and deliberation but do not like that terminology, for example.