Fall, 2016 (Issue 26.1–2)

“Benchmarking” Deliberative Quality across Sites

André Bächtigera and Marina Lindellb
aInstitute of Social Sciences, University of Stuttgart, Stuttgart, Germany
bSocial Science Research Institute (Samforsk), Åbo Akademi University, Turku, Finland

The goal of this article is to better know high deliberative quality when we see it, whereby we use a best practice case to “benchmark” deliberative quality across various sites. To date, empirical research on deliberation has not come up with “threshold” values for high and low deliberative quality. In a review of research on deliberative democracy based on the Discourse Quality Index (DQI; Steiner, Bächtiger, Spörndli, & Steenbergen, 2004), Dryzek (2007, p. 244) notes: “In applying the discourse quality index, it is hard to say whether the deliberation in any of the cases analyzed is actually good enough by any theoretical standards. The index is just a comparative measure.” Yet absolute standards for deliberative quality may be exceedingly difficult to obtain since it would require that we set cut values for the various dimensions of deliberative quality. For instance, we would need to know how often actors must provide qualified or sophisticated justification in order to classify a complete debate as “reasoned.”

Therefore, we draw on research on democratization and democratic quality and identify “best practice” cases which can serve as “benchmarks” to judge high deliberative quality (e.g., Bühlmann et al. 2012). We compare the deliberative quality in the context of two sites – parliament and a deliberative mini-public – using the same measurement instrument, the Discourse Quality Index (DQI; Steiner et al., 2004) and focusing on the same indicators, namely justification rationality, common good orientation, and respect toward counterarguments.

With regard to parliament, we use, first, a “best-practice” case, namely a Swiss debate on a new language article in the Constitution in the 1990s. This debate was conducted under “optimal institutional” conditions in politics – political actors deliberating in a committee behind closed doors in a consensus system on a low polarized issue including a motivation to find a solution that all linguistic groups could accept (Pedrini, Bächtiger, & Steenbergen, 2013).

Second, we single out four committee sessions on the language article that independent coders deemed as approaching deliberative ideals (including one session that was judged as “truly deliberative”; Bächtiger, Pedrini, & Ryser, 2010).

Third, we also consider a wide range of parliamentary debates coded in the project of Steiner et al. (2004); this involves a purposive sample of plenary and committee debates from the late 1980s and 1990s in Switzerland, Germany, the U.S. and the UK. The debates involved polarized issues such as socio-economic policies, moral issues such as abortion, and less polarized issues such as animal welfare. With regard to deliberative mini-publics, we focus on the “gold standard,” namely a deliberative poll. Deliberative polls provide supportive institutional conditions, which means that citizens get balanced information material, experts answer citizens’ questions, and facilitators ensure that small group discussions keep to the topic and are focused on all the arguments. Specifically, we draw on data from thirteen discussion groups conducted as part of Europolis, a transnational deliberative poll carried out in Brussels in May 2009 on the topic of (third-country) migration (Gerber, Bächtiger, Shikano, Reber, & Rohr, in press). In a way, Europolis could also form a “best practice case” for citizen deliberation since it provided participants with highly supportive institutional conditions for deliberative action; yet given the dearth of coded data on deliberative quality in mini-publics, we use Europolis only as a contrast case.

Table 1 displays major differences across institutional and issue conditions as well as sites. We first see that the Swiss committee debates on language policy excel relative to all other debates. This is especially true if one considers the four debate sequences which external coders identified as instances of “good” or even “true” deliberation. Here, the total amount of sophisticated justifications is 68 percent, common-good statements are at 31 percent, and common good statements and explicit respect is at 39%. This is much higher compared to the average of parliamentary debates analyzed in Steiner et al. (2004), where the respective scores were 39% (sophisticated justifications), 15% (common good orientation) and 13% (explicit respect).

The gap between the “best practice” case and the Europolis discussions among European citizens is quite large as well: in the latter, sophisticated justifications hover around 10%, whereas common good statements and explicit respect are at about 16%. But the Europolis discussions reveal an intriguing pattern: while justification rationality is lower than the average of all coded parliamentary debates, respect levels display a reversed trend: disrespectful speech is far less common in Europolis (4% vs. 24% in parliamentary debates), and explicit respect is also slightly higher in the Europolis discussions (16% vs. 12%). Finally, with regard to common good orientation, there are few differences between Europolis and parliamentary debates (including the “best practice” case), with common good orientation hovering around 15-20%. Overall, Table 1 not only shows the importance of sites for deliberative quality, it also provides us with some “benchmarks” for assessing high quality deliberation.

Table 1 (part 1): Deliberative quality in different sites

  No Justification Inferior Justification Qualified Justifcation Sophisticated Justification In-depth
Justification
Europolis
(N = 943)
18.5 39.2 32.2 9.2 0.9
Best practice case
(overall)
(N = 158)
10.1 12.7 24.1 45.6 7.6
“Best practice” case
(4 sequences)
(N = 100)
5.0 8.0 19.0 56.0 12.0
Parliamentary Debates (Swiss, US, German, UK)
(N = 4805)
14.7 17.9 29.1 30.1 8.6

Table 1 (part 2): Deliberative quality in different sites

  Group Interests Common Good Interests No Respect Implicit Respect Explicit Respect
Europolis
(N = 943)
8.8 15.7 4.2 79.9 15.9
Best practice case
(overall)
(N = 158)
10.1 22.2 9.5 64.6 26.0
“Best practice” case
(4 sequences)
(N = 100)
12.0 31.0 7.0 54.0 39.0
Parliamentary Debates (Swiss, US, German, UK)
(N = 4805)
5.0* 14.8* 23.6 71.8 11.7

Notes: Numbers are percentages; * N = 3202; Data from Steiner et al. (2004), Bächtiger et al. (2010) and Gerber et al. (in press).

However, an important question is whether different sites and especially citizens and elites should be assessed by the same set of quality standards. As Mansbridge (1999) writes: “the larger deliberative system … should be judged by much the same standards as classic deliberation in assemblies. Those standards must be loosened to accommodate the more informal character of the nongovernmental parts of the deliberative system, but in this loosening they do not lose their character.” Indeed, we cannot expect citizens to reach the same rationality levels than professional politicians who have both the time and material resources to engage intensively with an issue at hand. But the fact that Europolis discussions score higher on respect than average parliamentary debates is intriguing, also from a normative point of view: it seems that citizens realize ethical goals of deliberation (respect) which are not adequately fulfilled in the representative realm of politics (see also Pedrini, 2014).  

However, the figures presented in Table 1 do not take into account the deliberative functions of the various sites (Bächtiger & Beste, in press). From this perspective, the finding that parliamentary debates on average display fairly low levels on the various deliberative indicators may not be a normative deficiency: a contextual perspective would emphasize that public parliamentary debate is not about reflection or respect, but about “robust reasoning” where actors provide extensive elaboration on why ´x´ leads to ´y´. The fact that sophisticated justification rationality is quite substantial (more than 30%) in legislatures (even in polarized plenary debates) is an indication that this function is “in action.” By the same token, the fact that deliberative quality is much higher behind closed doors is desirable from a functional perspective as well. When pressures of representation are reduced and governing logics (including demands for agreement) set it, we can expect deliberative virtues – such as listening and respect – to flourish more.

To conclude, the “benchmarking” of deliberative quality is still in its infancy; we not only need (comparable) data from many more sites (and over time), we also need novel theoretical thinking of what high and low quality deliberation means in the context of various sites of a democratic system. Nonetheless, our article provides a first (and modest) attempt to better identify instances of high deliberative quality when we see them in the real world.

Notes on contributors

André Bächtiger (PhD, Bern) is chair of Political Theory at the University of Stuttgart, Germany. His research focuses on the challenges of mapping and measuring deliberation both in representative institutions and mini-publics.

Marina Lindell (PhD, Åbo Akademi) is a postdoctoral researcher at Åbo Akademi University, Finland. Her scholarship focuses on deliberation, opinion formation, and social psychology.

References

Bächtiger, A., & Beste, S. (in press). Deliberative citizens, (non-)deliberative politicians, and what that means for democracy: A rejoinder to false distinctions and outdated assumptions in deliberative and democratic thinking. Daedalus.

Bächtiger, A., Pedrini, S., & Ryser, M. (2010). Prozessanalyse politischer Entscheidungen: Deliberative Standards, Diskurstypen und Sequenzialisierung. In J. Behnke, T. Bräuninger, & S. Shikano (Eds.), Schwerpunkt Neuere Entwicklungen des Konzepts der Rationalität und ihre Anwendungen (pp. 193–226). Wiesbaden, Germany: VS Verlag für Sozialwissenschaften.

Bühlmann, M., Merkel, W., Müller, L., & Weßels, B. (2012). The Democracy Barometer: A new instrument to measure the quality of democracy and its potential for comparative research. European Political Science, 11(4), 519–536. https://doi.org/10.1057/eps.2011.46

Dryzek, J. S. (2007). Theory, evidence, and the tasks of deliberation. In S. W. Rosenberg (Ed.), Deliberation, participation and democracy: Can the people govern? (pp. 237–250). New York, NY: Palgrave Macmillan.

Gerber, M., Bächtiger, A., Shikano, S., Reber, S., & Rohr, S. (in press). Deliberative abilities and influence in a transnational deliberative poll (EuroPolis). British Journal of Political Science. https://doi.org/10.1017/S0007123416000144

Mansbridge, J. (1999). Everyday talk in the deliberative system. In S. Macedo (Ed.), Deliberative politics: Essays on democracy and disagreement (pp. 211–239). Oxford, England: Oxford University Press.

Pedrini, S., Bächtiger, A., & Steenbergen, M. R. (2013). Deliberative inclusion of minorities: Patterns of reciprocity among linguistic groups in Switzerland. European Political Science Review, 5(3), 483–512. https://doi.org/10.1017/S1755773912000239

Pedrini, S. (2014). Deliberative capacity in the political and civic sphere. Swiss Political Science Review, 20(2), 263–286. https://doi.org/10.1111/spsr.12074

Steiner, J., Bächtiger, A., Spörndli, M., & Steenbergen, M. R. (2004). Deliberative politics in action: Analyzing parliamentary discourse. Cambridge, England: Cambridge University Press.