André Bächtiger^a and Marina Lindell^b
aInstitute of Social Sciences, University of Stuttgart, Stuttgart, Germany
^bSocial Science Research Institute (Samforsk), Åbo Akademi University, Turku, Finland

The goal of this article is to better know high deliberative quality when we see it, whereby we use a best practice case to “benchmark” deliberative quality across various sites. To date, empirical research on deliberation has not come up with “threshold” values for high and low deliberative quality. In a review of research on deliberative democracy based on the Discourse Quality Index (DQI; Steiner, Bächtiger, Spörndli, & Steenbergen, 2004), Dryzek (2007, p. 244) notes: “In applying the discourse quality index, it is hard to say whether the deliberation in any of the cases analyzed is actually good enough by any theoretical standards. The index is just a comparative measure.” Yet absolute standards for deliberative quality may be exceedingly difficult to obtain since it would require that we set cut values for the various dimensions of deliberative quality. For instance, we would need to know how often actors must provide qualified or sophisticated justification in order to classify a complete debate as “reasoned.”

Therefore, we draw on research on democratization and democratic quality and identify “best practice” cases which can serve as “benchmarks” to judge high deliberative quality (e.g., Bühlmann et al. 2012). We compare the deliberative quality in the context of two sites – parliament and a deliberative mini-public – using the same measurement instrument, the Discourse Quality Index (DQI; Steiner et al., 2004) and focusing on the same indicators, namely justification rationality, common good orientation, and respect toward counterarguments.

With regard to parliament, we use, first, a “best-practice” case, namely a Swiss debate on a new language article in the Constitution in the 1990s. This debate was conducted under “optimal institutional” conditions in politics – political actors deliberating in a committee behind closed doors in a consensus system on a low polarized issue including a motivation to find a solution that all linguistic groups could accept (Pedrini, Bächtiger, & Steenbergen, 2013).

Second, we single out four committee sessions on the language article that independent coders deemed as approaching deliberative ideals (including one session that was judged as “truly deliberative”; Bächtiger, Pedrini, & Ryser, 2010).

Third, we also consider a wide range of parliamentary debates coded in the project of Steiner et al. (2004); this involves a purposive sample of plenary and committee debates from the late 1980s and 1990s in Switzerland, Germany, the U.S. and the UK. The debates involved polarized issues such as socio-economic policies, moral issues such as abortion, and less polarized issues such as animal welfare. With regard to deliberative mini-publics, we focus on the “gold standard,” namely a deliberative poll. Deliberative polls provide supportive institutional conditions, which means that citizens get balanced information material, experts answer citizens’ questions, and facilitators ensure that small group discussions keep to the topic and are focused on all the arguments. Specifically, we draw on data from thirteen discussion groups conducted as part of Europolis, a transnational deliberative poll carried out in Brussels in May 2009 on the topic of (third-country) migration (Gerber, Bächtiger, Shikano, Reber, & Rohr, in press). In a way, Europolis could also form a “best practice case” for citizen deliberation since it provided participants with highly supportive institutional conditions for deliberative action; yet given the dearth of coded data on deliberative quality in mini-publics, we use Europolis only as a contrast case.

Table 1 displays major differences across institutional and issue conditions as well as sites. We first see that the Swiss committee debates on language policy excel relative to all other debates. This is especially true if one considers the four debate sequences which external coders identified as instances of “good” or even “true” deliberation. Here, the total amount of sophisticated justifications is 68 percent, common-good statements are at 31 percent, and common good statements and explicit respect is at 39%. This is much higher compared to the average of parliamentary debates analyzed in Steiner et al. (2004), where the respective scores were 39% (sophisticated justifications), 15% (common good orientation) and 13% (explicit respect).

The gap between the “best practice” case and the Europolis discussions among European citizens is quite large as well: in the latter, sophisticated justifications hover around 10%, whereas common good statements and explicit respect are at about 16%. But the Europolis discussions reveal an intriguing pattern: while justification rationality is lower than the average of all coded parliamentary debates, respect levels display a reversed trend: disrespectful speech is far less common in Europolis (4% vs. 24% in parliamentary debates), and explicit respect is also slightly higher in the Europolis discussions (16% vs. 12%). Finally, with regard to common good orientation, there are few differences between Europolis and parliamentary debates (including the “best practice” case), with common good orientation hovering around 15-20%. Overall, Table 1 not only shows the importance of sites for deliberative quality, it also provides us with some “benchmarks” for assessing high quality deliberation.

Table 1 (part 1): Deliberative quality in different sites

Table 1 (part 2): Deliberative quality in different sites

Notes: Numbers are percentages; * N = 3202; Data from Steiner et al. (2004), Bächtiger et al. (2010) and Gerber et al. (in press).

However, an important question is whether different sites and especially citizens and elites should be assessed by the same set of quality standards. As Mansbridge (1999) writes: “the larger deliberative system … should be judged by much the same standards as classic deliberation in assemblies. Those standards must be loosened to accommodate the more informal character of the nongovernmental parts of the deliberative system, but in this loosening they do not lose their character.” Indeed, we cannot expect citizens to reach the same rationality levels than professional politicians who have both the time and material resources to engage intensively with an issue at hand. But the fact that Europolis discussions score higher on respect than average parliamentary debates is intriguing, also from a normative point of view: it seems that citizens realize ethical goals of deliberation (respect) which are not adequately fulfilled in the representative realm of politics (see also Pedrini, 2014).  

However, the figures presented in Table 1 do not take into account the deliberative functions of the various sites (Bächtiger & Beste, in press). From this perspective, the finding that parliamentary debates on average display fairly low levels on the various deliberative indicators may not be a normative deficiency: a contextual perspective would emphasize that public parliamentary debate is not about reflection or respect, but about “robust reasoning” where actors provide extensive elaboration on why ´x´ leads to ´y´. The fact that sophisticated justification rationality is quite substantial (more than 30%) in legislatures (even in polarized plenary debates) is an indication that this function is “in action.” By the same token, the fact that deliberative quality is much higher behind closed doors is desirable from a functional perspective as well. When pressures of representation are reduced and governing logics (including demands for agreement) set it, we can expect deliberative virtues – such as listening and respect – to flourish more.

To conclude, the “benchmarking” of deliberative quality is still in its infancy; we not only need (comparable) data from many more sites (and over time), we also need novel theoretical thinking of what high and low quality deliberation means in the context of various sites of a democratic system. Nonetheless, our article provides a first (and modest) attempt to better identify instances of high deliberative quality when we see them in the real world.

Notes on contributors

André Bächtiger (PhD, Bern) is chair of Political Theory at the University of Stuttgart, Germany. His research focuses on the challenges of mapping and measuring deliberation both in representative institutions and mini-publics.

Marina Lindell (PhD, Åbo Akademi) is a postdoctoral researcher at Åbo Akademi University, Finland. Her scholarship focuses on deliberation, opinion formation, and social psychology.

References

Bächtiger, A., & Beste, S. (in press). Deliberative citizens, (non-)deliberative politicians, and what that means for democracy: A rejoinder to false distinctions and outdated assumptions in deliberative and democratic thinking. Daedalus.

Bächtiger, A., Pedrini, S., & Ryser, M. (2010). Prozessanalyse politischer Entscheidungen: Deliberative Standards, Diskurstypen und Sequenzialisierung. In J. Behnke, T. Bräuninger, & S. Shikano (Eds.), Schwerpunkt Neuere Entwicklungen des Konzepts der Rationalität und ihre Anwendungen (pp. 193–226). Wiesbaden, Germany: VS Verlag für Sozialwissenschaften.

Bühlmann, M., Merkel, W., Müller, L., & Weßels, B. (2012). The Democracy Barometer: A new instrument to measure the quality of democracy and its potential for comparative research. European Political Science, 11(4), 519–536. https://doi.org/10.1057/eps.2011.46

Dryzek, J. S. (2007). Theory, evidence, and the tasks of deliberation. In S. W. Rosenberg (Ed.), Deliberation, participation and democracy: Can the people govern? (pp. 237–250). New York, NY: Palgrave Macmillan.

Gerber, M., Bächtiger, A., Shikano, S., Reber, S., & Rohr, S. (in press). Deliberative abilities and influence in a transnational deliberative poll (EuroPolis). British Journal of Political Science. https://doi.org/10.1017/S0007123416000144

Mansbridge, J. (1999). Everyday talk in the deliberative system. In S. Macedo (Ed.), Deliberative politics: Essays on democracy and disagreement (pp. 211–239). Oxford, England: Oxford University Press.

Pedrini, S., Bächtiger, A., & Steenbergen, M. R. (2013). Deliberative inclusion of minorities: Patterns of reciprocity among linguistic groups in Switzerland. European Political Science Review, 5(3), 483–512. https://doi.org/10.1017/S1755773912000239

Pedrini, S. (2014). Deliberative capacity in the political and civic sphere. Swiss Political Science Review, 20(2), 263–286. https://doi.org/10.1111/spsr.12074

Steiner, J., Bächtiger, A., Spörndli, M., & Steenbergen, M. R. (2004). Deliberative politics in action: Analyzing parliamentary discourse. Cambridge, England: Cambridge University Press.

Valentin Gold^a, Annette Hautli-Janisz^b, Katharina Holzinger^c, and Mennatallah El-Assady^d
aCenter of Methods in Social Sciences, University of Göttingen, Göttingen, Germany
^bDepartment of Linguistics, University of Konstanz, Konstanz, Germany
^cDepartment of Politics and Public Administration, University of Konstanz, Konstanz, Germany
^dDepartment of Computer and Information Science, University of Konstanz, Konstanz, Germany

The VisArgue project is an interdisciplinary collaboration of political science, linguistics and information science at the University of Konstanz (principal investigators: Miriam Butt, Katharina Holzinger, Daniel Keim), funded by the “eHumanities” research initiative of the German Federal Ministry of Education and Research (BMBF). Its main goal was the development of an automated measurement tool for the quality of deliberative communication.

The project has its theoretical foundations within the theory of deliberative democracy. Deliberative democracy promotes a form of democracy that is based on a consensus-oriented discourse instead of majority decisions and representation (Habermas, 1981; Gutmann & Thompson, 1996). At its core, the communication should be inclusive and based on extensive reasoning. Following Habermas, stakeholders participating in the discourse should be willing to adhere to “the unforced force of the better argument”.

While the empirical turn in deliberation research (Dryzek, 2000; Bächtiger, Niemeyer, Neblo, Steenbergen, & Steiner, 2010) has led to an increased understanding of deliberative decision-making, previous approaches rely on the application of manual coding schemes determining the deliberative quality within discourses (e.g. Steenbergen, Bächtiger, Spörndli, & Steiner, 2003; Wessler & Rinke, 2014). Only recently, the computational turn allows to analyze large quantities of text automatically. The VisArgue-project contributes to this turn by determining deliberative communication with automated means. We propose a novel linguistic and visual analytics toolbox that allows us to study deliberative communication in all its diverse aspects. This implies three questions:

• Which factors determine the deliberative quality of political discourse?
• How can we automatically detect and analyze these factors at the linguistic level?
• How can we use visualization in order to make patterns of deliberative communication interpretable in large amounts of text?

Based on several variants of deliberative democracy (cf. Thompson, 2008), we distinguish four dimensions that are crucial for the concept of deliberative communication (Gold et al., 2015): Participation and Inclusion, Atmosphere and Respect, Argumentation and Justification, and Accommodation and Persuasion. These four dimensions are further subdivided in different subdimensions belonging to similar theoretical concepts.

The VisArgue framework is designed to support scholars of deliberation communication in various ways. First, we propose a wide range of visual analytics tools for the interactive exploration of discourses, e.g. Lexical Episode Plots (Gold, El-Assady, & Rohrdantz, 2015) and Conversation Topic Analysis (El-Assady, Gold, Acevedo, Collins, & Keim, 2016). For a demonstration of these tools, please see http://presidential-debates.dbvis.de. Second, we propose a computational linguistic parsing system annotating various measures for the four deliberative dimensions. These measures result from the application of natural language processing tools, unsupervised content extractions, dictionary applications, and statistical analyses.

As the justification of claims is the core of the concept of deliberation, our central indicators are discourse connectors that identify causal or contrastive argumentation. We distinguish causal (weil ‘because’), conditional (wenn-dann ‘if-then’) and adversative (aber ‘but’) connectors and are able to automatically identify the type and scope of argument and even its components (such as reason or conclusion). To arrive at this annotation, several rules have to be identified that allow for the reliable disambiguation of ambiguous connectors and the argumentative phrases (Bögel, Hautli-Janisz, Sulger, & Butt, 2014).

In order to explore and interpret the various measures of deliberative communication, we propose a glyph-based visualization that is based on the annotation system but abstracts from all annotations. In general, glyphs are small iconic representations of information that are used to represent the different occurrences of the same characteristics across units in the data. The glyph shown in Figure 1 mirrors the four dimensions of deliberative communication, with each quadrant representing one dimension. Each subdimension is represented as a row and each annotation within a subdimension is represented as a small tile (the actual measure). In order to show the average length of each utterance (our unit of analysis), we include a small circular icon on the bottom left of the glyph. This is scaled to the length of the underlying utterances, indicating the relative size of its text. Moreover, to facilitate the comparison between glyphs, we account for the different types and strengths of measures by different visual mappings. For instance, the tile “average sentence complexity” in Figure 1 is marked with a plus, indicating that this turn is characterized by a higher level of sentence complexity in comparison to the other turns. Likewise, for emotions: The minus signals more negative emotions compared to the other turns.

Figure 1: Deliberation glyph with four dimensions

The glyph-based visualization supports the exploration and interpretation of deliberative communication. By providing the possibility to aggregate the glyphs with respect to different variables, e.g. the position of speakers or over topics, the turns within the discourse can be compared, resulting in conclusions with respect to the degree of deliberation. Figure 2 illustrates the mediation on “Stuttgart 21”, a conflict about re-building the rail infrastructure in Southern Germany. Four political positions are shown: pro- and contra Stuttgart 21, neutral (the mediator), and expert opinions. While the middle column illustrates the complete glyph, the right column only shows causal argumentation (green tiles) constituting of reasons and conclusions.

The overview reveals that the arbitrator provides little argumentation; the experts mostly present reasons; the pro and con parties frequently use both components. From the other measures in the left column, we can conclude that the mediator acts professionally, as he – addressing the public – uses simple and polite language. Furthermore, he interrupts other speakers, keeps them to the topics, often formulates concessions and emphasizes agreements while rarely uttering dissent. The parties to the conflict are similarly polite, but emphasize positive emotions, have a high degree of argumentative speech and add new topics to the table, thereby advancing the dialog.

Figure 2: Deliberation glyphs for the four positions in the S21 arbitration

The VisArgue framework can be applied to foster deliberation analyses. In general, the complete framework of deliberative decision-making comprises of the antecedents for deliberation (input dimension), the dynamics (throughput dimension) of deliberation, and the consequences of deliberation (output dimension). While the VisArgue project contributes to measuring variables at the throughput dimension, for deliberation to be analyzed, it is important to include both antecendents and consequences in the statistical analyses.

Here, we briefly want to describe two of our applications. First, we apply the novel measures to the verbatim minutes of simulated debates in which participants had to take unanimous decision on a contested issue (Gold & Holzinger, 2015). For each speaker, we test the effects of deliberative standards on their perception by other participants. The results show that participants are the less often perceived as constructive speakers the less someone participates. Moreover, less eloquent participants and those participants that more often express opposition are more often seen as having blocked the decision-making process.

Second, we can show how personality traits influence deliberative quality (Gold, 2016). Based on the simulated data, we see that participants scoring high on openness show less adversative argumentation. Likewise, these participants less often express conditions – contrary to participants scoring high on agreeableness. We conclude that while designing political institutions to foster deliberative communication is a good start, we need to adopt a more disaggregated focus disentangling the causal effects on the dynamics of deliberation.

Notes on contributors

Valentin Gold (PhD, Konstanz) is a postdoctoral researcher at the Center of Methods in Social Sciences at the University of Göttingen, Germany. In his research he focuses on text-mining applications and deliberative communication.

Annette Hautli-Janisz (PhD, Konstanz) is a postdoctoral researcher in the Department of Linguistics at the University of Konstanz, Germany. She has worked on various aspects of theoretical and computational linguistics, among them lexical semantics and discourse processing.

Katharina Holzinger (PhD, Augsburg) is Professor for International Politics and Conflict Management at the University of Konstanz, Germany. Her research focuses on deliberative democracy, traditional forms of governance and EU studies.

Mennatallah El-Assady is a PhD candidate in the Department of Computer and Information Science, University of Konstanz, Germany. Her research focuses on information visualization, natural language processing, and data mining.

References

El-Assady, M., Gold, V., Acevedo, C., Collins, C., & Keim, D. (2016). ConToVi: Multi-party conversation exploration using topic-space views. Computer Graphics Forum, 35(3), 431–440. https://doi.org/10.1111/cgf.12919

Bächtiger, A., Niemeyer, S., Neblo, M. A., Steenbergen, M. R., & Steiner, J. (2010). Disentangling diversity in deliberative democracy: Competing theories, their blind spots and complementarities. Journal of Political Philosophy, 18(1), 32–63. https://doi.org/10.1111/j.1467-9760.2009.00342.x

Bögel, T., Hautli-Janisz, A., Sulger, S., & Butt, M. (2014). Automatic detection of causal relations in German multilogs. In Proceedings of the EACL 2014 Workshop on Computational Approaches to Causality in Language (CAtoCL) (pp. 20–27). Gothenburg, Sweden: Association for Computational Linguistics. Retrieved from https://kops.uni-konstanz.de/handle/123456789/29255

Dryzek, J. S. (2000). Deliberative democracy and beyond: Liberals, critics, contestations. Oxford, England: Oxford University Press.

Gold, V. (2016). Personality traits and debate quality. Presented at the Symposium ”Exploring ignorance: Acquisition, selection and processing of information”, Konstanz, Germany.

Gold, V., El-Assady, M., Bögel, T., Rohrdantz, C., Butt, M., Holzinger, K., & Keim, D. (2015). Visual linguistic analysis of political discussions: Measuring deliberative quality. Digital Scholarship in the Humanities. https://doi.org/10.1093/llc/fqv033

Gold, V., & Holzinger, K. (2015). An automated text-analysis approach to measuring the quality of deliberative communication. Presented at the Annual Meeting of the American Political Science Association, San Francisco, CA.

Gold, V., Rohrdantz, C., & El-Assady, M. (2015). Exploratory text analysis using lexical episode plots. In E. Bertini, J. Kennedy, & E. Puppo (Eds.), Eurographics Conference on Visualization (EuroVis). The Eurographics Association. https://doi.org/10.2312/eurovisshort.20151130

Gutmann, A., & Thompson, D. F. (1996). Democracy and disagreement: Why moral conflict cannot be avoided in politics, and what should be done about it. Cambridge, MA: Harvard University Press.

Habermas J. (1981). Theorie des kommunikativen Handelns. Frankfurt am Main, Germany: Suhrkamp.

Steenbergen, M. R., Bächtiger, A., Spörndli, M., & Steiner, J. (2003). Measuring political deliberation: A discourse quality index. Comparative European Politics, 1(1), 21–48. https://doi.org/10.1057/palgrave.cep.6110002

Thompson, D. F. (2008). Deliberative democratic theory and empirical political science. Annual Review of Political Science, 11, 497–520. https://doi.org/10.1146/annurev.polisci.11.081306.070555

Wessler, H., & Rinke, E. M. (2014). Deliberative performance of television news in three types of democracy: Insights from the United States, Germany, and Russia. Journal of Communication, 64(5), 827–851. https://doi.org/10.1111/jcom.12115