Meeting Report: Theoretical Aspects of Crowd Sourced ODR
July 8, 2009, hosted by the Berkman Center for Internet & Society’s Law Lab
I. Workshop Goals
The meeting was convened by the Berkman Center’s Law Lab project and eCitizen Foundation, a nonprofit focused on improving civic participation with online tools. eCitizen sees a clear need for improved dispute resolution systems in civic engagement settings, particularly in online government dialogue platforms. The purpose of the workshop was to have an initial conversation regarding the research efforts and development approaches of each organization on the topic of “crowdsourced online dispute resolution” (cODR). A key goal for the workshop was to discuss overlapping research interests and potential points of synergy between the Berkman Center and eCitizen, particularly as each institution continues to focus on online dispute resolution in the broader contexts of public policy debates, online cooperation, digital institutions, and initiatives in open government.
II. Participants
The meeting brought together a diverse group of participants with equally diverse approaches to dispute resolution. The Berkman Center’s Law Lab team was led by Executive Director and Law Lab co-Principal Investigator Urs Gasser. Berkman participants also included Chris Bavitz (of the Center’s Cyberlaw Clinic); Joey Mornin, a lead researcher on the Law Lab’s cODR research track; Caroline Nolan, Project Coordinator for the Law Lab; and Chris Peterson, a Berkman Center researcher involved with various Law Lab projects. eCitizen was represented by Chief Policy Officer Dazza Greenwood and Chief Technology Officer Daniel Bennett. Both helped shape the day’s agenda, presented the dispute resolution prototype, and updated the group on eCitizen’s work and progress to date.
Participants made insightful contributions throughout the day. Ethan Katsh, a leading cyberlaw scholar and online dispute resolution expert, offered general insight into the field's current needs and challenges; Beth Noveck and Robynn Sturm of the White House Office of Science & Technology joined by phone to comment on dispute resolution in the Open Government Initiative; and John Clippinger of Berkman's Law Lab added his expertise on applying cooperative models to online legal scenarios.
Additional attendees included Scott David, of K&L Gates, who commented on the distinction between immersive versus augmented experiences and asked the group to consider how the blurring of the two might impact approaches to online dispute resolution systems. Michael Turillo, of the Spencer Trask Company, pointed to his experience on “Cybersettle,” an online accelerated dispute resolution system, as potentially an interesting model to draw from. He cited embedded factualness and collective understandings of what constitutes “valid” building blocks for successful ODR systems.
Diane Mueller, VP, XBRL Development at JustSystems, eCitizen’s IT development partner, helped the group move beyond thinking of designing a system based on rules, and challenged them to consider the implication of the actual content and presentation of the information in a posting. Her colleague, David Evans, President, CEO, and Chief Scientist at JustSystems, cited the issue of “inter-rater reliability” as a key element of a successful ODR model: How can we ensure a high percentage of agreement between “judges”of a potential dispute? In her closing remarks, Stephanie Zierten, Deputy General Counsel, Information Technology Division, for the state of Massachusetts, expressed her interest in applying cODR research concepts and prototyping efforts to a variety of government contexts at both the federal and state level.
III. The Sessions
A. Conceptual Questions in cODR
Urs Gasser began the meeting by outlining the current state of play in crowdsourced online dispute resolution. He first discussed the two central ideas driving Berkman's research thus far: “crowdsourcing” and “ODR.” He then mapped the conceptual issues that potentially arise when seeking to merge the two. His remarks emphasized the need to take a step back—engage in conversations with experts, survey wide-ranging existing literature, and understand current mechanisms—in order to understand better what we know across this landscape. What are the issues that surface across a diverse range of concepts, contexts, methods, and stakeholders? How might the theoretical inquiry shape the design of future mechanisms and institutions?
E. Project Plan 1.0
Berkman researcher Joey Mornin picked up on some of the key themes of Urs’ presentation, inviting the group to consider what sorts of use cases could be instructive, and potentially resolved, with a crowdsourced ODR mechanism. The group also discussed how the needs of each use case might translate into technical features and functionalities for a future platform. Several additional questions emerged: What current models can best inform our research efforts? Can an online platform address both online and offline disputes, taking into account both public and private considerations? How can flexibility and scalability be embedded in such a tool or process?
Play "Joey Mornin, Researcher, Berkman Center for Internet & Society"
B. Discussion: Who’s in the Crowd?
A roundtable discussion followed the first session. Ethan Katsh raised the question of the definition of dispute, noting that until about six months ago, this foundational issue had rarely been necessary to define in the field of ODR. Much of the rest of the discussion centered on the contours and makeup of the “crowd”—in terms of communities, behavioral patterns, motivations, norms, and so on.
C. Video Presentation: A cODR Platform Prototype
David Colarusso, a volunteer software developer from MIT, gave a video demonstration of his prototype of a crowdsourced ODR platform, titled “the Moderation Engine”. He emphasized that this is a very early stage prototype, not to be taken as a definitive direction for the project. The demonstration led to a deeper conversation regarding how online systems might be designed to encourage cooperation and collective action. How might conceptual research inform efforts to explore novel broad-based ODR solutions and concepts?
D. Current State of Play: Government Dialogue Platforms
The conversation then turned to government dialogue platforms, with Beth Noveck, Deputy Chief Technology Officer for Open Government and Robynn Sturm, Assistant Deputy Technology Officer for Open Government, joining the group by phone. Their comments touched on the origins and drive behind the Open Government Initiative and emphasized early, transitional, and experimental nature of the platforms currently being road tested by the Obama Administration.
Despite relatively low participation levels and the relatively high number of off-topic comments, the White House is learning much about how to improve the online conversation. With the need for openness and transparency widely acknowledged, they challenged the workshop group to consider—and participate in— the process of improving tools, practices, techniques, and approaches. What are scalable and accessible models of practice? A key area of future development is to build an efficient, replicable method of dispute resolution.
Play "Robynn Sturm, Assistant Deputy Technology Officer for Open Government"
E. Project Plan 1.0
The discussion then turned to members of the eCitizen Foundation, led by Daniel Bennett and Dazza Greenwood, and focused on the technical questions sparked by the creation of a third-party crowdsourced ODR platform that reasonably address some of the issues raised throughout the morning.
The remainder of the meeting consisted mainly of a freeform discussion of topics in cooperation and dispute resolution. The group considered how the conceptual questions regarding crowdsourced online dispute resolution shape our approaches to large-scale, open public dialogues in governmental programs. eCitizen outlined some of the key functionality features they were considering, and the group briefly discussed what the design and implementation process might look like.
F. Next Steps and Key Links
For the final session, Dazza Greenwood of eCitizen and Urs Gasser of the Berkman Center offered their thoughts on next steps; where their research and development tracks were headed; and how their common interests, shaped by ongoing dialogue, could inform the end products.
As such, eCitizen will continue its work on the specific case of government dialogue platforms, in the context of their broader work with online civic engagement. Berkman plans to formalize “crowdsourced online dispute resolution” as a research project, under the umbrella of the “Digital Institutions” project at the Law Lab. Berkman will also continue its effort to examine more closely the conceptual issues involved in merging crowdsourcing and ODR. This research will lead to a paper detailing these issues, with a set of design recommendations for the development and implementation of a crowdsourced online dispute resolution platform.
The paper will make use of empirical data from a variety of use cases, perhaps including government dialogue platforms, in collaboration with eCitizen. Berkman will also consider aspects of two Berkman projects, StopBadware.org and the Global Network Initiative, as potential use cases. The effort will ultimately develop a deeper understanding—as well as practical knowledge—of how cooperative models can improve the efficiency and effectiveness of online dispute resolution. Conversations, workshops, and future collaborations with organizations like the eCitizen Foundation will lead to practical implementations of these ideas, as well as opportunities for further empirical research.
