Binary Governance: Lessons from the GDPR’s Approach to Algorithmic Accountability (Q2401)
Revision as of 12:50, 9 December 2019 by Podehaye (talk | contribs) (Created claim: Property:P126: An impact assessment, in other words, is supposed to be a tripartite conversation between a regulated entity, the regulator, and third parties such as impacted persons or civil society organizations. In t...)
No description defined
| Language | Label | Description | Also known as |
|---|---|---|---|
| English |
Binary Governance: Lessons from the GDPR’s Approach to Algorithmic Accountability
|
No description defined
|
Statements
Secret profiles and decisions made based on secret profiling can threaten personhood and thus dignity by proscribing active individual involvement in the construction of this objectified version of the self.
0 references
the “ ‘data shadows’ . . . threaten to usurp the constitutive authority of the physical self despite their relatively attenuated and often misleading nature”
0 references
Algorithmic decision-making founded on individual profiling limits the choices and, thus, the freedom a person will have.
0 references
Limiting the choices we see—whether by failing to show opportunities or by offering only bad options—limits our freedom to make choices.
0 references
Failing to be transparent about the fact that individuals are being targeted or the reasons why they are targeted itself may threaten autonomy. Secret profiling and decision-making can lead to manipulation. Without knowing how we are being targeted or why, we can be manipulated into making choices that are not autonomous at all.
0 references
Concerns about autonomy and the potential for manipulation, to a great degree, motivated the indignation around Cambridge Analytica’s targeted manipulation of U.S. voters prior to the 2016 election (and motivated the California legislature to enact the California Consumer Privacy Act in 2018).
0 references
the dominant rationale for regulating algorithmic decision-making is an instrumental (or consequentialist) rationale. We should regulate algorithms, this reasoning goes, to prevent the consequences of baked-in bias and discrimination and other kinds of error.
0 references
The instrumental rationale for regulating algorithmic decision-making counsels that regulation should try to correct these problems, often by using systemic accountability mechanisms, such as ex ante technical requirements, audits, or oversight boards, to do so.
0 references
The other two rationales for regulating algorithmic decision-making, however, suggest that systemic oversight is not enough. Both dignitary and justificatory reasoning point towards including individual rights.
0 references
The dignitary argument posits that an individual human being should be respected as a whole, free person. Being subjected to algorithmic decision-making threatens individuals’ personhood by objectifying them. Objectification defeats autonomy: the freedom to make choices, be offered opportunities, or otherwise move freely through the world.
0 references
The first [version of the dignitary argument], largely European, criticism of algorithmic decision- making is that allowing a decision about humans to be made by a machine inherently treats humans as objects, showing deep, inherent disrespect for peoples’ humanity.
0 references
A second type of dignitary concern appeals, however, across cultural divides. Automatically making decisions based on what categories an individual falls into—that is, what correlations can be shown between an individual and others—can fail to treat that individual as an individual.
0 references
[The third type of] dignitary concerns include concerns (more familiar to Americans) about individual autonomy. Algorithmic decision-making founded on individual profiling limits the choices and, thus, the freedom a person will have.
0 references
The third category of concerns about algorithmic decision-making, justificatory concerns, aims to ensure the legitimacy of a decisional system. Justificatory concerns resonate strongly with calls for rule of law.
0 references
Human decision makers have the capacity to expand decisional context when it seems unfair to ignore information a machine might not know is relevant (“You are speeding on the way to the hospital”)
0 references
Collaborative governance is, at best, a highly tailored, site-calibrated regulatory system that aims to pull inputs from, obtain buy-in from, and affect the internal institutional structures and decision-making heuristics of the private sector, while maintaining the legitimacy, efficacy and public-interest orientation of public governance.
0 references
individual narratives about discrimination or bias may be more palatable to the public than agency-produced reports or statistics, and could feed back into collaborative governance by contributing to ongoing policy conversations about the broader governance regime.
0 references
Intriguingly the GDPR’s absence of public-facing and stakeholder-facing accountability suggests that individual transparency rights may have to serve a crucial accountability role in its system of collaborative governance. Thus, even for those focused on instrumental [] goals, individual rights in the GDPR may be necessary for producing effective systemic regulation, too
0 references
The GDPR for the most part envisions collaboration as taking place between regulators and regulated private parties, to the exclusion of third parties, such as civil society or external experts. This threatens both the substance and legitimacy of the regime. To some extent, this design flaw may reflect the relative weakness of civil society in the EU.
0 references
An impact assessment, in other words, is supposed to be a tripartite conversation between a regulated entity, the regulator, and third parties such as impacted persons or civil society organizations. In the GDPR, it is largely used internally or, at most, in conversation with regulators.
0 references