Introduction
Employers are naturally drawn to the idea of using AI in working processes to bypass inherent human biases. Yet, it’s crucial to acknowledge that humans develop these AI systems and rely on historical data that is not free from past human biases. Consequently, AI tools are likely to reflect, to a certain extent, the biases of those who design and train them this almost always results in Non-Compliant AI Implementations.
The Italian Data Protection Authority, Garante, imposed a €2.5 million fine on Deliveroo Italy s.r.l. for the unlawful processing of personal data of approximately 8,000 Deliveroo riders and various infringements of the EU General Data Protection Regulation (GDPR). The fine was announced on August 2, 2021, following an investigation into Deliveroo’s practices.
The main issues identified by the Garante included Deliveroo’s failure to provide transparent information about the algorithm used to manage riders’ work shifts and the collection of a disproportionate amount of riders’ personal data. This violated the principles of lawfulness, transparency, data minimization, and storage limitation under the GDPR. The Garante ordered Deliveroo to correct these GDPR violations, including those related to accountability, transparency, data storage limitation, and measures to protect the rights and interests of riders.
The Deliveroo Case: A Learning Point
Deliveroo’s case serves as a critical learning point for businesses employing AI. The Garante’s investigation revealed that Deliveroo failed to comply with GDPR requirements, including data minimisation, lawful processing, and proper disclosure about the algorithm used for managing work shifts. This highlights the importance of having a robust privacy framework in place, particularly when dealing with AI.
DPA Findings
The Garante’s findings highlighted several key areas of concern:
- Controller-Processor Roles: The Garante clarified that even if a company operates a platform with significant management by a parent company if it directly processes personal data, it is considered a data controller for that processing.
- Privacy Disclosure Requirements: The Garante emphasised that privacy disclosures must be distinct documents, not just subsections of terms of use, and should include all legally required disclosures. This is particularly important when collecting sensitive data like real-time geolocation.
- Data Retention or Minimization: The policy for data retention must be specific and reflect the appropriate retention periods for each type of processing, rather than being generic.
- Profiling: The processing of drivers’ data to determine their availability or reliability for work shifts was considered profiling. Such profiling requires enhanced disclosure about the logic used and its expected consequences for the data subject.
- Data Protection Impact Assessment (DPIA): The Garante pointed out the need for a DPIA due to the high risk to the rights and freedoms of individuals involved in Deliveroo’s processing activities. The assessment should consider various factors, such as the innovative use of digital platforms, the nature of the technology used, and the scope of application.
- DPO Obligations and Records of Processing Activities: The Garante also stressed the importance of DPO obligations, even at the group level, and the need for detailed records of processing activities (ROPA) under Article 30 of the GDPR.
The AI Lifecycle and Formiti’s Role
Formiti leads in AI data lifecycle management, encompassing data sourcing, preparation, model development partnerships, and model evaluations by humans. The AI Lifecycle involves these key steps to avoid quality issues and launch delays, with an AI Privacy Assessment Audit being an essential tool in the lifecycle of AI development.
AI and Data Privacy: The New Frontier
AI technology poses unique challenges in data privacy. Unlike traditional IT systems, AI relies on complex algorithms and massive data sets, making compliance with global data protection security requirements more intricate, both technologically and ethically. Formiti’s AI Privacy Assessment Service addresses these challenges by focusing on key areas including data minimisation, processing adequacy, and purpose limitation.