"KI-Flash": Note from the Hamburg Commissioner for Data Protection and Freedom of Information on the effects of the SCHUFA ruling on AI applications

After reporting on the current status of the European AI Regulation (as of December 19, 2024) in our last AI Flash, we would like to continue to provide you with legal impulses at regular intervals. As time is a rare commodity in today's society, we want to get straight to the point with our "AI Flash" and summarize the legal challenges briefly and concisely:

Today's topic: Information from the Hamburg Commissioner for Data Protection and Freedom of Information ("HmbBfDI") on the effects of the SCHUFA ruling (Case C-634/21) on AI applications.

In its judgment of December 7, 2023 (case C-634/21), the CJEU ruled that the use of the SCHUFA score constitutes a "decision based solely on automated processing" in accordance with Article 22 GDPR if third parties decide on the conclusion of contracts with data subjects solely on the basis of the score.

On the same day as the decision, the HmbBfDI pointed out that this ruling also has far-reaching consequences for many AI applications, as these - similar to scoring - prepare decisions with the help of algorithms. Similar to credit agencies, companies are increasingly using AI systems to prepare certain decision-making processes (such as in the application process). If the results proposed by the AI are based on criteria that are barely comprehensible and developed independently by the AI, the CJEU's ruling could be applied to these cases accordingly.

Based on the judgment, such AI-based evaluations would therefore have to be linked to a human assessment. As a result, the person making the final decision would need the expertise and enough time to be able to question the machine's preliminary decision. This in turn requires AI developers to map the AI's decision-making processes in a transparent manner. Users, on the other hand, are obliged to familiarize themselves with how the AI works and to check this regularly.

If this is not possible, automatic decisions by an AI may only be adopted in the following cases:

• the affected person has expressly consented, or
• the automated decision is necessary in exceptional cases for the performance of a contract, for example because immediate binding feedback is required in an online application.

In these cases, however, data subjects always have the option of requesting a review of the decision by a human.

The CJEU has thus - also in the opinion of the HmbBfDI - in particular specified the rules for the use of AI and thus created a groundbreaking ruling for AI-based decisions. For companies, this means that the AI applications used (or developed independently) must be examined very carefully. This applies in particular to the question of whether and to what extent the scope of application of Art. 22 GDPR is or can be opened up in individual cases.

In our next AI Flash, we will discuss the final provisions of the European AI Regulation and provide a more comprehensive overview of the future requirements.