The legal basis of legitimate interests: Focus sheet on measures to implement in case of data collection by web scraping

02 July 2024

The collection of data accessible online by web scraping must be accompanied by measures to guarantee the rights of data subjects.

This how-to sheet is open for public consultation until September the 1st 2024. More information.

This content is a courtesy translation of the original publication in French. In the event of any inconsistencies between the French version and this English translation, please note that the French version shall prevail.

Reminder on the doctrine of the CNIL

The practice of data scraping has significantly increased, especially with the rapid expansion of generative AI systems which use vast amounts of freely accessible online data. However, the use of such techniques poses inherent risks to the rights and freedoms of individuals, who do not control the re-use of their online accessible data. The generalization of web scraping has thus transformed the nature of internet use, enabling third parties to read, collect, and reuse any data published online by individuals, which poses significant and unprecedented risks for data subjects.

The CNIL has regularly called for vigilance regarding these practices and issued a series of recommendations to be respected to implement them. The CNIL has also advocated for the creation of an ad hoc legislative framework (see, in particular, the CNIL opinion of December 15, 2022 on the “Polygraphe” project, in French), aiming at securing entities using such practices, regulating them and protecting personal data freely accessible online.

In some cases, the CNIL has deemed such practices prohibited in the absence of a legal framework (in particular where processing operations are carried out by competent authorities for the purpose of detecting infringements). Conversely, they have been accepted in other cases, provided that stringent safeguards were implemented, for example for searching the internet for information leaks (RIFI).

For the moment, in the absence of a specific legal framework, this how-to sheet recalls the controllers’ obligations and specifies the conditions under which such processing may be implemented for developing an AI system.

The legality of web scraping depends in particular on the possibility of relying on a valid legal basis. Collecting data accessible online to create a training dataset may be based on the legitimate interest, provided that it complies with the conditions set out in the legitimate interest how-to sheet.

For more information: see the how-to sheet “Relying on the legal basis of legitimate interests to develop an AI system”).

Mandatory measures

Certain measures are mandatory under the principle of data minimisation (Article 5.1.c of the GDPR):

define precise collection criteria in advance and apply filters to exclude the collection of unnecessary data categories (e.g. banking transaction data, geolocation data, data on vulnerable persons such as minors, sensitive data, etc., when they are not necessary);
ensure that irrelevant data collected despite these criteria is deleted immediately after collection or as soon as it is identified as such (e.g. if, in a public forum, people’s pseudonyms are collected when only the content of the comments is needed).

What to do in case of incidental collection of sensitive data?

Particular attention should be paid to the collection of sensitive data when using web scraping tools that involve the processing of large volumes of data. The controller is required to implement all measures to automatically exclude the collection of non-relevant sensitive data, such as applying filters to exclude the collection of certain categories of data or to exclude certain sites with inherently sensitive data.

If, despite these measures, the organization incidentally and residually processes sensitive data that it had not intended to collect, this shall not be considered illegal. This aligns with the Court of justice of the European Union’s stance, which recalls that that prohibition applies to the operator of a search engine ‘within the framework of his responsibilities, powers and capabilities’ (CJEU, Grand Chamber, 24 September 2019, GC and Others, C-136/17). On the other hand, if the organization becomes aware that it is processing sensitive data, in particular through the data subject, it must immediately and automatically delete such data as far as possible.

As a reminder, the processing of sensitive data may be authorised by way of exception when the personal data were manifestly made public by the data subject. It must be ascertained whether the data subject had intended, explicitly and by a clear affirmative action, on the basis of settings selected with full knowledge of the facts, to make the personal data accessible to the general public or, rather, to a more or less limited number of selected persons (CJEU, 4 July 2023, Meta Platforms,C-252/21).

For more information: see the how-to sheet “Ensuring that the processing is lawful”.

Additional safeguards

To ensure the balance necessary under the legitimate interest, the controller must generally implement additional safeguards that limit the impact on individuals’ rights and freedoms. As stated in the how-to sheet “Relying on the legal basis of legitimate interests to develop an AI system”, the choice of appropriate measures depends in particular on the intended use of the trained AI and the actual impact of that system on the data subjects. In particular, the CNIL recommends the implementation of several or even all of the following measures:

exclude, by default, the collection of data from certain websites that contain particularly intrusive data given their sensitivity (such as pornographic sites, health forums, websites or social networks used mainly by minors) or the level of information they allow to obtain about people (such as genealogical sites or which contain large volumes of structured data on people);
exclude collection from websites which clearly object web scraping and the reuse of their content for the purpose of building for AI training databases (for example, using robot.txt or ai.txt files);
create an “push-back list”, managed by the controller allowing data subjects to object to the collection of their data on certain websites or online platforms by providing information that identifies them on those websites, including before the data collection occurs;
provide data subjects with the option to object to the processing at their discretion;
limit the collection to freely accessible data (i.e. content accessible to any unregistered user without account creation) that were manifestly made public by the data subjects, i.e. that users have explicitly chosen beforehand to make publicly accessible to an unlimited audience (e.g. based on individual settings), to ensure that they do not lose control of the data they publish online (e.g. excluding data posted on social networks for private use, such as information in private profiles or groups);
apply anonymisation or pseudonymisation processes immediately after data collection;
disseminate as widely as possible information about data collection and data subjects’ rights (e.g. through online articles), ensuring that an updated list of scraped websites is published;
prevent any combination of data based on individual identifiers, for example by replacing them with random pseudonyms unique to each piece of content (e.g. each post on a freely accessible online forum) rather than each identifier, unless the controller demonstrates the need to gather different data about a particular individual for the development of the AI system or model in question.
register contact details in the list that could be set up by the CNIL, in order to identify companies using scraping tools for online data collection in order to inform data subjects and enable them to exercise their GDPR rights with the controller;

Project for a register of organisations processing data collected through web scraping for AI development

Web scraping practices, already widespread, are becoming particular important with the development of AI. Given the issues these practices raise, particularly in terms of transparency and exercising rights, the CNIL is considering establishing a register on its website where organizations scraping data for AI development could register.

Which processing would be concerned?

This register would concern organizations processing data collected via web scraping to create training datasets for AI system development falling within the material scope of the GDPR or the provisions of Title III of the French Data Protection Act. It would not, however, include organizations using web scraping for other purposes (marketing, media monitoring, etc.).

Who could register?

Both public and private organizations that process data collected via web scraping for AI development could register. This would include controllers who implement web scraping tools themselves, who use a processor or who process data previously collected by web scraping (e.g. datasets accessible online or acquired from third parties and created through web scraping, such as Common Crawl databases).

The registration would not be limited to French companies, but would include all those within the territorial scope of the GDPR and the French Data Protection Act.

For public organizations, the CNIL would also maintain a dedicated page, listing administrations authorized to carry out data scraping within the framework of their powers. This would cover processing activities authorised by relevant texts on which the CNIL would have provided an opinion.

What information would be included in the online register?

For each organization, the following information would appear in the register:

the identity (corporate name) of the controller;
a link to their privacy policy;
optionally, the list of categories of scraped websites or, at least, a reference to the privacy policy where the data sources must be listed, subject to applicable exceptions (for more information, see the how-to sheet on the information of data subjects);
the procedures for data subject rights requests.

Would registration in the register be mandatory?

No. Registration would not be mandatory even if your organization processes data collected through web scraping to create AI training datasets. However, registration would ensure greater transparency of processing activities and facilitate individuals’ exercise of their rights. Therefore, where the controller seeks to rely on the legal basis of legitimate interest, registration could be considered an additional measure in the balancing test.

Would registration in the register ensure compliance of data processing operations?

No. The CNIL would not verify the compliance of the processing carried out by organizations that register. The presence of an organization on that list would not imply that its data processing activities comply with regulations, in particular with regards to data protection.

How do I register?

Organizations could register by completing a form available online.

Registration should be carried out by the legal representative, a person with those authority within the organization or, where applicable, the Data Protection Officer (DPO).

To register, the controller would need to provide certain information necessary for identification: name and first name of the individual completing the registration, contact details (email address), SIREN number of the organization. This information would not be made public.

Would it be possible to change the information in the register or to withdraw from the register?

Yes, it would be possible to request the removal or modification of the information provided. Controllers would be strongly encouraged to regularly verify that the information provided is neither inaccurate nor outdated. The data would be deleted after a period of two years from the last modification by the controller.

< Previous : The legal basis of legitimate interests: Focus sheet on open source models

Table of content

Next : Informing data subjects >