Generative AI: Opportunities and challenges for Cybersecurity
The rise of artificial generative intelligence is opening up exciting new opportunities, but it is also raising unprecedented challenges in cybersecurity. A veritable Swiss-made knife capable of creating content from scratch, this breakthrough technology is both a driver of productivity and innovation, and a potential vector of threats if it is exploited carelessly or for malicious purposes. Between opportunities and risks, Generative AI represents a major challenge for cybersecurity.
Chapter 1: Generative AI: Ally or Threat to Cybersecurity?
Generative AI has transformed the world of work, accelerating productivity, innovation, and decision-making. However, its adoption in the enterprise raises new cybersecurity challenges. How can these technological advances be reconciled with effective data and systems protection? At Crédit Agricole, we believe that a responsible and secure adoption of AI is not only possible, but also essential to take full advantage of it.
Generative AI tools, such as ChatGPT, Mistral AI, DeepSeek, offer IT and business teams unprecedented capabilities: automatic authoring, code generation, cybersecurity assistance, and much more. However, they are not free from risks, in particular:
✖ Data leaks: Using online generative AI in a public platform can expose confidential information to its input (prompts) or when injecting documents.
✖ Misinformation and advanced phishing: Cybercriminals exploit generative AI to generate fake emails or deepfakes and thus usurp identities with worrying realism.
✖ Software vulnerabilities: Automatically generated code may contain security vulnerabilities if best practices are not followed (mastery of programming languages, rigourous source control (code reviews, tests, peer-review, etc.), integration of embedded vulnerability scanning tools, having a robust and automated integration and continuous deployment (CI/CD) environment, etc.).
✖ Risks of hallucination, regurgitation of information, etc...
✖ Result feedback bias (e.g. Amazon HR)
In the face of these challenges, a proactive approach makes it possible to secure our uses while taking advantage of the benefits of generative AI.
Best Practices for Secure Generative AI
Within the Crédit Agricole Group, we have joined forces across the different business lines to implement concrete measures aimed at protecting our systems and data while integrating generative AI responsibly. Here are our key recommendations:
1. Train and sensitise teams
➡ Getting trained: As we have seen, the rise of AI is opening up exciting new opportunities, but this technology also has risks and limitations that are important to know. With this in mind, the Group, through the Group DataLab, has designed dedicated training modules on the IFCAM portal to raise awareness among all employees of the challenges and best practices related to the exploitation of these emerging technologies. This approach aims to develop an informed and responsible use of generative AI within our Group.
➡ Adopt a critical posture: Check the sources and content generated before using them in a professional setting. Generative AIs can help you with your daily tasks but they can be wrong, it is your business skills that must be taken as proof.
2. Protect the data of the company and its customers and strengthen data security
➡ Do not enter sensitive data in public AI tools (customer information, confidential codes, internal projects, etc.). This is the reason why the Group Datalab has launched "Sécuri'Chat", a Generative AI deployed on Crédit Agricole's cloud infrastructure allowing to "chat" with a Generative AI in natural language and to integrate public to confidential classification documents. It is currently accessible to Crédit Agricole S.A. employees.
➡ Control access and governance: depending on AI tools and use cases, a restriction of access to specific profiles may be necessary and accompanied by adequate supervision.
➡ Control access to source data: several AI tools currently rely on the principle of delegated access (the tool accesses data to which the user is already authorised), and access to unsecured solutions has therefore been blocked.
3. Carry out developments under acceptable safety conditions
➡ Favour internal or secure solutions: Use models hosted on our infrastructure and validated by our IT team
4. Seeking sovereignty and validating the balance between potential and risks
➡ Validate the balance between potential and risks: Clearly identify whether the use of generative AI is necessary for a project.
➡ Seeking sovereignty: In order to control the economic and political social impacts (AI/supplier models), it is important to seek sovereignty. With this in mind, the Group DataLab, in collaboration with four Group entities, is framing a project whose goal is to create a sovereign code LLM (based on an open source code LLM that will be customised)
5. Guiding the Use of AI Tools
➡ Align the use of AI tools with current standards and regulations, in particular the Group AI normative framework (version of the IA Act* published in July 2024) which was made available in the form of a procedure note (NP 2024-44 published in October 2024 in order to define in addition the framework and requirements to be applied by each entity in the operation when using AI solutions), as well as the normative corpus of the Group’s Information Systems Security.
Conclusion: A Secure Generative AI, an asset for the Company
Generative AI is a driver of innovation, productivity and automation. It is also changing the way we work, if used with care.
Therefore, it is essential to combine performance and security, by supporting our teams in a controlled and responsible adoption of these new technologies, without compromising data protection.
Chapter 2: The use of AI by cybercriminals
As mentioned, generative AI is a powerful technology that, like any other technology, is accompanied by its own risks, which can go as far as malicious exploitation by cybercriminals.
Here are the main cyber risks related to the misuse of generative AI and how it can be used during cyber attacks:
1. Improved Phishing and Social Engineering
✖ Voice and video deepfakes
- The power of new AIs enables the creation of voice and video deepfakes to steal a leader’s identity and order actions.
- For example, a cybercriminal was able to imitate the voice of the CEO to convince an employee to transfer funds.
✖ Identity Theft and Advanced Fraud: Creating fake profiles and documents
- It is now possible with some new AI to create fake identity photos and realistic avatars.
When used for fraudulent purposes, identity theft can be used to open fraudulent bank accounts, take out loans, or carry out insurance scams.
✖ Advanced scams with conversational AI.
2. Malicious Code Generation and Vulnerability Exploitation
✖ Support for the creation of malware and ransomware
- Use for on-demand generation of malicious code.
Example: Scripting for sophisticated attack paths and generating custom ransomware.
✖ Optimise attacks on security vulnerabilities
- Generative AI can scan source code for exploitable vulnerabilities.
Example: Automatically identifying vulnerabilities in applications.
3. Automating Large-Scale Cyber Attacks
✖ Mass creation of fraudulent content
- Because of its design and capacity, generative AI can automatically generate thousands of fake posts on social media or forums to manipulate opinion.
Example: cryptocurrency scams, fake customer reviews.
✖ More sophisticated DDoS attacks
- Generative AI can analyse network traffic and adapt attacks to bypass automated defences.
Example: Smart DDoS attacks that target specific server vulnerabilities.
4. Data theft
✖ Creation of credible fake emails
- Generative AI can produce highly realistic, fault-free phishing emails tailored to the language and tone of the targeted organisation.
Examples include fake supplier emails, urgent payment requests, messages imitating colleagues or executives (fraud against the president).
5. Manipulation of Information and Disinformation
✖ Fake News and misleading content
- Generative AI can generate fake articles, images, and videos to manipulate information.
For example, spreading rumours about a company to damage its image could have consequences on its reputation or stock market price.
Conclusion: Generative AI - A Double-Edged Technology
Generative AI offers huge advantages, which is why the Crédit Agricole Group has decided to support the controlled use of AI. However, it is also a tool of choice for cybercriminals. Therefore, it is essential to adopt good cybersecurity practices, in particular through:
🔹 Raising employee awareness of new phishing and deepfake techniques.
🔹 Monitoring the use of generative AI in companies to avoid data leakage.
🔹 The implementation of advanced security solutions (detection of AI fraud, reinforced authentication).
Generative AI is not a risk in itself, but its malicious use must be anticipated and its risk controlled to protect companies and their employees.
The major challenge for our group is to detect and counter attacks produced with AIs to maintain the trust of our customers.
Generative AI is also at the service of cybersecurity, as evidenced by the “CA Générative Search” use case deployed by the Crédit Agricole Group. Indeed, this solution aims to enable employees to search for precise information in natural language from different documentary corpus, generic or specialised (for example of the regulatory type...), thanks to a technology combining Generative AI and augmented Q&A (RAG). For more information, see the article published on IT2025.
*AI Act (Artificial Intelligence Act): The European Artificial Intelligence Regulation (also referred to as the AI Act or RIA), adopted and validated by the Council of the European Union on 21 May 2024, was published in the Official Journal of the EU on 12 July 2024. These regulations came into force on August 1, 2024.