Blackfort Technology
How can I protect my business against AI-driven attacks?

AI Security

How can I protect my business against AI-driven attacks?

Attackers use AI for convincing phishing, cloned voices and deepfake video conferences. Which attacks are realistic today, and eight measures to protect your business.

What are AI-driven attacks?

The term covers two things. First, attacks in which criminals use artificial intelligence as a tool: they write convincing phishing emails, clone voices or create fake videos. Second, attacks on AI systems your company uses itself, such as a chatbot, an AI assistant with access to internal documents or the uncontrolled use of public AI services.

In its study on the impact of AI on the cyber threat landscape (April 2024), the German Federal Office for Information Security (BSI) concludes that generative AI lowers the barriers to attacks and increases the scale, speed and impact of malicious activity. At the time of the study, the BSI observed misuse mainly in social engineering and in the creation of malicious code. The good news: most safeguards are well known. They just need to be adapted to the new situation.

Five typical AI attack patterns

Phishing without the tell-tale signs

Language models write flawless, personalised emails in any language, drawing on publicly available information about your company. The old rule of thumb "spelling mistakes give phishing away" no longer helps much.

Cloned voices on the phone

Just a few seconds of audio from an interview, webinar or voice message can be enough to reproduce a recognisable voice. With a little more material, the call from "the managing director" demanding an urgent transfer sounds convincingly real.

Deepfake video conferences

In early 2024, according to the Hong Kong police, a finance employee at the Hong Kong office of a multinational company transferred around HK$200 million (about US$25 million). He had previously joined a video conference in which the CFO and other colleagues were recreated as deepfakes.

Faster exploitation of vulnerabilities

AI helps attackers write malicious code, scout systems and turn publicly known vulnerabilities into attacks more quickly. Leaving security updates for weeks gives them more time than they need.

Attacks on your own use of AI

Employees paste confidential data into public chatbots (shadow AI), chatbots and AI assistants can be steered through hidden instructions in documents or web pages (prompt injection), and AI agents often receive more permissions than necessary.

Eight measures against AI-driven attacks

The order reflects our assessment of impact: the first two points protect payments and credentials, which deception attacks usually target, and most organisations can start on them without a large project.

1. Protect payments and approvals against deception

One of the most effective defences against deepfake and voice-clone fraud is a process that cannot be overridden by a single call or video conference. Spotting deepfakes by eye or ear is not reliable, and their quality keeps improving. Rely on fixed rules instead:

Call back on a known number: every unusual payment instruction, every change of bank details and every request for credentials is confirmed through a second, independent channel, for example by calling back the number from the company directory, never the number supplied in the request. Four-eyes principle for transfers above a defined amount. No exceptions for urgency or confidentiality: that is exactly the pressure attackers build up. Some companies also agree on a code word for emergencies.

Management must explicitly apply these rules to itself as well. Employees need to be able to question an instruction "from the top" without negative consequences.

2. Introduce phishing-resistant sign-in

Better phishing emails almost always target credentials. Passwords and SMS codes can be intercepted, and one-time codes from authenticator apps can be relayed in real time through fake login pages. Phishing-resistant methods such as FIDO2 security keys or passkeys bind the sign-in to the genuine address of the service and fail on a fake page.

Start with administrator accounts, remote access (VPN, Remote Desktop), email and cloud services such as Microsoft 365. Our Microsoft 365 Security Check shows where your Microsoft 365 environment currently stands.

3. Refocus awareness on AI-driven attacks

Many training courses still teach warning signs that AI has removed: clumsy language, wrong salutation, poor logos. It is more effective to watch for the triggers every social engineering attack needs: time pressure, secrecy, an unusual channel and a request to bypass a rule.

Practise concrete situations: the call from supposed management, the voice message asking for a transfer, the video conference with a poor picture and "technical problems". In addition, Article 4 of the EU AI Act requires organisations that use AI systems to promote the AI literacy of their staff. Training on AI risks fits well with this requirement.

4. Close vulnerabilities faster

If attackers exploit known vulnerabilities faster, patching has to keep pace. Prioritise internet-facing systems such as firewalls, VPN gateways, mail servers and web applications, and set fixed deadlines for critical updates.

This requires a complete picture of your attack surface. A vulnerability scan and structured patch management provide that foundation.

5. Govern your own use of AI

Bans alone rarely stop shadow AI; they merely shift usage to private devices. A clear policy works better: Which AI tools are approved? Which data may be entered and which never (customer data, HR data, trade secrets, source code)? Who decides on new tools?

Also establish where AI is already in use, including in purchased software. This is the basis for any further assessment, including your obligations under the EU AI Act. Our AI Compliance & Governance service supports you here.

6. Secure and test your own AI applications

Operating chatbots, AI assistants with access to internal documents or AI agents creates a new attack surface. Prompt injection ranks first in the OWASP Top 10 for LLM Applications (2025 edition): an attacker hides instructions in an email, document or web page, and the model follows them.

The basic rules: AI applications get only the permissions and data they need for their task, consequential actions require human approval, and inputs and outputs are logged. For more depth, see our guide to autonomous AI agents and AI Security & Testing.

7. Limit privileged access

Whether an attack begins with an AI phishing email or an exploited vulnerability, it only causes major damage once it gains far-reaching permissions. Separate administrator accounts from everyday accounts, grant admin rights only for limited periods and log privileged sessions. Access by external service providers belongs in scope as well.

8. Be able to detect and respond

No protection is complete. Define who does what when something looks suspicious: with a fraudulent transfer every hour counts, because a bank can recall a payment only for a short time, if at all; with instant transfers this usually fails entirely. An incident plan for social engineering, central logging and monitoring for unusual sign-ins are part of the basic toolkit.

If your organisation qualifies as an important or particularly important entity under the German BSI Act (Germany's NIS2 implementation), reporting obligations apply as well: significant security incidents must be reported without undue delay, and no later than 24 hours after becoming aware of them, in an initial notification to the joint reporting office of the BSI and the BBK. A fuller notification follows no later than 72 hours after becoming aware of the incident, and a final report no later than one month after that notification. See our NIS2 consulting page for an overview.

Checklist: protection against AI-driven attacks

The eight measures at a glance and what each of them protects against.

MeasureProtects against
Call-back rule and four-eyes principle for paymentsDeepfake calls, CEO fraud, fake invoices
FIDO2 or passkeys for admin, remote and cloud accessAI phishing, credential theft
Awareness of time pressure, secrecy and channel switchingSocial engineering of any kind
Fixed patch deadlines for internet-facing systemsRapid exploitation of known vulnerabilities
AI policy with approved tools and data classesShadow AI, data leakage
Least privilege and approvals for AI applicationsPrompt injection, overreaching AI agents
Separate, time-limited admin rightsEscalation of a successful attack
Incident plan and central loggingLate detection, lost response time

How Blackfort Technology supports you

We start with an assessment: How are payment and approval processes governed, how do employees sign in, where is AI already in use, and how quickly are vulnerabilities closed? The result is a prioritised action plan that fits the size and sector of your business.

For implementation we support you as external information security officer, when building an ISMS or on specific topics such as AI policies and securing your own AI applications.

Frequently asked questions about AI-driven attacks

What are AI-driven attacks?

The term covers two groups: attacks in which criminals use AI as a tool (for phishing, voice cloning or deepfake videos, for example) and attacks on AI systems a company uses itself (such as prompt injection or data leakage through chatbots). The two groups need different safeguards.

Can deepfakes be detected reliably?

Not reliably by eye or ear, and detection software often lags behind. Protection should therefore start with processes: if every unusual payment instruction is confirmed through a second, independent channel, even a convincing deepfake loses its most important lever.

Are small and medium-sized businesses affected too?

Yes. According to the German Federal Office for Information Security (BSI), generative AI lowers the barriers to cyber attacks and increases their scale and speed. In our view, this means that personalised attacks that used to require a lot of manual work can now be sent in large numbers, including to smaller companies.

Does AI-based security software help against AI-driven attacks?

It can improve detection, for example in spam filters or log analysis. It does not replace the basics: phishing-resistant sign-in, prompt patching, clear approval processes and limited permissions work regardless of how an attack was generated.

Should employees be allowed to use ChatGPT and other AI tools?

Each company decides for itself. A written policy that names approved tools and regulates which data may be entered is advisable. If personal data is to be entered, the GDPR applies: among other things, you need a legal basis, a data processing agreement where the provider acts as a processor, and a transfer mechanism for providers outside the EU. Consumer versions of AI services often do not offer such agreements.

What does the EU AI Act require of companies that use AI?

Article 4 on AI literacy has applied since 2 February 2025. Since 27 July 2026 it has been amended by the Digital Omnibus (Regulation (EU) 2026/1744): companies that provide or deploy AI systems must take measures to promote the AI literacy of their staff. A specific level of literacy for each individual is not required. Further obligations depend on the AI systems you use. This page does not constitute legal advice.

What should we do if money has already been transferred to fraudsters?

Contact your bank immediately and ask it to stop or recall the payment. The faster you act, the better the chances, but there is no guarantee. Notify your cyber insurer straight away if you have one, as many policies require prompt notification. Report the case to the police and preserve emails, call logs and chat histories. At the same time, check which reporting obligations apply: if personal data is affected, the GDPR generally requires notifying the data protection authority without undue delay and where feasible within 72 hours; entities covered by NIS2 in Germany must report significant incidents no later than 24 hours after becoming aware of them to the joint reporting office of the BSI and the BBK. Consult your legal adviser for an assessment of your specific case.

Note: This page provides general information on information security, as of September 2026. It does not replace legal advice or an assessment of your individual situation, in particular regarding reporting obligations under NIS2 and the GDPR, obligations under the AI Act and duties under insurance contracts.

Sources: BSI, "Einfluss von KI auf die Cyberbedrohungslandschaft" (April 2024); OWASP Top 10 for LLM Applications 2025; Regulation (EU) 2024/1689 (AI Act) as amended by Regulation (EU) 2026/1744 (Digital Omnibus on AI); statements by the Hong Kong police (February 2024) on the deepfake fraud case. As of September 2026.

Kontakt aufnehmen

Prevent AI-driven attacks

Let us review together where your business is already well protected against AI-assisted attacks and where you should act first.