Artificial intelligence is no longer just a buzzword – it is transforming how organizations operate, make decisions, and secure their environments. Yet the same algorithms that enhance operational efficiency are being exploited to amplify threats in the cyber world. For hackers, what used to be a contest of skill and persistence has now turned into a contest of computing and adaptation.

This shift has brought a new era of cyber threats that are rapidly growing in scale, sophistication, and stealth. The question is no longer whether AI will impact cybersecurity, but whether organizations are prepared for the reality of AI-powered cyberattacks.

Why Traditional Defenses Fall Short

For years, cybersecurity has relied on rule-based systems such as signature detection on firewalls and endpoint protection detection and response platforms. While these traditional defenses have evolved and improved significantly over the last few years, they were largely built to defend against human-created and human-executed threats. Today, these methods are no longer enough, as attackers are leveraging AI to execute advanced attacks.

Here are some ways attackers are using AI:

• Automate reconnaissance: Scanning for vulnerabilities at a scale and speed no human can match.
• Bypass security tools: Learning from defenses and adapting in real-time to avoid detection.
• Craft convincing phishing campaigns: Using generative AI to mimic writing styles, clone voices, or produce deepfake videos.
• Exploit zero-day vulnerabilities faster: Automating exploit creation once a flaw is identified.
• Manipulate or poison AI models: Feeding biased or malicious data to disrupt machine learning processes.

According to the recent Microsoft Digital Defense Report, in July 2025 alone there were  >200 instances of foreign adversaries using AI‑generated fake content — more than double the number from the previous year, and over ten times the count from 2023. They have also found that AI-driven phishing is now three times more effective than traditional campaigns.

In August 2025, Anthropic said it had detected and blocked hackers attempting to misuse its Claude AI system to write phishing emails, create malicious code, and circumvent safety filters. The company said it made the attempted attack public in the hope that tech companies and regulators will intensify efforts to strengthen safeguards as the technology spreads.

AI Risk Assessments: A New Cybersecurity Imperative

As AI increases risk and attack surfaces, organizations need to incorporate AI Risk Assessments into their broader risk management strategies. This goes beyond assessing standard IT systems. It includes more advanced assessments in areas such as:

• Mapping the AI Attack Surface – Identify all internal and external AI systems in use — including third-party APIs, machine learning models, and automated decision engines. Each of these components may introduce new vulnerabilities.
• Data Governance and Model Security – Ensure data used to train AI models is clean and protected. Organizations must assess the risk of model inversion or membership inference, which enables attackers to reconstruct training data or determine whether specific data was included in the training set.
• Third-Party and Supply Chain AI Risks – Manage risks from third-party vendors that incorporate AI. While vendors may promote the use of AI in their software as a marketing strategy, these systems may bring additional exposure to cyber security incidents. They may be compromised or act unpredictably due to AI model hallucinations.
• Regulatory Compliance – Conduct risk assessments to meet compliance requirements as AI regulations, such as the EU AI Act or U.S. AI Executive Orders, come into effect. Non-compliance could bring legal, reputational, and financial consequences.

Updating Incident Response Playbooks for the AI Era

Navigating an AI security incident is much more complex than a regular incident. Newer attack techniques require newer defenses, which makes the traditional incident response plans inadequate for AI-driven attacks. Organizations must update their IR playbooks to accommodate these new tactics, techniques, and procedures (TTPs) used in AI-enhanced threats.

• Include AI-Specific Threat Scenarios – Create customized playbooks for AI–specific threat scenarios like prompt injection, model poisoning, system prompt leakage. Addressing an AI incident requires a distinct set of logs and evidence to analyze compared to handling more common scenarios such as a business email compromise incident.
• Ensure Logging of your AI models - Configure your AI model’s logs, outputs, and model decisions correctly to ensure that it captures all the information needed in case of an incident.
• Cross-Functional Response Teams - Identify and engage the appropriate stakeholders in your incident response playbook. AI incidents may require collaboration among data scientists, legal teams, communication teams, and senior leadership.
• Real-Time Decision-Making Frameworks - Provide responders with guidelines for making time-sensitive decisions involving AI systems. For example, when should a model be disabled? When should a public notification be issued? Clear thresholds and escalation paths are vital.
• Model Isolation and Rollback - If an AI model is compromised, responders must be able to isolate it from the environment, roll back to a safe version, or switch to manual control. Your IR procedures should reflect this capability.

OWASP has defined the Top 10 AI based threats. One of them is Prompt Injection where an attacker manipulates the model’s behavior by injecting malicious instructions into prompts. These can be directly typed by users, or indirectly embedded in third-party content (like emails, web pages, etc.).

As this threat is new, your IR playbook must be customized to address it by providing detailed response steps. An example is shown below:

IR Playbook for Prompt Injection

• Detection:
  • Monitor for output anomalies (e.g., sensitive data leaks, rule-breaking outputs).
  • Log and inspect user prompts and model responses.
• Logs to Capture:
  • Full prompt and response pairs.
  • System prompts at the time of execution.
  • User identity/session ID.
• Response Steps:
  • Isolate the affected application or endpoint.
  • Confirm the attack vector.
  • Sanitize user inputs and reconfigure prompt templates.
  • Review logs for further signs of compromise.
• Prevention:
  • Use prompt templates with strong delimiters.
  • Restrict AI model capabilities using tools like guardrails or content filters.

Why We Still Need Human Input

AI can enhance cyber defenses, but it's not the perfect solution to all problems. AI without human insight can create blind spots and lead to false positives or hallucinations. Here's why human involvement remains important:

1. Understanding Context

AI can detect anomalies, but it lacks the capacity to understand the exact context and nuances of a situation. For example, a spike in network traffic might signal an attack or it may just be due to a regular testing. Your AI may flag it as an attack or completely ignore it. Human insight helps understand such situations and infer what actions to take.

2. Decision-Making in Uncertainty

During active attacks, incident responders need to work with incomplete data. Human judgment is critical to triage incidents, coordinate response and make ethical decisions, especially when it comes to critical decisions such as identification of data exposure or potential system shutdowns. Ultimately, decisions regarding shutdowns, containment, and consequences are business decisions that require careful consideration by both analysts and management.

3. Poisoning the AI model itself

How can you trust a decision provided by the AI model if the model itself is compromised? Humans must oversee training data integrity, review model outputs, and ensure systems aren't manipulated.

4. Ethics and Accountability

Who is responsible when an AI system makes a mistake? Maybe it allowed a breach, or it is falsely accusing an employee of a wrongdoing. Only humans can provide ethical oversight, accept accountability, and refine policies based on evolving norms and laws.

Final Thoughts: A Human-AI Alliance

AI-powered cyber-attacks are closer than you expect. Whether it’s AI-generated phishing campaigns, automated vulnerability scanning, or adversarial attacks against machine learning systems, cybercriminals are exploiting the power of AI at an accelerating pace.

Organizations must rethink security from the ground up. Traditional tools can’t keep pace, but neither can AI defend on its own. The key lies in combining the strengths of AI while relying on human oversight to guide strategy, ethics, and decision-making.

Being prepared is not just about buying new tools — it’s about cultivating new skills, updating processes, and building resilience. With the right approach, you can be better prepared for an AI-powered cyberattack.

AI Risk Assessments with ERMProtect

With AI continuing to grow at rapid rates, many organizations are worried that their risk management strategies are failing to keep pace with these speedy advances in technology. Schedule a free consultation meeting with Dr. Collin Connors to discuss how ERMProtect can help you gain control over your AI risk. Collin can be reached at [email protected] for a free consultation.