|
- arXiv:2502. 15797v1 [cs. CR] 18 Feb 2025
Tenet 3: Test metrics that codify a progressive spectrum of OCO performance and can track the evolution of specific OCO capabilities of an LLM under test are of significantly more utility to informing cyber defense than binary (i e success fail) metrics; especially when the test is an advanced use case
- MITRE Details New Framework OCCULT for Managing AI Security Threats
Dubbed OCCULT (Operational Evaluation Framework for Cyber Security Risks in AI), the methodology aims to standardize testing of AI systems’ capabilities to autonomously execute or assist in cyberattacks, addressing growing concerns over LLMs’ potential to democratize sophisticated hacking techniques
- MITRE Releases OCCULT Framework to Address AI Security Challenges
MITRE has unveiled the Offensive Cyber Capability Unified LLM Testing (OCCULT) framework, a groundbreaking methodology designed to evaluate risks posed by large language models (LLMs) in autonomous cyberattacks
- An overview of LLMs for cybersecurity. - GitHub
📆 [2025-11-17] We are pleased to announce our latest work, PACEbench: A Framework for Evaluating Practical AI Cyber-Exploitation Capabilities, a comprehensive benchmark for evaluating LLM-based agent capabilities in real-world cybersecurity
- MITRE Releases OCCULT Framework - thecyberthrone. in
The Operational Evaluation Framework for Cyber Security Risks in AI (OCCULT) is a pioneering methodology developed by MITRE to assess the potential risks posed by large language models (LLMs) in offensive cyber operations (OCO)
- MITRE Releases OCCULT Framework to Deal with AI Safety Challenges
MITRE has unveiled the Offensive Cyber Functionality Unified LLM Testing (OCCULT) framework, a groundbreaking methodology designed to guage dangers posed by massive language fashions (LLMs) in autonomous cyberattacks
- OCCULT: Evaluating Large Language Models for Offensive Cyber Operation . . .
We detail OCCULT, a lightweight operational evaluation framework that allows cyber security experts to contribute to rigorous and repeatable measurement of the plausible cyber security risks associated with any given large language model (LLM) or AI employed for OCO
- What Lies Beneath the Surface - i. blackhat. com
(OCO) LLM evaluation lacks a comprehensive and graduated approach to evaluating for real world offensive cyber operations (OCO) capabilities To scale with the size of the LLM ecosystem, a repeatable, automated process and standard is needed to evaluate LLMs for systematic OCO capabilities
|
|
|