SOSS Fusion 2024

The Artificial Intelligence Cyber Challenge (AIxCC) is a two-year research competition sponsored by US DARPA in collaboration with ARPA-H. AIxCC  asks competitors to design novel AI systems to secure critical code, specifically finding and fixing its vulnerabilities, and provides significant prizes for the top winners. This presentation will provide an introduction to AIxCC, including its strategies and approaches. It will especially focus on the approach of releasing research as open source software (OSS) to support technology transfer, as well as discussing in-progress results of AIxCC.

We are seeing an increase in the number of AI powered applications. At the same time, we are seeing that AI software repeats the same security mistakes as traditional software, but at an accelerated time frame and with higher risks. In this talk -- planned as a tutorial --, we aim to show how AI applications can be developed in a safe way, starting with datasets and software dependencies, building a secure software supply chain, and only accepting models in production that have clear, untampered provenance (both SLSA but also analyzing the capabilities of the models to eliminate future risks). For example, we want to be able to trace back from a bad inference in production to the potential poisoned input in the training dataset. We will show how we can reduce cost of retraining models in the event of an ML framework compromise by analyzing the blast radius and only retraining impacted models. To keep the audience engaged, we will follow the development story of an ML model from data collection and training all the way to deploying the model in production. At each stage, we will go over the supply chain security risks and show how these can be mitigated.

Over the last decade, enterprises have had to accelerate adoption of open source software for data science, machine learning, and AI. These numerically-intensive workloads posed unique new challenges for businesses and IT due to both technology and organizational dynamics.

As the creators of the PyData movement and as a foundational distributor of open source Python tools to millions of enterprise and individual users, Anaconda has had a front-row seat to these kinds of challenges. In this talk, we will draw upon data from our annual State of Data Science industry survey to understand the kinds of challenges that businesses face while trying to adopt even well-known OSS data & ML technology.

We’ll then look towards new deep-learning and AI workloads, and the new dimensions of the security challenges there. These include novel challenges posed by exotic hardware, just-in-time compilation, binary distribution, and data-oriented supply chain attacks.

The talk will conclude with some key principles to guide thinking about software and data supply chains in the new era of machine learning and AI software deployment.

In recent years, major progress in machine learning (ML) has led to a corresponding boom in the broader artificial intelligence (AI) space, opening up commercial applications in text, image, audio, and video generation. However, data scientists and ML engineers still face many security issues that may lead to arbitrary code execution even in the space of "classical" ML, which often involves classification or regression on tabular data.

This talk will outline some of the model-centric attacks that you should be aware of and hone in on two types of attacks: malicious code injection on pickled model files, and malicious code written and executed by an LLM. We'll create a basic setup for these two attacks and see how Flyte, an open source ML orchestrator, can help mitigate some of the risks associated with these two attacks.

Finally, we'll analyze the limitations of the solutions provided by Flyte, abstract some of the ideas out in an orchestrator-agnostic way, and cover other open source tools, like `safetensors` and `onnx`, which we can leverage on top of Flyte to reduce these risks even further.

In this age of open source in machine learning, ML practitioners increasingly rely on public model hubs for downloading foundation models to fine tune instead of creating models from scratch. However, compromised artifacts are very easy to share on these hubs. ML model files are vulnerable to Model Serialization Attacks (MSA), the injection of malicious code that will execute automatically when the file is deserialized. MSAs are the Trojan horses of ML, capable of turning a seemingly innocuous model into a backdoor to your system. So, what can you do about it? In this talk, we explore two strategies to use open-source tools to mitigate the risk of MSAs and other supply chain attacks on ML: model scanning with ModelScan by Protect AI and cryptographic signing with Sigstore by OpenSSF. Model scanning is our window into the black box model files. Cryptographic signatures link an artifact to a source’s identity, backed up by a trusted authority. Scanning and signing are both widely used defenses for traditional software artifacts, but they have not been widely adopted in AI yet. WWe will demonstrate how these tools can bridge the AI/ML security gap, and stop Trojan Horses at the gate.

It surprises no one that attack surfaces expand as swiftly as AI and ML technologies advance, yet the security landscape lags behind. Join us for an eye-opening session where we dive deep into the dark world of AI security through the lens of attackers.

We'll tread carefully between different attacks, accompanied by demos, revealing the strategies and techniques used to compromise AI and LLMs. From reconnaissance and spoofing via supply chain attacks all the way to LLM poisoning, jailbreaking, and compromise—AI attacks are far from just prompt injection. Witness firsthand how attackers exploit vulnerabilities, manipulate AI systems, and leverage AI for malicious purposes.

You will gain a fundamental understanding of AI security and the nature of AI attacks, offering a rare glimpse into the adversarial mindset. By understanding the attacker’s perspective, you will be better prepared for a new era where threats are evolving and attackers are feeling increasingly comfortable in the AI domain.