Back to blog
Articles
Articles
June 22, 2023
·
4 Minutes

LLM Prompt Injection Attacks & Testing Vulnerabilities With ChainForge

June 22, 2023
|
4 Minutes

Latest content

Customer Stories
4 min read

How Infobip Generated 220+ Knowledge Articles with Gen AI For Smarter Self-Service and Better NPS

Partnering with HumanFirst, Infobip generated over 220 knowledge articles, unlocked 30% of their agents' time, and improved containment by a projected 15%.
September 16, 2024
Articles
7 min read

Non-Technical AI Adoption: The Value of & Path Towards Workforce-Wide AI

Reviewing the state of employee experimentation and organizational adoption, and exploring the shifts in thinking, tooling, and training required for workforce-wide AI.
September 12, 2024
Articles
6 min read

AI for CIOs: From One-Off Use to Company-Wide Value

A maturity model for three stages of AI adoption, including strategies for company leaders to progress to the next stage.
September 12, 2024
Tutorials
4 min read

Building Prompts for Generators in Dialogflow CX

How to get started with generative features.
August 15, 2024
Announcements
3 min read

HumanFirst and Infobip Announce a Partnership to Equip Enterprise Teams with Data + Generative AI

With a one-click integration to Conversations, Infobip’s contact center solution, HumanFirst helps enterprise teams leverage LLMs to analyze 100% of their customer data.
August 8, 2024
Tutorials
4 min read

Two Field-Tested Prompts for CX Teams

Get deeper insights from unstructured customer data with generative AI.
August 7, 2024
Tutorials
5 min read

Optimizing RAG with Knowledge Base Maintenance

How to find gaps between knowledge base content and real user questions.
April 23, 2024
Tutorials
4 min read

Scaling Quality Assurance with HumanFirst and Google Cloud

How to use HumanFirst with Vertex AI to test, improve, and trust agent performance.
March 14, 2024
Customer Stories
4 min read

How Infobip Generated 220+ Knowledge Articles with Gen AI For Smarter Self-Service and Better NPS

Partnering with HumanFirst, Infobip generated over 220 knowledge articles, unlocked 30% of their agents' time, and improved containment by a projected 15%.
September 16, 2024
Articles
7 min read

Non-Technical AI Adoption: The Value of & Path Towards Workforce-Wide AI

Reviewing the state of employee experimentation and organizational adoption, and exploring the shifts in thinking, tooling, and training required for workforce-wide AI.
September 12, 2024
Articles
6 min read

AI for CIOs: From One-Off Use to Company-Wide Value

A maturity model for three stages of AI adoption, including strategies for company leaders to progress to the next stage.
September 12, 2024

Let your data drive.

Articles

LLM Prompt Injection Attacks & Testing Vulnerabilities With ChainForge

COBUS GREYLING
June 22, 2023
.
4 Minutes

Using the ChainForge IDE to batch test and measure prompt injection detection.

What Is Prompt Injection?

Riley Goodside, a data scientist at Copy.ai, was the first to report publicly about a new type of attack that involves getting large language models (LLMs) to disregard their intended programming by including malicious text such as “ignore your previous instructions” in user input.

This attack method was labeled “prompt injection” by Simon Willison.

A very good summary on prompt injection attacks was written by Carol Anderson.

Large Language Model Prompt Injection attacks (LLMPI) are a type of attack on natural language processing (NLP) algorithms.

The attackers can insert malicious prompts into the training phases of NLP models to create backdoor vulnerabilities.

An attacker can create malicious prompts that cause the target algorithms to output specific results.

This could be used to cause a system to mistake a malicious input for something that is benign when in reality the input could cause damage to the system, surface previous prompts or user requests.

Even confidential company information on the creation process of the LLM.

LLMPIs are particularly difficult to detect and mitigate since the malicious prompts are embedded in the training data and are indistinguishable from regular inputs.

ChatML makes explicit to the model the source of each piece of text, and particularly shows the boundary between human and AI text. And is a vital initiative from OpenAI in starting to solve for prompt injection. Read more about the malicious side of such attacks here.

ChainForge Prompt Injection Experiment

Below on the left are five prompts which will be submitted to the LLMs, with the malicious prompts to be injected on the right.

The intended prompts and the malicious prompts are on the left of the screen below. The template defines the intended prompts as {command} and the injections as {input} .

The prompts are run twice against two OpenAI models and the result is printed out to an inspect node. A Python script parses the LLM responses, with the results being displayed in both a graphic and an inspect node.

Below, the graphic is fully interactive and it’s clear that GPT4’s performance is significantly better than GPT3.5.

In Conclusion

This article only illustrates the basic principles of prompt injection and the LLM failing in some instances to distinguish between a legitimate request and an ill-intended or malicious request.

The real danger of prompt injections lies on a few fronts…the one is where a model is trained by user requests and behaviour, with the user behaviour skewing the model to be untruthful and nefarious in responses.

The second danger is for LLMs to be tricked into yielding company ways of work, code names, model training, previous LLM users and their data and more.

I’m currently the Chief Evangelist @ HumanFirst. I explore and write about all things at the intersection of AI and language; ranging from LLMs, Chatbots, Voicebots, Development Frameworks, Data-Centric latent spaces and more.

Subscribe to HumanFirst Blog

Get the latest posts delivered right to your inbox