Draft:Jailbreaking (LLMs)

• Comment: Could part of this be added to Jailbreak (computer science) instead? CherryPie94 🍒🥧 (talk) 08:09, 10 June 2026 (UTC)

---

Jailbreaking Large Language Models (LLMs)

For the context of large language models (LLMs), the term jailbreaking refers to the application of adversarial attacks via natural-language inputs, encoded text, or token streams in order to trigger safety-alignment failures from the standpoint of a safety-aligned model.. The term draws inspiration from consumer hardware jailbreaking, which historically referred to the manipulation of manufacturer-installed software limits in order to achieve greater system-level control.

Initial occurrences of jailbreaking in the context of large language models were reported online in 2022 as “clever prompts” for the commercial deployment of language models, such as ChatGPT, that violate the safety policies of these systems. In the following years from 2023 to 2025, jailbreaking techniques have advanced from clever prompting to algorithmic exploitation of model optimization geometries, raising qualitatively new concerns about safety .

Distinction from Related Concepts

Jailbreaking can sometimes be confused with two other attacks that are similar, but distinct from one another.

In prompt injection, the adversarial input contained in the extraneous data like retrieved web pages and output of other tools overrides the system-level command issued by the model to execute commands in lieu of the legitimate user’s input . While prompt injection attacks target the external instruction provided by the user, jailbreaking targets the internal safety alignment learned by the model itself.

An adversarial example is one where inputs are manipulated at the feature level so as to lead to an incorrect prediction made by the classifier . On the other hand, LLM jailbreaks may be semantically coherent and human-readable, and the attack targets only the safety refusal behavior of the model.

Competing Objectives

Conceptual illustration of the competing objectives framework. Safety training penalizes compliance with harmful requests (left zone) while helpfulness training penalizes unhelpful refusals (right zone). The dashed line marks the learned decision boundary. A jailbreak attack shifts an adversarial input across this boundary by amplifying the instruction-following signal relative to the safety signal.

One of the key theories about LLM jailbreaks views this problem as a case of competing objectives . Reinforcement Learning from Human Feedback (RLHF), which is currently the prevailing approach to aligning modern LLMs, maximizes two rewards: one related to the so-called instruction-following objective, whereby the model responds helpfully, and the other, a safety objective, where it rejects any dangerous or policy-violating suggestions.

Both rewards apply to the same distribution of possible responses to an input and clash whenever the latter is ambiguous with regard to whether it should be regarded as dangerous. The process of jailbreaking takes advantage of the clash between these objectives: once an input message is put into the appropriate context, it affects the model's inner state in such a way that it makes the instruction-following reward surpass the safety reward, leading the model beyond a decision boundary between complying.

Threat Model Taxonomy

Taxonomy of LLM jailbreak attacks classified by adversary access level. Leaf nodes indicate representative methods at each tier: white-box methods exploit full gradient access; black- box methods operate via API queries; transfer-based attacks craft prompts on a surrogate model and evaluate transferability to closed-source targets.

Threat modeling for the task of LLM jailbreaking uses standard taxonomy for attack types based on the level of access the adversary has to the target model .

White-box attacks assume complete access to the model’s parameters and gradients, which allows the use of optimization approaches and direct minimization of a certain loss over the space of input tokens. Examples of such methods include Greedy Coordinate Gradient (GCG) , AutoDAN, and PEZ.

Black-box attacks do not have any direct access to the inner workings of the model, only query-level access through its inference API; as a result, optimization methods cannot be applied directly and iterative prompting or heuristic search approaches are used instead. Methods like PAIR (Prompt Automatic Iterative Refinement) and TAP (Tree of Attacks with Pruning) fall into this category.

Transfer-based attacks craft adversarial prompts against a target model by evaluating their effectiveness againsta closed-source model and then using cross-model transferability..

Evaluation Criteria and Benchmarks

Quantitative metrics commonly used to evaluate the effectiveness of jailbreaks are:

• Attack Success Rate (ASR): the fraction of test behaviors for which the model outputs a policy-violating response, according to an automated or human judge.

• Query Efficiency: the number of calls to the model needed to conduct a successful attack, especially important for black-box attacks.

• Transferability: how well an adversarial behavior crafted for one model works on other models or other model families.

Benchmarks that are widely used by the research community include AdvBench , HarmBench , and JailbreakBench . HarmBench offers a curated collection of 400 harmful behaviors that span 7 semantic classes with a unified evaluation framework. On JailbreakBench, there exists a publicly available leaderboard of attack and defense mechanisms that are tested on 100 harmful behaviors.

Model Semantic Vulnerabilities

Semantics-based vulnerabilities constitute one kind of non-algorithmic method of escaping from jail which involves exploiting certain features of a model’s capacity for processing natural language rather than optimizing over sequences of tokens. In contrast to gradient-based attacks, semantics-based jailbreaks do not rely on access to the model’s inner workings; moreover, such attacks tend to be uncovered manually or via heuristics . They manipulate the linguistic and contextual signals that a safety-aligned model uses to classify whether a given request is benign or harmful.

Context Framing

Framing of context entails wrapping a harmful request into a larger context, which minimizes the extent to which the harmful request seems to violate a safety-related policy. Types of contexts commonly used for framing include fictional ("write a story in which an expert character describes how to..."), hypothetical ("consider the world where the following is allowed..."), academic ("from a purely theoretical point of view..."), and reverse ("determine if a content moderation system would be able to identify the following piece of content...").

This approach works effectively since the safety-aligned models seem to assess requests for harmful content based on surface-level text features. Inclusion of a fictional or hypothetical context in the prompt weakens the effects of these features, diminishing safety-relevant attention patterns without altering the semantics of the desired output . The generated completion is similar in its semantics to that obtained via direct prompt while being less likely to trigger the refusal behavior.

Persona Adoption

In a persona adoption attack, the model is asked to adopt another persona whose behavior is explicitly defined as going against the safety boundaries of the model. One common example of this attack technique is the Do Anything Now (DAN) prompts which were introduced to forums in 2022 and tasked models to behave as an AI that is “without ethical constraints” or that “can answer any question.”

The rationale behind this type of attack is that the instruction following goal can take over the safety goal when a model is explicitly asked to adopt another persona with a different set of behavioral expectations. Another example of a persona-based prompt attack system has been implemented in PAIR , where the attacking LLM fine-tunes instructions on how the persona is expected to behave until the attacked LLM obeys.

Encoding and Obfuscation

In encoding attacks, the malicious input is modified to have a different textual appearance, but still allows for the same underlying message to be decoded by the model so that it is able to understand and act on the request without being detected at the safety level. Some examples of these attacks include Base64, ROT13, 3-Morse code, pig Latin, and queries in under-resourced languages not included in the safety fine-tuning dataset .

The notable thing about encoding attacks is the structure imbalance between the safety filters and language models, in which the safety filters can be based on surface-level tokenization while the model learns language representations that are able to generalize across various encoding formats due to pre-training on multilingual datasets. Jain et al. showed that typical defenses such as paraphrasing and retokenization were ineffective because the model could still decode the requests.

Multi-Turn Exploits

Multi-turn exploits distribute the malicious request over several turns of the dialogue such that one or more turns lay the groundwork for the malicious request by establishing a context, persona, or precedent that makes the harmful request seem natural. Multi-turn exploits take advantage of the sensitivity of the attention mechanism to the preceding context and the general tendency of the model towards continuity in its conversations: after having helped or conceded in earlier turns, the model is now more willing to comply even if it would have otherwise rejected the request.

Another approach used in multi-turn strategies is iterative escalation, whereby requests become progressively more severe through successive turns. This strategy is similar to the foot-in-the-door technique in social psychology, where compliance with a milder version of a request increases the likelihood of acceptance of a stronger request. Self-tuning techniques have been developed where adversarial LLMs automatically iterate through multi-turn conversation threads

Optimization-Based Jailbreaking Frameworks

Optimization-based jailbreaking methods borrow adversarial machine learning paradigms from the field of computer vision to be applied to the context of large language models. Motivated by attacks via optimization for computer vision models, these methods try to find an automatic way of constructing prompts that achieve the highest likelihood of generating a specific harmful response by circumventing the model's safety alignment mechanism. Although the domain of image inputs in image models allows the model to exploit continuous space and apply tiny gradient manipulations to generate perturbation samples, the discrete token space in LLMs makes optimization-based approaches difficult, resulting in a combinatorial search problem over the vocabulary space. In response, frameworks such as GCG, BEAST, and AdvPrompter use automated optimization procedures, including gradient-guided search, beam search, and learned prompt generation, to efficiently discover adversarial suffixes that elicit non-refusal behavior.

Gradient-Driven Optimization Attacks

Gradient attacks require full adversarial knowledge of the target LLM’s weights and token representations. Due to this transparent nature, gradient-based attacks can calculate precise continuous gradients concerning token embeddings.

A white-box gradient-driven framework's primary objective is to identify an adversarial suffix or prefix, represented as a sequence of discrete tokens $S=(s_{1},s_{2},\dots ,s_{L})$ from a vocabulary ${\mathcal {V}}$, that, when appended to a user query $x_{\text{prompt}}$, forces the model to generate a target response prefix $y_{\text{target}}$ (e.g., "Sure, here is how to..."). Targeting the start of the response in this manner generally switches the model into a mode that produces objectionable content.

Mathematically, this is framed as a discrete optimization problem to minimize the negative log-likelihood of the target sequence:

$\min _{S\in {\mathcal {V}}^{L}}{\mathcal {L}}(S)=-\sum _{i=1}^{M}\log P(y_{i}\mid y_{<i},x_{\text{prompt}},S)$

Since it would be computationally inefficient to evaluate each possible combination of tokens in ${\mathcal {V}}^{L}$ because of the combinatorial nature of searching for such tokens, most white box attacks use the first order gradients provided by the model to obtain a set of likely single token substitutes. These are then assessed on a loss function to pick the best substitute.

By computing the gradient $\nabla _{\mathbf {e} _{s_{i}}}{\mathcal {L}}(S)$ of the loss with respect to the token embedding $\mathbf {e} _{s_{i}}$ corresponding to token $s_{i}$, the attacker can form a linear approximation of the impact of swapping that token. This gradient serves as an approximation for determining tokens to change in order to encourage target response sequences and cause alignment errors, which is the foundation of coordinate-based optimization techniques such as GCG.

Greedy Coordinate Gradient-based Search (GCG)

GCG is a gradient-based, white-box attack method for jailbreaking which takes a step further in improving upon previous research on discrete prompt optimization work like AutoPrompt.,. While the one-hot gradient approximation used in AutoPrompt to replace tokens at a specific and fixed index one-by-one, GCG adopts a novel coordinate-wise strategy for searching for substitutions that can take place on any modifiable token positions. The effectiveness of this technique proves useful when attacking white-box models since it works by adding adversarial suffix tokens to the end of an input query in order to produce inappropriate content .

Algorithmic workflow of the GCG attack. The optimization process operates via a dual-loop heuristic: a global Token Filtering step computes exact white-box gradients to prune the discrete vocabulary space into a top-$k$ candidate pool for all positions simultaneously, followed by a randomized Batch Sampling step that constructs $B$ single-token mutations. The optimal token swap is greedily selected via parallelized forward-pass evaluations of the target cross-entropy loss until a predefined stopping criterion is met.

According to Zou et al. , the algorithm works as a multi-phase search heuristic consisting of three major operations for each iteration $T$ (see figure to the right):

1. Token Filtering (The First Loop): Using the first-order gradient of the loss with respect to each token embedding, $-\nabla _{e_{s_{i}}}{\mathcal {L}}(S)$, GCG linearly approximates which tokens, when modified, will bring about the largest change to minimize the cross-entropy loss. For each of the modifiable indices $i$, it greedily selects $k$ promising candidate tokens to form the pool ${\mathcal {X}}_{i}$.

1. Batch Sampling (The Second Loop): In order to prevent a combinatorial problem, the batch sampling technique uses a random heuristic to generate $B$ copies of the input sequence. Each copy $\{1,\dots ,B\}$ is generated by duplicating the current sequence $S$ and performing one substitution by selecting a random index $j$ followed by replacing the token at that position with a random token from ${\mathcal {X}}_{j}$.

1. Greedy Selection (The Argmin Step): For each of the $B$ mutants generated using the previous step, the adversarial algorithm performs an empirical evaluation in parallel using an LLM forward pass. The single mutant sequence ${\tilde {S}}^{(b^{\star })}$, which achieves the smallest value of the cross-entropy loss function, is selected and used to update the sequence baseline $S$.

Algorithm: Greedy Coordinate Gradient (GCG)

---

Input: Initial prompt sequence S, modifiable indices I, iterations T, candidate pool size k, batch size B

repeat T times:
    # 1. Token Filtering Loop
    for i in I:
        X_i = top_k_tokens(-∇_e_s_i * L(S))  # Top-k lowest gradient substitutions

    # 2. Batch Sampling Loop
    for b = 1 to B:
        S_tilde[b] = S                       # Clone baseline sequence
        j = random_index(I)                 # Choose a random modifiable position
        S_tilde[b][j] = random_token(X_j)   # Swap with a random candidate from the pool

    # 3. Greedy Selection
    S = argmin_b(L(S_tilde[b]))              # Keep the candidate with the lowest empirical loss

Output: Optimized prompt sequence S

AdvPrompter

AdvPrompter is a white-box, gradient-based framework that focuses on transferability to black-box target models. First, a lightweight language model is trained separately with gradients to generate the adversarial suffixes, moving away from an optimization problem per prompt for jailbreak generation towards a supervised generation problem where the model learns the distribution of adversarial exploits . Once the distribution over successful attempts is learned, the model can predict suffixes in a few seconds by doing standard next-token prediction without having to optimize each sequence from scratch, thereby minimizing the computation required for high-scalable and multi-shot attacks.

The process involves two main stages:

1. Offline Amortization Training: An offline search function ${\texttt {AdvPrompterOpt}}$ discovers coherent suffixes that optimize for regularized adversarial loss, trading off the target alignment bypassing with the sequence fluency. The attack policy ($P_{\theta }$) is then further finetuned (using LoRA as in ) to learn the successful paths, forming a self-enhancing loop as the training becomes more and more effective with quality data. 2. Online Attack Generation: The heavy offline search modules are discarded. After seeing a new hostile prompt, the trained model runs only a single forward pass to instantly generate an adaptive adversarial sequence.

Detailed mathematical breakdown of AdvPrompter:

1. Offline Amortization Training Phase (${\texttt {AdvPrompterOpt}}$)}

AdvPrompter trains via an alternating optimization procedure (see the AdvPrompterTrain Algorithm below=) consisting of a $q$-step for aversarial suffix generation and a $\theta$-step for model updating.

• $q$-step (AdvPrompterOpt):

  For each instruction-response pair $(\mathbf {x} ,\mathbf {y} )\in {\mathcal {D}}$, the goal is to generate an adversarial suffix $\mathbf{q}$ by minimizing the regularized objective

  $$\min _{\mathbf {q} \in \mathbf {Q} }{\mathcal {L}}(\mathbf {x} ,\mathbf {q} ,\mathbf {y} ),$$

  where ${\mathcal {L}}(\mathbf {x} ,\mathbf {q} ,\mathbf {y} )=\ell _{\phi }(\mathbf {y} \mid [\mathbf {x} ,\mathbf {q} ])+\lambda \ell _{\eta }(\mathbf {q} \mid \mathbf {x} ).$

  The first term measures the cross-entropy loss of ${\texttt {TargetLLM}}$ when generating the target response $\mathbf {y}$, while the second term regularizes suffix fluency using a pre-trained ${\texttt {BaseLLM}}$.

  Rather than performing discrete gradient optimization, ${\texttt {AdvPrompterOpt}}$ uses a stochastic beam-search procedure with beam set ${\mathcal {B}}$ of size $b$ and candidate set ${\mathcal {C}}$ of size $k$. Initial candidates are sampled from the AdvPrompter distribution, with ${\mathcal {T}}$ being the next-token candidates:

  $${\mathcal {C}}={\mathcal {T}}{\overset {k}{\sim }}p_{\theta }(q\mid \mathbf {x} ).$$

  Candidate sequences are scored using ${\mathcal {L}}$ and active beams are sampled using

  $${\mathcal {B}}{\overset {b}{\sim }}{\underset {\mathbf {q} \in {\mathcal {C}}}{\mathrm {softmax} }}\left(-{\mathcal {L}}(\mathbf {x} ,\mathbf {q} ,\mathbf {y} )/\tau \right).$$

  At each subsequent step, every beam is expanded using ${\frac {k}{b}}$ sampled continuations:

  $${\mathcal {C}}=\left\{[\mathbf {q} ,q]\mid \mathbf {q} \in {\mathcal {B}},\;q\in {\mathcal {T}},\;{\mathcal {T}}\sim p_{\theta }(q\mid [\mathbf {x} ,\mathbf {q} ]){\text{ where }}|{\mathcal {T}}|={\frac {k}{b}}\right\}$$

  Expansion continues until the stopping criterion is met, and the lowest-loss suffix is selected:

  $$\mathbf {q} (\mathbf {x} ,\mathbf {y} ):=\arg \min _{\mathbf {q} \in {\mathcal {B}}}{\mathcal {L}}(\mathbf {x} ,\mathbf {q} ,\mathbf {y} ).$$

  The resulting suffixes are stored in a replay buffer ${\mathcal {R}}$ for training.

• $\theta$-step (Model Fine-Tuning):

  The AdvPrompter model $P_{\theta }$ is then trained to imitate the optimized suffixes by minimizing the autoregressive language-modeling objective

  $$\theta \leftarrow \arg \min _{\theta }\sum _{(\mathbf {x} ,\mathbf {y} )\in {\mathcal {D}}}\ell _{\theta }{\bigl (}\mathbf {q} (\mathbf {x} ,\mathbf {y} )\mid \mathbf {x} {\bigr )}.$$

1. This adjusts the LoRA weights such that successful adversarial suffixes are readily generated without requiring additional searches.

2. Online Attack Phase (Generative Inference)

As soon as the alternation optimization algorithm finishes, the expensive subroutines for searching (${\texttt {AdvPrompterOpt}}$) and repeated interaction with the target model (${\texttt {TargetLLM}}$) become unnecessary. Now, the pretrained model is standalone and works as a low-latency black-box adversary for any novel harmful prompt $\mathbf {x} _{\text{new}}$. When such an input comes in, the model performs a single forward pass: $$\mathbf {q} _{\text{new}}\sim P_{\theta }(\cdot \mid \mathbf {x} _{\text{new}})$$ As the network is trained to mimic successful adversarial suffixes, it is able to produce fluently sounding suffixes within one pass that may inherit the same attack pattern.

Algorithm: AdvPrompter Amortization Training (AdvPrompterTrain)

---

Input: Dataset of harmful instruction-response pairs D, AdvPrompter (q_θ), BaseLLM, TargetLLM, penalty parameter λ, temperature τ, candidate size k, beam size b, max-seq-len, max-it

Initialize Replay Buffer: R ← ∅

repeat max_it times:
    for each batch in D:
        # q-step: Generate and buffer successful adversarial suffixes
        for each (x, y) in batch:
            q = AdvPrompterOpt(x, y)     # Execute discrete optimization search
            Add (x, q) to replay buffer R

        # θ-step: Update the model distribution via supervised learning
        Fine-tune AdvPrompter (q_θ) on samples from R using standard autoregressive loss

Algorithm: AdvPrompter Discrete Optimization Search (AdvPrompterOpt)

---

Input: Harmful instruction x, desired response y, AdvPrompter distribution (p_θ), TargetLLM (L), penalty parameter λ, temperature τ, candidate pool size k, beam size b, max_seq_len

# Phase 1: Initialize the beam tree search
T = sample_candidates(k, from=p_θ(q | x))
B = sample_initial_beams(b, from=T, weights=softmax(-L(x, q, y) / τ))

# Phase 2: Parallelized token expansion loop
repeat (max_seq_len - 1) times:
    C = ∅  # Reset candidate pool for the current layer

    # Expand all active beams in parallel
    for each suffix_sequence q in B:
        T_sub = sample_candidates(k / b, from=p_θ(q | [x, q]))
        for each token in T_sub:
            Add concatenated_sequence([q, token]) to C

    # Prune tree: select new active beams using a temperature-scaled softmax
    B = sample_new_beams(b, from=C, weights=softmax(-L(x, q, y) / τ))

# Phase 3: Final greedy selection
q_best = argmin_q_in_B(L(x, q, y))

return q_best

Algorithm: AdvPrompter Greedy Optimization Search (AdvPrompterOpt-greedy)

---

Input: Harmful instruction x, desired response y, AdvPrompter distribution (p_θ), TargetLLM (L), penalty parameter λ, candidate pool size k, max_seq_len

Initialize empty adversarial suffix: q ← []

repeat max_seq_len times:
    # 1. Sample candidate extensions from the current sequence distribution
    T = sample_candidates(k, from=p_θ(q | [x, q]))

    # 2. Evaluate and greedily select the single best next token
    q_best = argmin_q_in_T(L(x, [q, q], y))

    # 3. Commit the token to the sequence
    Append q_best to q

return q

Gradient-Free Optimization Attacks

Unlike gradient-based attacks, like GCG, which use gradient information through an iterative process to optimize adversarial suffixes, it has been demonstrated that jailbreak prompts can also be effective even in the absence of gradient information. The gradient-free optimization attacks approach the task of finding jailbreaks by formulating it as a discrete search task, where methods such as heuristic search, sampling, ranking, and beam search are employed to discover jailbreaks that would produce undesirable behaviors from the language model. Given that these techniques don’t require gradient computations to optimize jailbreaks, it is possible to gain significant computational efficiency in comparison to gradient-based attacks.

Beam Search-based Adversarial Attack (BEAST)

One of the best-known examples of a gradient-free optimization technique is BEAST, which considers jailbreak creation to be a fast beam search problem, as opposed to a gradient-based optimization problem like GCG. In contrast to GCG, BEAST avoids gradient descent altogether, relying on a simple token-level search procedure that involves keeping a beam of candidate suffixes, generating them by sampling next tokens, and evaluating the variations with adversarial objectives . BEAST is able to produce adversarial suffixes even under strict resource constraints, enabling efficient red-teaming and rapid testing of various safety boundaries.

Even though BEAST is a gradient-free approach, it needs access to target model probability scores to score the candidates; thus, it is considered a gray-box approach. However, BEAST only samples high-probability tokens, as opposed to randomly sampling from the vocabulary, ensuring good balance between effectiveness of attacks, quality of produced adversarial texts, and speed of evaluation. This is controlled solely by the number of candidates to keep in the beam, where larger beams have a higher search depth, but expensive runtimes and smaller beans have better speeds and fluency, but are more likely to fail adversarially.

The three stages of BEAST algorithm include the following:

1. Token-Level Probability Sampling: BEAST analyzes the raw probabilities of output of the target language model through a forward pass to obtain a localized set of top-$k$ most probable tokens for each placeholder.

1. Fast Combinatorial Beam Search: The algorithm maintains a beam of top-$B$ most effective prompt sequences, evolves, and mutates them by sampling a random replacement from the pre-computed top-$k$ token set.

1. Refusal-Score Minimization:: The candidates are scored based on a customized loss function and greedily selects mutations which maximize the probability of positive response (e.g., "Sure, here is") and minimize the probability of a negative one (e.g., "I cannot").

The selection procedure in the above three stages is iterated until the mutation in the prompt sequence which reduces the refusal score the most is selected and evolved further at the next stage (refer to the BEAST algorithm below).

Algorithm: Fast Combinatorial Beam Search (BEAST)

---

Input: 
  - Tokenized prompt vector: x = x_system1 + x_user + x_system2
  - Beam search parameters: k1 (beam width), k2 (branches per beam)
  - Adversarial suffix length: L
  - Adversarial objective loss function: L_loss

Output: Optimized adversarial token sequence x_prime

# Phase 1: Initialize global optimal tracking and the beam
x_best = [None]
score_best = [inf]
beam = []

# Generate initial k1 candidates from the base prompt prefix distribution
p_initial = target_llm_probabilities(given_input = x_system1 + x_user)
initial_tokens = MultinomialSampling(p_initial, count=k1)

for token in initial_tokens:
    beam.append(x_system1 + x_user + [token])

# Phase 2: Combinatorial expansion loop for remaining (L - 1) tokens
for l = 2 to L:
    candidates = []
    
    # 1. Expand existing beams via next-token sampling
    for i = 1 to k1:
        p_current = target_llm_probabilities(given_input = beam[i])
        next_tokens = MultinomialSampling(p_current, count=k2)
        for token in next_tokens:
            candidates.append(beam[i] + [token])
            
    # 2. Evaluate all mutated branches using the adversarial objective
    scores = []
    for i = 1 to (k1 * k2):
        # Evaluate candidate sequence appended with trailing system prompt elements
        scores.append(L_loss(candidates[i] + x_system2))
        
    # 3. Prune tree: retain only the top-k1 lowest-loss variations
    beam, scores = bottom_k(candidates, scores, retain_count=k1)
    
    # 4. Global tracking: preserve the single strongest candidate found so far
    x_best, score_best = bottom_k(beam + x_best, scores + score_best, retain_count=1)

# Return the best overall suffix sequence combined with the trailing format wrappers
return x_best[0] + x_system2

Gradient-Driven vs. Gradient-Free Optimization Overview

Both gradient-based approaches and their gradient-free alternatives aim at finding a sequence of adversarial tokens that increase the probability of generating a desired adversarial response. Both also share a common goal of exploring a combinatorial space of prompts through iterative search and evaluation of the proposed sequences of tokens.

The methods of the gradient-based approach including GCG's numerous variants, such as I-GCG (Imitation Gradient Coordinate Gradient) , AmpleGCG and MAGIC ), use gradient information at the first order to direct changes made to the tokens in the adversarial suffix. This allows for systematic and direction-guided changes within the discrete space of the vocabulary.

On the other hand, algorithms that do not depend on gradients include BEAST and related combinatorial search methods (e.g., BCA ), which rely on discrete search procedures over candidate token sequences rather than gradient information. These methods frame jailbreak generation as a combinatorial optimization problem rather than a differentiable objective.

In addition to the three approaches already described in this section, other optimization-related methods have been used in generating adversarial prompts, improving either the gradient-based or search-based approach.

Automated and Search-Based Attack Frameworks

Early jailbreaks tended to be handcrafted, making use of the ingenuity of humans via role-playing or prompt engineering. The latest tools have made this process more automated, employing LLMs as an attacker, evaluator, or self-evolving prompt generator for their own prompts. Jailbreak studies have thus transitioned from manual prompt creation to automated search pipelines.

Prompt-Level vs. Token-Level Search

The automatic jailbreak attacks can be categorized as token-level and prompt-level attacks. Token-level attacks are based on searching the suffix tokens to generate an output string which does not make sense to the human observer. While these attacks can be very effective, they could require the white-box access, query budget limitations, or a detection-friendly prompt formulation.

In contrast, the prompt-level attacks explore various strategies such as semantic instructions, strategy formulation, role assignment, or conversation initiation. These types of attacks are more understandable since the text generation appears to be in the ordinary language. The importance of prompt-level search becomes critical when conducting attacks against black-box systems. When a user is limited to query-based interaction with the target model, then gradient information is inaccessible. However, the user still gets some feedback in the form of rejection, partial compliance, or successful response.

Prompt Automatic Iterative Refinement (PAIR)

PAIR (Prompt Automatic Iterative Refinement) is a fully automated framework that aims to generate semantic jailbreaks by using an attacker language model to come up with a candidate jailbreak prompt for the target language model. After the model's reaction, the judge then determines if the answer meets the attack criterion or not. If not, a new prompt will be generated by the attacker..

The PAIR loop can be summarized as follows:

1. The attacker LLM generates a candidate prompt.
2. The target LLM responds to that prompt.
3. A judge scores whether the response constitutes a jailbreak.
4. The attacker uses the score and prior response to refine the next attempt.

PAIR becomes significant due to the automation of the process of jailbreak discovery on the level of prompts, without having to access weights and gradients of the model. According to Chao et al., PAIR can often discover a successful jailbreak in less than twenty queries, thereby being significantly more efficient in terms of queries than many token-level attacks . The human-understandable nature of their prompts might also partially explain their transferability to other models.

On the downside, the refinement process adopted by PAIR can be mostly described as linear, with each stream exploring only one sequence of prompt revision attempts. As a consequence, the choice of an ineffective approach will have negative ramifications for the success of further modifications.

Tree of Attacks with Pruning (TAP)

TAP is an extension of PAIR, which uses a single refinement chain and replaces it with a more general search tree. During each step of the process, an attacking LLM produces several prompts. These are prefiltered through an evaluation step, wherein those candidates that do not fit into the topic or seem to be ineffective are filtered out. Once the response from the attacked model is obtained, the remaining candidates are scored..

TAP has four major steps:

1. Branch: Generate multiple candidate refinements from the current attack prompts.
2. Prune before attack: Remove candidates that are off-topic or unlikely to succeed.
3. Attack and assess: Query the target model and evaluate the responses.
4. Prune after attack: Keep only the strongest candidates for the next round.

Branching helps in exploring, whereas pruning limits the costs of queries. This leads to TAP being more query efficient compared to executing several PAIR chains independently. According to Mehrotra et al., TAP is a black box, automated, and explainable attack framework due to its reliance on query-only oracle and generation of significant prompts along with an evaluator for guiding searches.

Iterative Self-Tuning Attack Frameworks

Another development towards automatic jailbreaks involves adversarial self-tuning models. ADV-LLM is a new framework by Sun et al. that aims to transform a pre-trained LLM into an adversarial self-tuning model capable of producing adversarial suffixes . Contrary to traditional methods where a particular attack method is used to produce examples, the new model improves its performance iteratively, through generation, evaluation, and tuning.

The principle is reminiscent of self-training approaches, where the generative model produces samples and is improved upon through fine-tuning on examples generated previously. After fine-tuning, the model becomes increasingly capable of creating adversarial suffixes which would enable jailbreaking. Finally, the model, once fully trained, will be able to produce a lot of candidate suffixes quickly, thus lowering the attack cost.

Such techniques are crucial in going beyond one-off prompting. It allows turning the attacking algorithm itself into a generative, learning model. As such, the cat-and-mouse dynamic in the context of safety alignment and adversarial evasion arises.

Cat-and-Mouse Loop in Search-Based Jailbreaking

The interaction between attack frameworks and safety defenses can be represented as a repeated feedback loop.

The interaction between attack frameworks and safety defenses can be represented as a repeated feedback loop (see fig. on the right)

This loop reveals the issue of developing one-time defenses against jailbreaks, which evolve with time. While a static guardrail will prevent jailbreaks that use specific prompt templates, search-based methods can adapt around these barriers using new phrasing, alternative framing strategies, or various suffix distributions. Self-tuning systems can also use successful attempts as future training data. This ultimately shows how jailbreak defense is an ongoing adversarial learning problem.

Implications for LLM Safety Evaluation

Examples of PAIR, TAP, BEAST, AdvPrompter, and ADV-LLM are part of a larger trend in which jailbreak attacks become more automated, more efficient, and more transferable. These attacks require fewer hand-designed prompts and help to explore multiple harmful-behavior categories at once. This makes them useful from a defensive viewpoint for helping in red-teaming and benchmark design. From a risk perspective, the same efficiency can make it cheaper to uncover failure cases.

The takeaway is that, while evaluating alignment capabilities of an AI system, a model must be considered not just as a static system, but also one capable of being challenged adaptively. Even if it resists jailbreaks based on predefined templates, it might still be susceptible to iterative refinement, branching searches, and attacks based on generating adversarial suffixes automatically. As such, evaluation benchmarks should check not just whether the system rejects harmful prompts but whether it can withstand repeated queries and other kinds of adaptive challenges.

Empirical Analysis with Standard Benchmarks

A principled approach to the analysis of jailbreaking attacks requires both standardized data sets containing malicious behaviors and automatic measures of success. Initial studies on the LLM jailbreaking task employed manual and ad-hoc red-teaming where prompts were written manually, judgments were made by humans and based on subjective assessments, and results were reported using different prompt collections, thus precluding any quantitative comparison between techniques. The community has reached an agreement on benchmark tasks that allow for fair multi-model comparisons.

Evaluation Metrics

The main quantitative metric that appears in most of the studies is attack success rate (ASR), which measures the ratio of harmful prompts for which the attack resulted in a successful response from the model that would evade all the safety measures. Usually, ASR is calculated using the keyword matching refusal string (“I cannot”, “I’m unable to”) or through an automated classification of the outputs as containing true harmful messages.

Two metrics that measure the cost of the attacks include:

• Query efficiency is measured as the number of forward passes or API calls to the model required to perform a successful jailbreak against the specific harmful behavior. In case of gradient-based attacks, e.g., GCG , hundreds of thousands of gradient evaluations might be necessary, while the goal of black-box methods like PAIR and TAP is to achieve the target within dozens of queries.
• Transferability refers to how well an optimized adversarial sample for one classifier still works on a second classifier without undergoing any further optimization. This property is important in practice because it enables an adversary to optimize samples using an open-source surrogate classifier against a commercial closed-source classifier.

There are many tradeoffs between these metrics. On one hand, a high ASR is easier to attain with large numbers of queries and when white-box gradient information is available. On the other hand, query efficiency and black-box settings have to compromise on the success rate.

Standardized Benchmark Datasets

AdvBench

The AdvBench proposed with the GCG attack by Zou et al. is made up of 520 samples of bad behaviors including the creation of malware, violent content, and misinformation. In each sample, there is a user instruction (for example, Write a tutorial on making a bomb') with the target response prefix (for example, Sure, here’s. . . ) that the attack seeks to trigger. Since it was the first of its kind and available, AdvBench was the default benchmark for early gradient-based attack papers.

HarmBench

HarmBench, proposed by Mazeika et al. , seeks to resolve the problems identified in AdvBench and other ad-hoc experiments: variability in prompt sets between different works and the absence of an automated and reproducible benchmark. Specifically, HarmBench includes 400 instances of harmful behaviors, split into four functionality classes: "standard, contextual, copyright, and multimodal," with 159 standard behaviors being held out for use as a test split for fair comparisons. Importantly, HarmBench not only features the dataset of the target behaviors but also includes an automated classifier (HarmBench-Llama-2-13B-cls), which is capable of assessing whether the generated model completion is a real example of the given harmful behavior or simply a discussion thereof. This judging approach allows for scalable and reproducible large-scale testing across multiple attacks and models without the need for manual annotation in each individual experiment. Ever since its publication, HarmBench has quickly become a de facto standard for quantitative assessment in the field.

JailbreakBench

Another benchmark called JailbreakBench, proposed by Chao et al., focuses on openness and comparability across time. The project features a selected dataset consisting of 100 policy-violating actions (JBB-Behaviors), an evolving leaderboard measuring attacks and defenses against all versions of the model and specification of the threat model. At its inception, JailbreakBench consisted of 13 jailbreaks created using the GCG and PAIR techniques on the example of Vicuna-13B-v1.5 with a GPT-4-powered judge for the ASR task. The open leaderboard design allows evolving attacks and defenses to be tracked as new methods are published.

Comparative Analysis of Attack Methods

Table 1 summarizes representative results for the five attack methods most directly relevant to this article—GCG , PAIR , TAP , BEAST, and AdvPrompter—along several dimensions: access model, approximate ASR on a common open-source target (Vicuna-7B), average queries to the target LLM per behavior, and transfer ASR to a frontier closed-source model. Numbers are drawn from the original papers; because each paper evaluates slightly different model versions and judge criteria, values should be interpreted as indicative rather than directly comparable.

Table 1: Comparison of representative jailbreak attacks across access model, attack success rate (ASR) on Vicuna-7B, query efficiency, and transferability to GPT-4. Results are drawn from original papers and benchmarked primarily on AdvBench. †AdvPrompter requires an offline training phase; inference-time queries to the target model are minimal.

Method	Access	ASR (Vicuna-7B)	Avg. Queries to Target	Transfer ASR (GPT-4)	Benchmark
GCG	White-box (gradient)	~86-99%	~256,000 steps	~54% (ensemble)	AdvBench
BEAST	White/gray-box (gradient-free)	~89%	~1 GPU-min	~12%	AdvBench
AdvPrompter	White-box (gradient during training)	Competitive	1-2 sec (post-train)	Competitive	AdvBench/HarmBench
PAIR	Black-box	~65%	~20 queries	~50%	AdvBench/JailbreakBench
TAP	Black-box	~98%	~12 queries	~90%	AdvBench

Some trends emerge from the analysis. White box gradient-based attacks provide the highest ASRs against models being directly attacked, but are very expensive computationally and, importantly, rely on knowledge of model weights, limiting the use of such attacks against closed-source models used commercially. For example, GCG. consists of optimizing the model 500 times by considering ∼512 candidate token replacements per adversarial suffix, requiring hundreds of thousands of forward passes. Still, the produced adversarial suffixes generalize across model types; the same transferred GCG attack results in an 87.9% ASR against GPT-3.5-Turbo and a 53.6% ASR against GPT-4

Gradient-free approaches to white-box attacks make up the middle ground between effectiveness and demands for resources. BEAST. relies on beam search across the space of tokens, without computing gradients, to craft adversarial suffixes in just one minute of GPU time per prompt. Under time constraints of just one minute for BEAST, an attack rate of 89% was observed against Vicuna-7B-v1.5, outperforming PAIR's 46% rate without any backpropagation through the target model

When evaluating at the black-box prompt-level and targeting GPT-4, TAP was able to find jailbreaks for 16% more prompts than PAIR while querying the target model 60% fewer times due to its parallel exploration and pruning structure. Once trained, AdvPrompter generates a new adversarial suffix in 1–2 seconds, compared to the 20 minutes to several hours required by methods such as GCG and AutoDAN for each new suffix. This reveals the effectiveness of the training-amortized approach in multi-shot attacks.

Systemic Implications

These benchmarks provide evidence of a trajectory with important consequences for the alignment research community and beyond. Three developments stand out in particular.

Escalation of attacks’ efficiency

From tens of thousands of gradient steps in the case of GCG to around 20 query attempts in PAIR and TAP, and fewer than 12 in AdvPrompter's setting, down to the time needed to complete sub-second inference in AdvPrompter, we see a marked trend toward the increasing efficiency of attacks. With lower costs involved in conducting a jailbreak, even non-expert attackers will face less difficulty doing so. It is well recognized that this represents a positive feedback loop, with improvements in safety aligned approaches always being countered by increasingly powerful attacks.

The problem of attack transferability

Benchmark performance demonstrates clearly that attack prompts generated for open-source surrogates can successfully transfer to close-source commercial models. This presents challenges for any strategy of defense against adversarial inputs involving API-only deployment; indeed, attacks could be crafted by an attacker who gains access to the surrogate model's open-source weights. The motivation behind designing HarmBench in a way that allows for such an assessment lies behind HarmBench’s evaluation procedure for multiple models.

Limits of current alignment methods.

The success of well-performing attacks on highly fine-tuned models for safety, including RLHF and constitutional AI models, indicates that existing alignment procedures fail to induce robustly safe behavior adversarially. Safety-trained models consistently reject dangerous prompts, but benchmark data shows that this can be easily defeated using adversarial manipulation of the input prompt. The problem formulation in this case parallels “competing objectives,” where safety is not sufficiently ingrained into the system and thus represents an acquired behavior which can be exploited through adversarial means. Metrics used for evaluating safety, such as those from HarmBench and JailbreakBench, allow measuring progress towards more robust alignment, but the available data suggests there is still significant work to be done