Fix outdated examples, broken links, and stale references across docs

docs/0_get_started/command_line_usage.md

@@ -102,9 +102,9 @@ Here are some models from transformers that have worked well for us:
 - `roberta-base` 
 - `xlnet-base-cased`
 
-## Evaluating Models with `textattack eval-model`
+## Evaluating Models with `textattack eval`
 
-Any TextAttack-compatible model can be evaluated using `textattack eval-model`. TextAttack-trained models can be evaluated using `textattack eval --num-examples <num-examples> --model /path/to/trained/model/`
+Any TextAttack-compatible model can be evaluated using `textattack eval`. TextAttack-trained models can be evaluated using `textattack eval --num-examples <num-examples> --model /path/to/trained/model/`
 
 ## Other Commands
 

docs/0_get_started/installation.md

@@ -1,7 +1,7 @@
 Installation
 ==============
 
-To use TextAttack, you must be running Python 3.6 or above. A CUDA-compatible GPU is optional but will greatly improve speed. 
+To use TextAttack, you must be running Python 3.9 or above. A CUDA-compatible GPU is optional but will greatly improve speed. 
 
 We recommend installing TextAttack in a virtual environment (check out this [guide](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/)).
 
@@ -67,7 +67,7 @@ Besides, we highly recommend you to use virtual environment for textattack use,
 see [information here](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#removing-an-environment). Here is one conda example: 
 
 ```bash
-conda create -n textattackenv python=3.8
+conda create -n textattackenv python=3.9
 conda activate textattackenv
 conda env list
 ```

docs/0_get_started/quick_api_tour.rst

@@ -14,14 +14,14 @@ Let us attack a BERT model fine-tuned for sentimental classification task. We ar
     >>> tokenizer = transformers.AutoTokenizer.from_pretrained("textattack/bert-base-uncased-imdb")
 
 
-TextAttack requires both the model and the tokenizer to be wrapped by a :class:`~transformers.models.wrapper.ModelWrapper` class that implements the forward pass operation given a list of input texts. For models provided by Transformers library, we can also simply use :class:`~transformers.models.wrapper.HuggingFaceModelWrapper` class which implements both the forward pass and tokenization.
+TextAttack requires both the model and the tokenizer to be wrapped by a :class:`~textattack.models.wrappers.ModelWrapper` class that implements the forward pass operation given a list of input texts. For models provided by Transformers library, we can also simply use :class:`~textattack.models.wrappers.HuggingFaceModelWrapper` class which implements both the forward pass and tokenization.
 
 .. code-block::
 
     >>> import textattack
     >>> model_wrapper = textattack.models.wrappers.HuggingFaceModelWrapper(model, tokenizer)
 
-Next, let's build the attack that we want to use. TextAttack provides prebuilt attacks in the form of :class:`~transformers.attack_recipes.AttackRecipe`. For this example, we will use :ref:TextFooler attack 
+Next, let's build the attack that we want to use. TextAttack provides prebuilt attacks in the form of :class:`~textattack.attack_recipes.AttackRecipe`. For this example, we will use the TextFooler attack.
 
 
 .. code-block::

docs/1start/FAQ.md

@@ -145,7 +145,7 @@ The `attack_one` method in an `Attack` takes as input an `AttackedText`, and out
 We formulate an attack as consisting of four components: a **goal function** which determines if the attack has succeeded, **constraints** defining which perturbations are valid, a **transformation** that generates potential modifications given an input, and a **search method** which traverses through the search space of possible perturbations. The attack attempts to perturb an input text such that the model output fulfills the goal function (i.e., indicating whether the attack is successful) and the perturbation adheres to the set of constraints (e.g., grammar constraint, semantic similarity constraint). A search method is used to find a sequence of transformations that produce a successful adversarial example.
 
 
-This modular design unifies adversarial attack methods into one system, enables us to easily assemble attacks from the literature while re-using components that are shared across attacks. We provides clean, readable implementations of 16 adversarial attack recipes from the literature (see [our tool paper](https://arxiv.org/abs/2005.05909) and [our benchmark search paper](https://arxiv.org/abs/2009.06368)). For the first time, these attacks can be benchmarked, compared, and analyzed in a standardized setting.
+This modular design unifies adversarial attack methods into one system, enables us to easily assemble attacks from the literature while re-using components that are shared across attacks. We provides clean, readable implementations of 24 adversarial attack recipes from the literature (see [our tool paper](https://arxiv.org/abs/2005.05909) and [our benchmark search paper](https://arxiv.org/abs/2009.06368)). For the first time, these attacks can be benchmarked, compared, and analyzed in a standardized setting.
 
 
 

docs/1start/attacks4Components.md

@@ -2,7 +2,7 @@
 
 To unify adversarial attack methods into one system, We formulate an attack as consisting of four components: a **goal function** which determines if the attack has succeeded, **constraints** defining which perturbations are valid, a **transformation** that generates potential modifications given an input, and a **search method** which traverses through the search space of possible perturbations. The attack attempts to perturb an input text such that the model output fulfills the goal function (i.e., indicating whether the attack is successful) and the perturbation adheres to the set of constraints (e.g., grammar constraint, semantic similarity constraint). A search method is used to find a sequence of transformations that produce a successful adversarial example.
 
-This modular design enables us to easily assemble attacks from the literature while re-using components that are shared across attacks. TextAttack provides clean, readable implementations of 16 adversarial attacks from the literature. For the first time, these attacks can be benchmarked, compared, and analyzed in a standardized setting.
+This modular design enables us to easily assemble attacks from the literature while re-using components that are shared across attacks. TextAttack provides clean, readable implementations of 24 adversarial attacks from the literature. For the first time, these attacks can be benchmarked, compared, and analyzed in a standardized setting.
 
 - Two examples showing four components of two SOTA attacks
   ![two-categorized-attacks](/_static/imgs/intro/01-categorized-attacks.png)
@@ -44,7 +44,7 @@ A `SearchMethod` takes as input an initial `GoalFunctionResult` and returns a fi
 
 ### Four components in Attack Recipes we have implemented
 
-- TextAttack provides clean, readable implementations of 16 adversarial attacks from the literature.
+- TextAttack provides clean, readable implementations of 24 adversarial attacks from the literature.
 
 - To run an attack recipe: `textattack attack --recipe [recipe_name]`
 
@@ -62,6 +62,14 @@ A `SearchMethod` takes as input an initial `GoalFunctionResult` and returns a fi
 <tbody>
   <tr><td style="text-align: center;" colspan="6"><strong><br>Attacks on classification tasks, like sentiment classification and entailment:<br></strong></td></tr>
 
+<tr class="odd">
+<td style="text-align: left;"><code>a2t</code> <span class="citation" data-cites="yoo2021a2t"></span></td>
+<td style="text-align: left;"><sub>Untargeted Classification</sub></td>
+<td style="text-align: left;"><sub>Part-of-speech match, Max modification rate, SBERT sentence encoding cosine similarity, Word embedding distance</sub></td>
+<td style="text-align: left;"><sub>Counter-fitted word embedding swap (or BERT Masked Token Prediction, in the `mlm` variant)</sub></td>
+<td style="text-align: left;"><sub>Greedy-WIR (gradient)</sub></td>
+<td ><sub>Attack tuned for use in adversarial training, from (["Towards Improving Adversarial Training of NLP Models" (Yoo et al., 2021)](https://arxiv.org/abs/2109.00544))</sub></td>
+</tr>
 <tr class="even">
 <td style="text-align: left;"><code>alzantot</code>  <span class="citation" data-cites="Alzantot2018GeneratingNL Jia2019CertifiedRT"></span></td>
 <td style="text-align: left;"><sub>Untargeted {Classification, Entailment}</sub></td>
@@ -142,6 +150,14 @@ A `SearchMethod` takes as input an initial `GoalFunctionResult` and returns a fi
 <td style="text-align: left;"><sub>Greedy-WIR</sub></td>
 <td ><sub>Greedy attack with word importance ranking , Reducing the input while maintaining the prediction through word importance ranking (["Pathologies of Neural Models Make Interpretation Difficult" (Feng et al., 2018)](https://arxiv.org/pdf/1804.07781.pdf))</sub></td>
 </tr>
+<tr class="even">
+<td style="text-align: left;"><code>leap</code> <span class="citation" data-cites="ma2023leap"></span></td>
+<td style="text-align: left;"><sub>Untargeted Classification</sub></td>
+<td style="text-align: left;"><sub>Max modification rate, Stopword modification</sub></td>
+<td style="text-align: left;"><sub>WordNet-based synonym swap</sub></td>
+<td style="text-align: left;"><sub>Particle Swarm Optimization (Levy-flight/adaptive-inertia variant)</sub></td>
+<td ><sub>Levy-flight/adaptive-inertia variant of the `pso` search method, from (["LEAP: Efficient and Automated Test Method for NLP Software" (Ma et al., 2023)](https://arxiv.org/abs/2308.11284))</sub></td>
+</tr>
 <tr class="odd">
 <td style="text-align: left;"><code>kuleshov</code> <span class="citation" data-cites="Kuleshov2018AdversarialEF"></span></td>
 <td style="text-align: left;"><sub>Untargeted Classification</sub></td>
@@ -213,6 +229,7 @@ A `SearchMethod` takes as input an initial `GoalFunctionResult` and returns a fi
 </tr>
 
 <tr><td style="text-align: center;" colspan="6"><strong><br>General: <br></strong></td></tr>
+<tr><td style="text-align: left;" colspan="6"><sub>Every other recipe here is locked to one task and one goal function, which is why the table splits into "classification" vs. "sequence-to-sequence" sections. <code>bad-characters</code> is different: its <code>build()</code> takes a <code>goal_function_type</code> argument and can be configured to attack classification models, NER models, or seq2seq/translation models. Since it spans both task types (and NER, which is neither), it doesn't fit either section above, so it's listed here under "General" instead.</sub></td></tr>
 
 <tr class="odd">
 <td style="text-align: left;"><code>bad-characters</code> <span class="citation" data-cites=""></span></td>
@@ -223,6 +240,33 @@ A `SearchMethod` takes as input an initial `GoalFunctionResult` and returns a fi
 <td><sub>Uses imperceptible character-level perturbations including homoglyph substitutions, Unicode reordering, deletions, and invisibles. Based on (["Bad Characters: Imperceptible NLP Attacks" (Boucher et al., 2021)](https://arxiv.org/abs/2106.09898)).</sub></td>
 </tr>
 
+<tr><td style="text-align: center;" colspan="6"><strong><br>Multi-lingual attacks on non-English classification models (Python API only, no CLI recipe name -- see <a href="https://github.com/QData/TextAttack/issues/423">#423</a>):<br></strong></td></tr>
+
+<tr class="even">
+<td style="text-align: left;"><code>FrenchRecipe</code></td>
+<td style="text-align: left;"><sub>Untargeted Classification</sub></td>
+<td style="text-align: left;"><sub>Repeat modification, Stopword modification (French)</sub></td>
+<td style="text-align: left;"><sub>WordNet synonym swap, location swap, name swap (French)</sub></td>
+<td style="text-align: left;"><sub>Greedy-WIR</sub></td>
+<td ><sub>Contextualized-perturbation attack adapted for French NLP models.</sub></td>
+</tr>
+<tr class="odd">
+<td style="text-align: left;"><code>SpanishRecipe</code></td>
+<td style="text-align: left;"><sub>Untargeted Classification</sub></td>
+<td style="text-align: left;"><sub>Repeat modification, Stopword modification (Spanish)</sub></td>
+<td style="text-align: left;"><sub>WordNet synonym swap, location swap, name swap (Spanish)</sub></td>
+<td style="text-align: left;"><sub>Greedy-WIR</sub></td>
+<td ><sub>Contextualized-perturbation attack adapted for Spanish NLP models.</sub></td>
+</tr>
+<tr class="even">
+<td style="text-align: left;"><code>ChineseRecipe</code></td>
+<td style="text-align: left;"><sub>Untargeted Classification</sub></td>
+<td style="text-align: left;"><sub>Repeat modification, Stopword modification (Chinese)</sub></td>
+<td style="text-align: left;"><sub>HowNet word swap, Masked-LM word swap, Morphonym &amp; Homophone character swap</sub></td>
+<td style="text-align: left;"><sub>Greedy-WIR (weighted-saliency)</sub></td>
+<td ><sub>Contextualized-perturbation attack adapted for Chinese NLP models.</sub></td>
+</tr>
+
 </tbody>
 </font>
 </table>

docs/1start/support.md

@@ -189,7 +189,7 @@ Follow these steps to start contributing:
    documentation for a new code file should just be two lines. Our docs will 
    automatically generate from the comments you added to your code. If you're 
    adding an attack recipe, add a reference in `attack_recipes.rst`. 
-   If you're adding a transformation, add a reference in `transformation.rst`, etc. 
+   If you're adding a transformation, add a reference in `docs/api/transformations.rst`, etc. 
 
    You can build the docs and view the updates using `make docs`. If you're 
    adding a tutorial or something where you want to update the docs multiple

docs/1start/what_is_an_adversarial_attack.md

@@ -53,7 +53,7 @@ Naturally, many have wondered about what adversarial examples for NLP models mig
 
 ![Two definitions of NLP adversarial examples](/_static/imgs/intro/mr_aes.png)
 
-*Two different ideas of adversarial examples in NLP. These results were generated using TextAttack on an LSTM trained on the Rotten Tomatoes Movie Review sentiment classification dataset. These are *real* adversarial examples, generated using the DeepWordBug and TextFooler attacks. To generate them yourself, after installing TextAttack, run ‘textattack attack — model lstm-mr — num-examples 1 — recipe RECIPE — num-examples-offset 19’ where RECIPE is ‘deepwordbug’ or ‘textfooler’.*
+*Two different ideas of adversarial examples in NLP. These results were generated using TextAttack on an LSTM trained on the Rotten Tomatoes Movie Review sentiment classification dataset. These are *real* adversarial examples, generated using the DeepWordBug and TextFooler attacks. To generate them yourself, after installing TextAttack, run ‘textattack attack --model lstm-mr --num-examples 1 --recipe RECIPE --num-examples-offset 19’ where RECIPE is ‘deepwordbug’ or ‘textfooler’.*
 
 Because two text sequences are never indistinguishable, researchers have proposed various alternative definitions for adversarial examples in NLP. We find it useful to group adversarial attacks based on their chosen definitions of adversarial examples. 
 

docs/3recipes/attack_recipes.rst

@@ -149,8 +149,8 @@ Attacks on classification models
 Attacks on sequence-to-sequence models
 ############################################
 
-17. MORPHEUS (It’s Morphin’ Time! Combating Linguistic Discrimination with Inflectional Perturbations)
-18. Seq2Sick (Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples)
+18. MORPHEUS (It’s Morphin’ Time! Combating Linguistic Discrimination with Inflectional Perturbations)
+19. Seq2Sick (Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples)
 
 
 .. automodule:: textattack.attack_recipes.morpheus_tan_2020
@@ -166,7 +166,14 @@ Attacks on sequence-to-sequence models
 General
 ############################################
 
-19. BadCharacters (Bad Characters: Imperceptible NLP Attacks)
+Unlike the recipes above, which are each locked to one task and one goal
+function, ``BadCharacters2021`` takes a ``goal_function_type`` argument and
+can be configured to attack classification models, NER models, or
+seq2seq/translation models. Since it spans both of the sections above (and
+NER, which is neither), it's listed here instead of being duplicated or
+assigned to one arbitrarily.
+
+20. BadCharacters (Bad Characters: Imperceptible NLP Attacks)
 
 
 .. automodule:: textattack.attack_recipes.bad_characters_2021
@@ -181,9 +188,9 @@ TextAttack also includes recipes for non-English classification models, each a
 language-specific adaptation of a contextualized-perturbation attack (see
 https://github.com/QData/TextAttack/issues/423):
 
-20. FrenchRecipe -- Attack French Recipe (Contextualized Perturbation for French NLP Adversarial Attack)
-21. SpanishRecipe -- Attack Spanish Recipe (Contextualized Perturbation for Spanish NLP Adversarial Attack)
-22. ChineseRecipe -- Attack Chinese Recipe (Contextualized Perturbation for Chinese NLP Adversarial Attack)
+21. FrenchRecipe -- Attack French Recipe (Contextualized Perturbation for French NLP Adversarial Attack)
+22. SpanishRecipe -- Attack Spanish Recipe (Contextualized Perturbation for Spanish NLP Adversarial Attack)
+23. ChineseRecipe -- Attack Chinese Recipe (Contextualized Perturbation for Chinese NLP Adversarial Attack)
 
 
 .. automodule:: textattack.attack_recipes.french_recipe