✨ NEW: stable heading anchors: shared slug module, presets, HTML ids, explicit-id priority

docs/syntax/cross-referencing.md

@@ -17,6 +17,12 @@ There are three primary ways to create targets:
 2. Annotating a syntax block/inline/span with an `{#id}` attribute (using the [attrs_block](#syntax/attributes/block) and [attrs_inline](#syntax/attributes/inline) extensions)
 3. Adding a `name` option to a directive
 
+```{versionchanged} 5.2.0
+An explicit target placed before a section heading now becomes the section's
+primary HTML id, used by tocs, permalinks and `objects.inv`; the implicit
+heading id is kept as a secondary anchor, so existing fragment links keep working.
+```
+
 ::::{myst-example}
 
 (heading-target)=
@@ -69,7 +75,7 @@ Headings within documents can also be assigned an implicit target,
 by setting the `myst_heading_anchors` configuration option.
 This is should be set to an integer, between 1 and 6, indicating the depth of headings to assign targets to.
 
-The anchor "slugs" are created according to the [GitHub implementation](https://github.com/Flet/github-slugger): heading titles are lower cased, punctuation is removed, spaces are replaced with `-`, and uniqueness is enforced by suffix enumeration.
+By default the anchor "slugs" are created according to the [GitHub implementation](https://github.com/Flet/github-slugger): heading titles are lower cased, punctuation is removed, spaces are replaced with `-`, and uniqueness is enforced by suffix enumeration (`x`, `x-1`, `x-2`, ...). The slug function is configurable via `myst_heading_slug_func` (e.g. the `"gitlab"` preset); see [auto-generated header anchors](#syntax/header-anchors).
 
 For example, using `myst_heading_anchors = 2`:
 
@@ -157,10 +163,15 @@ By default, MyST will resolve link destinations according to the following rules
 
    {style=lower-roman}
    1. First, explicit targets in the same file are searched for, if not found
-   2. Then, implicit targets in the same file are searched for, if not found
+   2. Then, implicit [heading anchors](#syntax/header-anchors) in the same file are searched for, if not found
    3. Then, explicit targets across the whole project are searched for, if not found
    4. Then, intersphinx references are searched for, if not found
-   5. A warning is emitted and the destination is left as an external link.
+   5. Then, any other anchor that exists in the same file is used (e.g. a heading beyond the `myst_heading_anchors` depth), if not found
+   6. A warning is emitted and the destination is left as an external link.
+
+   ```{versionchanged} 5.2.0
+   Step v was added: such links previously emitted a warning.
+   ```
 
 :::{note}
 Local file path resolution and cross-project references are not available in [single page builds](#myst-docutils)

docs/syntax/optional.md

@@ -528,27 +528,43 @@ The paths to other files should be relative to the current file:
 
 ### Anchor slug structure
 
-The anchor "slugs" created aim to follow the [GitHub implementation](https://github.com/Flet/github-slugger):
+By default the anchor "slugs" created aim to follow the [GitHub implementation](https://github.com/Flet/github-slugger):
 
 - lower-case text
 - remove punctuation
 - replace spaces with `-`
-- enforce uniqueness *via* suffix enumeration `-1`
+- enforce uniqueness *via* suffix enumeration: duplicates of `x` become `x`, `x-1`, `x-2`, ... (the base slug is never mutated)
 
-To change the slug generation function, set `myst_heading_slug_func` in your `conf.py` to a function that accepts a string and returns a string.
+The slug generation function is selected with the `myst_heading_slug_func` option in your `conf.py`.
+The recommended form is a named preset:
 
-:::{versionadded} 0.19.0
-`myst_heading_slug_func` can now also be set to a string,
-which will be interpreted as an import path to a function,
-e.g. `myst_heading_slug_func = "mypackage.mymodule.slugify"`.
+```python
+myst_heading_slug_func = "github"  # the default; also available: "gitlab"
+```
+
+The `"gitlab"` preset follows [GitLab's algorithm](https://docs.gitlab.com/ee/user/markdown.html#heading-ids-and-links) (which, unlike GitHub, strips surrounding whitespace, squeezes repeated hyphens and prefixes digit-only slugs with `anchor-`).
+
+:::{versionchanged} 5.2.0
+`myst_heading_slug_func` now accepts the preset names `"github"` (default) and `"gitlab"`.
+
+As a legacy form, it can still be set to an import path string
+(e.g. `myst_heading_slug_func = "mypackage.mymodule.slugify"`, added in 0.19.0)
+or, in `conf.py`, directly to a callable that accepts a string and returns a string.
+:::
+
+:::{note}
+Anchor slugs are additionally emitted as (secondary) HTML `id` attributes on the rendered headings, so that `#slug` fragments resolve in the published HTML.
+The primary docutils id is left first and unchanged, so previously published fragments keep working.
+Set `myst_heading_anchors_html_ids = False` to disable this and emit only the docutils id.
 :::
 
 ### Inspect the links that will be created
 
-You can inspect the links that will be created using the command-line tool:
+You can inspect the links that will be created using the command-line tool
+(use `--slug-func` to preview a preset, matching `myst_heading_slug_func`):
 
 ```console
-$ myst-anchors -l 2 docs/syntax/optional.md
+$ myst-anchors -l 2 --slug-func github docs/syntax/optional.md
 <h1 id="optional-myst-syntaxes"></h1>
 <h2 id="admonition-directives"></h2>
 <h2 id="auto-generated-header-anchors"></h2>

myst_parser/cli.py

@@ -7,6 +7,7 @@
 
 from myst_parser.config.main import MdParserConfig
 from myst_parser.parsers.mdit import create_md_parser
+from myst_parser.slugs import SLUG_PRESETS
 
 
 def print_anchors(args=None):
@@ -29,9 +30,17 @@ def print_anchors(args=None):
     arg_parser.add_argument(
         "-l", "--level", type=int, default=2, help="Maximum heading level."
     )
+    arg_parser.add_argument(
+        "--slug-func",
+        choices=sorted(SLUG_PRESETS),
+        default="github",
+        help="Slug preset for anchor ids (matches myst_heading_slug_func presets).",
+    )
     args = arg_parser.parse_args(args)
     parser = create_md_parser(MdParserConfig(), RendererHTML)
-    parser.use(anchors_plugin, max_level=args.level)
+    parser.use(
+        anchors_plugin, max_level=args.level, slug_func=SLUG_PRESETS[args.slug_func]
+    )
 
     def _filter_plugin(state: StateCore) -> None:
         state.tokens = [

myst_parser/config/main.py

@@ -138,19 +138,30 @@ def check_inventories(_: "MdParserConfig", field: dc.Field, value: Any) -> None:
 def check_heading_slug_func(
     inst: "MdParserConfig", field: dc.Field, value: Any
 ) -> None:
-    """Check that the heading_slug_func is a callable."""
+    """Check that the heading_slug_func is a preset name or a callable."""
     if value is None:
         return
     if isinstance(value, str):
-        # attempt to load the function as a python import
-        try:
-            module_path, function_name = value.rsplit(".", 1)
-            mod = import_module(module_path)
-            value = getattr(mod, function_name)
-        except ImportError as exc:
+        from myst_parser.slugs import SLUG_PRESETS
+
+        if value in SLUG_PRESETS:
+            value = SLUG_PRESETS[value]
+        elif "." in value:
+            # attempt to load the function as a python import (legacy form)
+            try:
+                module_path, function_name = value.rsplit(".", 1)
+                mod = import_module(module_path)
+                value = getattr(mod, function_name)
+            except (ImportError, AttributeError, ValueError) as exc:
+                raise TypeError(
+                    f"'{field.name}' could not be loaded from string: {value!r}"
+                ) from exc
+        else:
             raise TypeError(
-                f"'{field.name}' could not be loaded from string: {value!r}"
-            ) from exc
+                f"'{field.name}' is not a known preset "
+                f"({', '.join(sorted(SLUG_PRESETS))}) "
+                f"or a python import path: {value!r}"
+            )
         setattr(inst, field.name, value)
     if not callable(value):
         raise TypeError(f"'{field.name}' is not callable: {value!r}")
@@ -301,11 +312,21 @@ def __repr__(self) -> str:
         metadata={
             "validator": check_heading_slug_func,
             "help": (
-                "Function for creating heading anchors, "
-                "or a python import path e.g. `my_package.my_module.my_function`"
+                "Function for creating heading anchors: "
+                'a preset name ("github", "gitlab"), '
+                "or (legacy) a python import path "
+                "e.g. `my_package.my_module.my_function`"
             ),
             "global_only": True,
-            "doc_type": "None | Callable[[str], str] | str",
+            "doc_type": "None | str | Callable[[str], str]",
+        },
+    )
+
+    heading_anchors_html_ids: bool = dc.field(
+        default=True,
+        metadata={
+            "validator": instance_of(bool),
+            "help": "Also emit heading anchor slugs as (additional) HTML ids",
         },
     )
 

myst_parser/mdit_to_docutils/base.py

@@ -7,7 +7,14 @@
 import os
 import posixpath
 import re
-from collections.abc import Callable, Iterable, Iterator, MutableMapping, Sequence
+from collections.abc import (
+    Callable,
+    Container,
+    Iterable,
+    Iterator,
+    MutableMapping,
+    Sequence,
+)
 from contextlib import contextmanager, suppress
 from datetime import date, datetime
 from types import ModuleType
@@ -51,6 +58,7 @@
     MarkupError,
     parse_directive_text,
 )
+from myst_parser.slugs import github_slugify, unique_slug
 from myst_parser.warnings_ import MystWarnings, create_warning
 
 from .html_to_nodes import html_to_nodes
@@ -243,6 +251,11 @@ def _render_finalise(self) -> None:
                 self._heading_slugs
             )
 
+        # ensure this setting is set for the later slug-id transform
+        self.document.settings.myst_heading_anchors_html_ids = (
+            self.md_config.heading_anchors_html_ids
+        )
+
         # ensure these settings are set for later footnote transforms
         self.document.settings.myst_footnote_transition = (
             self.md_config.footnote_transition
@@ -859,6 +872,10 @@ def generate_heading_target(
             )
         else:
             node["slug"] = slug
+            # note: the slug is additionally emitted as a (secondary) HTML id
+            # by the `AddSlugIds` transform, which runs only after *all*
+            # docutils/sphinx id assignment, so that it cannot claim an id
+            # another element would otherwise receive
             self._heading_slugs[slug] = (node.line, node["ids"][0], implicit_text)
 
     def render_heading(self, token: SyntaxTreeNode) -> None:
@@ -2055,40 +2072,27 @@ def clean_astext(node: nodes.Element) -> str:
     return node.astext()
 
 
-_SLUGIFY_CLEAN_REGEX = re.compile(r"[^\w\u4e00-\u9fff\- ]")
-
-
 def default_slugify(title: str) -> str:
-    """Default slugify function.
-
-    This aims to mimic the GitHub Markdown format, see:
+    """Default slugify function (GitHub-style).
 
-    - https://github.com/jch/html-pipeline/blob/master/lib/html/pipeline/toc_filter.rb
-    - https://gist.github.com/asabaylus/3071099
+    .. deprecated:: 5.2.0
+        Use :func:`myst_parser.slugs.github_slugify` instead.
     """
-    return _SLUGIFY_CLEAN_REGEX.sub("", title.lower().replace(" ", "-"))
+    return github_slugify(title)
 
 
 def compute_unique_slug(
     token_tree: SyntaxTreeNode,
-    slugs: Iterable[str],
+    slugs: Container[str],
     slug_func: None | Callable[[str], str] = None,
 ) -> str:
-    """Compute the slug for a token.
-
-    This directly mirrors the logic in `mdit_py_plugins.anchors_plugin`
-    """
-    slug_func = default_slugify if slug_func is None else slug_func
+    """Compute the slug for a heading token, unique against existing slugs."""
+    slug_func = github_slugify if slug_func is None else slug_func
     tokens = token_tree.to_tokens()
     inline_token = tokens[1]
     title = "".join(
         child.content
         for child in (inline_token.children or [])
         if child.type in ["text", "code_inline"]
     )
-    slug = slug_func(title)
-    i = 1
-    while slug in slugs:
-        slug = f"{slug}-{i}"
-        i += 1
-    return slug
+    return unique_slug(slug_func(title), slugs)