[BUG] putMapping fails with HTTP 400 on OpenSearch 3.4.0 — ES mapping format incompatibility

[fabrizzio-dotCMS]

Summary

When dotCMS boots in Phase 3 (OS only), all content-type putMapping calls to OpenSearch 3.4.0 fail with HTTP 400. As a result, 0 documents are indexed in OS despite the indices being physically created.

Observed behavior

• OS indices are created successfully at startup
• All putMapping calls return HTTP 400 from OS
• Working and live indices remain at 0 documents
• GET /api/v1/esindex/ shows OS indices with health: null, docs: 0

Root cause

The mapping JSON generated by ESMappingAPIImpl / MappingHelper targets Elasticsearch 7.x format. IndexMappingRestOperationsOS (or MappingOperationsOS.putMapping()) passes this payload unchanged to OpenSearch 3.4.0, which rejects it.

Known incompatibilities between ES 7.x and OS 3.x mapping formats include:

• _type field mappings (removed in OS 3.x)
• include_type_name parameter handling
• Some field type aliases

Fix

Audit the mapping JSON produced by MappingHelper / ESMappingAPIImpl.getMapping() for OS 3.4.0 compatibility. Add required format transformations in the OS-specific mapping path (IndexMappingRestOperationsOS) before submitting to OS.

Start by capturing the exact 400 response body from OS to identify which fields are being rejected.

Impact

• Phase 3 (OS only) is completely non-functional — no content can be indexed
• Blocks full migration validation

Discovered via

QA test run — Phase 3 boot against dotcms/dotcms:latest with single-node-os-migration stack (OS 3.4.0).

Related

• Refs [TASK] Migrate ESMappingAPIImpl #34933
• PR task(Migrate ESMappingAPIImpl) Refs #34933 #35289

[fabrizzio-dotCMS]

Reproduction verified (OS 3.4.0 — no dotCMS instance needed)

Root cause: `addCustomMapping` generates explicit field-level `"analyzer":"my_analyzer"` mappings for text fields. OS 3.4.0 rejects these with HTTP 400 when the index was created without defining `my_analyzer` in its analysis settings.

---

Reproduction steps

Requirement: OS 3.4.0 running at `localhost:9201` (the `os-migration` Docker stack is sufficient).

```bash

Step 1 — Create OS index WITHOUT custom analyzers

(simulates OSIndexAPIImpl.clearIndex() fallback path or silent settings load failure)

curl -X PUT "http://localhost:9201/dotcms_repro"
-H "Content-Type: application/json"
-d '{"settings":{"number_of_shards":1,"number_of_replicas":0,"mapping":{"total_fields":{"limit":10000}}}}'

Step 2 — PUT os-content-mapping.json (dynamic templates) → HTTP 200 ✓

Dynamic templates do NOT validate analyzer existence at PUT time

curl -X PUT "http://localhost:9201/dotcms_repro/_mapping"
-H "Content-Type: application/json"
-d @dotCMS/src/main/resources/os-content-mapping.json

Step 3 — PUT field-level mapping with explicit my_analyzer → HTTP 400 ✗ (THE BUG)

This is the exact JSON generated by ESMappingUtilHelper.putContentTypeMapping()

curl -X PUT "http://localhost:9201/dotcms_repro/_mapping"
-H "Content-Type: application/json"
-d '{"properties":{"webpagecontent":{"properties":{"title":{"type":"text","analyzer":"my_analyzer"},"title_dotraw":{"type":"keyword","ignore_above":8191}}}}}'

→ {"error":{"reason":"analyzer [my_analyzer] has not been configured in mappings"},"status":400}

Step 4 — Same PUT on index created WITH os-content-settings.json → HTTP 200 ✓

SETTINGS=$(cat dotCMS/src/main/resources/os-content-settings.json)
curl -X PUT "http://localhost:9201/dotcms_repro_good" -H "Content-Type: application/json" -d "{"settings":$SETTINGS}"
curl -X PUT "http://localhost:9201/dotcms_repro_good/_mapping"
-H "Content-Type: application/json"
-d '{"properties":{"webpagecontent":{"properties":{"title":{"type":"text","analyzer":"my_analyzer"},"title_dotraw":{"type":"keyword","ignore_above":8191}}}}}'

→ {"acknowledged":true}

Cleanup

curl -X DELETE "http://localhost:9201/dotcms_repro"
curl -X DELETE "http://localhost:9201/dotcms_repro_good"
```

---

Two triggering paths

Path A — OSIndexAPIImpl.clearIndex() (line 199)

```java
final CreateIndexStatus res = createIndex(indexName, getDefaultIndexSettings(), 0);
```

getDefaultIndexSettings() looks for opensearch-content-settings.json — the file does not exist in the classpath. The hardcoded fallback has no analysis block → index is created without my_analyzer.

Path B — ContentletIndexOperationsOS.createContentIndex() (lines 211–214)

```java
try {
settings = JsonUtil.getJsonFileContentAsString(OS_SETTINGS_FILE);
} catch (Exception e) {
Logger.error(this, "cannot load " + OS_SETTINGS_FILE + ", skipping", e);
}
// If os-content-settings.json is absent from the classpath → settings = null
// → index created without analyzers
```

If os-content-settings.json is missing from the classpath (e.g. a Docker image built before PR #35103 was merged), the exception is silently swallowed and the index is created without custom analyzers.

---

Why dynamic templates succeed but field mappings fail

os-content-mapping.json uses dynamic templates — OS only validates template syntax at mapping time, not whether the referenced analyzer actually exists. The HTTP 400 only fires when addCustomMapping → ESMappingUtilHelper.getMappingForField() sends direct field-level "analyzer":"my_analyzer" mappings (generated for TextField, TextAreaField, WysiwygField, etc.).

---

Fix locations

1. OSIndexAPIImpl.getDefaultIndexSettings() (dotCMS/src/main/java/com/dotcms/content/index/opensearch/OSIndexAPIImpl.java, line 458): should load os-content-settings.json as the fallback file since opensearch-content-settings.json does not exist.

2. ContentletIndexOperationsOS.createContentIndex() (dotCMS/src/main/java/com/dotcms/content/index/opensearch/ContentletIndexOperationsOS.java, lines 211–214): the silent catch on settings load failure should either propagate the error or at minimum prevent the subsequent putMapping call when no analyzers are available.

[github-actions]

PRs linked to this issue

• fix(opensearch): ensure OS indices always created with custom analyzers — fixes #35305 by @fabrizzio-dotCMS

PRs linked to this issue

• fix(opensearch): ensure OS indices always created with custom analyzers — fixes #35305 by @fabrizzio-dotCMS
• fix(opensearch): ensure OS indices always created with custom analyzers — fixes #35305 by @fabrizzio-dotCMS