edac: support the DIMM/rank sysfs layout (skx_edac / i10nm_edac / sb_edac) in addition to csrow*/chN

[at-blacknight]

Summary

The EDAC collector currently reads memory error counts exclusively from the
csrow*/chN sysfs layout. After #3720 (per-channel metrics with dimm_label),
this is mc*/csrow*/ch*_ce_count etc. However, a large class of modern Intel
memory controller drivers do not populate csrow* at all — they register
error counts in the DIMM/MEM layer instead. On those systems the EDAC collector
silently reports nothing at csrow/channel granularity.

This is a follow-up to the review discussion on #3656.

The two layouts

The EDAC core exposes sub-controller error counts in one of two mutually
exclusive representations, depending on the driver's registration model and the
CONFIG_EDAC_LEGACY_SYSFS kernel option:

• Legacy csrow layout: mc*/csrow*/chN_ce_count, chN_dimm_label, … —
  what the collector reads today.

• DIMM layout (DIMM-model drivers: sb_edac from Sandy/Ivy-Bridge-EP
  onward, skx_edac, i10nm_edac, and others): no usable csrow* entries.
  Instead the controller exposes per-DIMM directories:

  /sys/devices/system/edac/mc/mc0/dimm0/dimm_label
  /sys/devices/system/edac/mc/mc0/dimm0/dimm_ce_count
  /sys/devices/system/edac/mc/mc0/dimm0/dimm_ue_count
  

  Depending on driver/configuration the per-DIMM directories may be named
  dimm* or rank*, but the file shape (dimm_label, dimm_ce_count,
  dimm_ue_count) is the same.

On a DIMM-model host the collector today gets controller-level node_edac_*
totals and, at best, a degenerate node_edac_csrow_* series labelled
csrow="unknown" (the no-info fallback) — no real per-module breakdown, which
is exactly the level operators need to isolate a failing DIMM. The per-channel
metric added in #3720 reads the same csrow*/chN path, so it is also empty on
these hosts.

Proof this is needed / that it works

I hit this in production on sb_edac hosts. To work around it I built and
deployed a small sidecar that reads the DIMM layer and writes the counts via the
textfile collector, and it has been running reliably for some time.

On one faulty host, the sidecar's DIMM-layer read attributes ~10.6 billion
correctable errors precisely to four DIMMs across two channels:

node_edac_dimm_correctable_errors_total{controller="0",dimm="3", dimm_label="CPU_SrcID#0_Ha#0_Chan#1_DIMM#0"} 3388086162
node_edac_dimm_correctable_errors_total{controller="0",dimm="4", dimm_label="CPU_SrcID#0_Ha#0_Chan#1_DIMM#1"} 1546438102
node_edac_dimm_correctable_errors_total{controller="0",dimm="9", dimm_label="CPU_SrcID#0_Ha#0_Chan#3_DIMM#0"} 2282115649
node_edac_dimm_correctable_errors_total{controller="0",dimm="10",dimm_label="CPU_SrcID#0_Ha#0_Chan#3_DIMM#1"} 3402668185
# channels 0 and 2 clean (0)

For the same host at the same time, native node_exporter reports only:

node_edac_csrow_correctable_errors_total{controller="0",csrow="unknown"} 4260502

i.e. a single csrow="unknown" aggregate of ~4.26M against a true ~10.6B — a
roughly 2,500× undercount, with no indication of which module is failing. The
data is present and reliable in sysfs; node_exporter just isn't reading the layer
that has it. I'd rather fold this upstream than maintain a sidecar.

Open question — what metric shape would you prefer?

This is the main thing I'd like guidance on before writing a PR, because it's a
naming/compatibility call that's yours to make. The channel layer and the DIMM
layer are really the same measurement — sub-controller error counts — surfaced
by different drivers, and importantly the two layouts are mutually exclusive
per host (a machine is either legacy csrow*/chN or modern dimm*/rank*,
never both).

My instinct is to lean toward a common metric shape across both layouts,
e.g. node_edac_module_correctable_errors_total /
node_edac_module_uncorrectable_errors_total with labels
controller + dimm_label, populated from whichever layout the host exposes.
The reasoning, for what it's worth:

• Because the layouts are mutually exclusive per host, a shared metric name can
  never double-count — it's genuinely one measurement from two sources.

• It lets operators write a single hardware-agnostic query/alert. With separate
  metric names, every dashboard and alert rule has to or the two series
  together and risks a silent blind spot if one is forgotten:

  # common shape
  sum by (instance, controller, dimm_label) (rate(node_edac_module_correctable_errors_total[1h]))
  
  # separate metrics — has to be OR'd everywhere, forever
  sum by (instance, controller, dimm_label) (rate(node_edac_channel_correctable_errors_total[1h]))
    or
  sum by (instance, controller, dimm_label) (rate(node_edac_dimm_correctable_errors_total[1h]))
  

• dimm_label is already the natural shared key — it exists in both layouts
  (legacy chN_dimm_label, modern dimm*/dimm_label) and edac: Add per-channel error metrics with DIMM labels #3720 just
  standardised it.

Worth being upfront about my own experience here: the sidecar above emits the
DIMM counts under a node_edac_dimm_* name with controller/dimm/
dimm_label labels — i.e. the companion-metric shape (Option B). That was a
pragmatic bet given where the collector was at the time, and it works, so it's
proof the DIMM-layer read is sound. But having lived with it, I think the
common shape (Option A) is the more cohesive end state — a single
node_edac_module_* that consumers can query regardless of which layout a host
happens to expose, rather than everyone learning that "channel" and "dimm" are
the same thing under two names.

The companion node_edac_dimm_* option (Option B) is of course still on the
table — it matches the one-name-per-granularity precedent of node_edac_csrow_*
and node_edac_channel_*, and it's literally what I'm running — so if you'd
rather keep naming consistent with the existing metrics I'm very happy to go
that way. I don't want to presume; you've got the full picture on naming
conventions and back-compat. Whichever you prefer, I'll build to it.

One small related point: #3720 sanitises # out of dimm_label, whereas the
raw sysfs value (and my sidecar) keeps it, e.g.
CPU_SrcID#0_Ha#0_Chan#3_DIMM#1. Whichever metric shape we land on, it'd be
good to agree a single dimm_label normalisation so the label is identical
across layouts.

Constraints / how it fits the collector rules

• Reads only /sys files — no external commands, no root, no vendor-specific
  transformation of values.
• Honours the configurable sysfs root via sysFilePath() (no hard-coded paths).
• Detection is layout-based: if csrow* exists, use today's path; otherwise
  glob dimm[0-9]* / rank[0-9]* under the controller.
• Test coverage would add a sys.ttar fixture for a DIMM-layout controller
  plus golden end-to-end output.

I have a working implementation and am glad to open a PR once we've settled on
the metric shape you'd prefer. This builds directly on the per-channel /
dimm_label groundwork in #3720, so cc @arunsrini3082 and @SuperQ — I'd really
value your steer on the naming question above before I start, since you two have
the clearest view of where the EDAC collector is heading. Thanks for the work on
#3720.

[arunsrini3082]

@at-blacknight — can confirm this affects our fleet. Running on i10nm_edac hosts, all controllers expose dimm* dirs, no csrow*. We have a live example right now:

$ cat /sys/devices/system/edac/mc/mc7/dimm0/dimm_ce_count
4
$ cat /sys/devices/system/edac/mc/mc7/dimm0/dimm_label
CPU_SrcID#1_MC#3_Chan#0_DIMM#0

Node_exporter reports this as node_edac_csrow_correctable_errors_total{controller="7", csrow="unknown"} 4 — correct count, zero attribution.

On naming: my vote is Option B (node_edac_dimm_*). node_edac_channel_* was just merged in #3720 and renaming it immediately to node_edac_module_* would break anyone who just adopted it. A recording rule can abstract over both for dashboards. Option A is cleaner architecturally but the timing is wrong — it should have been the shape for #3720 itself.

Important: the "mutually exclusive per host" assumption doesn't hold for AMD. We also run amd64_edac_mod hosts, and there the controller exposes both layouts simultaneously — csrow*/chN AND rank* dirs coexist under the same mc*:

$ ls /sys/devices/system/edac/mc/mc0/
ce_count  csrow0  csrow1  rank0  rank1  rank2  ... rank15  ...

On AMD, the rank* dirs have synthetic EDAC-generated labels (mc#0csrow#0channel#0) and carry the same error counts as csrow*. Reading both would double-count. The csrow*/chN data is the authoritative source on AMD.

This means the detection logic can't rely on "if dimm/rank exists, use it" — it needs to be csrow takes priority; dimm/rank is the fallback only when csrow is absent. Concretely:

• Intel legacy (sb_edac, older): csrow* present → use today's path
• Intel DIMM-model (i10nm_edac, skx_edac): no csrow* → fall back to dimm*/rank*
• AMD (amd64_edac_mod): both present → prefer csrow*, ignore rank*

One thing to resolve on dimm_label normalization: #3720 strips # from labels (CPU_SrcID#1_MC#3_Chan#0_DIMM#0 → CPU_SrcID1_MC3_Chan0_DIMM0). Since # is valid in Prometheus label values, there's no technical requirement for this. Whatever the new code does, it should match edacDimmLabel() exactly so the label is identical across both layouts on mixed fleets. Worth deciding explicitly rather than having it diverge.