nvkms-dpy: don't clamp DP sinks to 6 bpc on undefined EDID depth

[Wintch]

What

The DisplayPort branch of nvDpyGetOutputColorFormatInfo() treats "the sink did not declare
a color depth" as "the sink wants 6 bpc", and drives the link at 18 bpp for every mode. The
DSI branch of the same function already treats that same input as 8 bpc, so the two
branches disagree on identical input.

Present on main at
nvkms-dpy.c:3468;
measured on the 595.71.05 tag, where the same line is 3456.

Why it happens

For a digital sink with EDID >= 1.4,
nvt_edid.c:932
maps the "undefined" encoding of the Video Input Definition byte (offset 0x14, bits 6-4 =
000) to zero:

    default :
        pInfo->input.u.digital.bpc = 0;
        break;

The DP branch then does:

    } else if (pDpyEvo->parsedEdid.info.input.u.digital.bpc < 8) {
        colorFormatsInfo.rgb444.maxBpc = NV_KMS_DPY_ATTRIBUTE_CURRENT_COLOR_BPC_6;
        colorFormatsInfo.yuv444.maxBpc = NV_KMS_DPY_ATTRIBUTE_CURRENT_COLOR_BPC_UNKNOWN;

0 < 8 is true, so the sink is driven at 6 bpc. The DSI branch a few lines above handles the
same "we don't know" case the other way:

    default:
        nvAssert(!"Unsupported bpc for DSI");
        // fall through
    case 8:
        colorFormatsInfo.rgb444.maxBpc = NV_KMS_DPY_ATTRIBUTE_CURRENT_COLOR_BPC_8;

Why nothing corrects the value afterwards

Tree-wide (grep -rn 'digital\.bpc[[:space:]]*=' src/), input.u.digital.bpc is written in
three places:

• src/common/modeset/timing/nvt_edid.c:914-932 — the base-block switch above.
• nvt_edidext_displayid20.c:314
  — the Display Parameters block of a DisplayID 2.x extension.
• nvkms-dpy.c:2258
  — inside CreateParsedEdidFromNVT_TIMING(), reached only from GetFixedModeTimings()
  (nvkms-dpy.c:627) for DSI panels and DP serializers, which synthesize a parsed EDID from
  RM-supplied timings with a caller-provided bpc. It does not run for a sink that has a real
  EDID.

So for a real sink EDID the only override is the DisplayID 2.x one, and that is gated purely
on the extension's version byte, at
nvt_edid.c:1101:

    case NVT_EDID_EXTENSION_DISPLAYID:
        if ((pExt[1] & 0xF0) == 0x20) // displayID2.x as EDID extension
            getDisplayId20EDIDExtInfo(...);
        else                          // displayID13 as EDID extension
            getDisplayIdEDIDExtInfo(...);

The DisplayID 1.3 parser never writes digital.bpc. So for any DisplayPort sink with a
DisplayID 1.x extension — or with no DisplayID extension — that leaves base-block Color Bit
Depth undefined, the 6 bpc clamp is unconditional. On an ordinary monitor that surfaces as
banding, which is easy to misattribute to the panel.

Why fix the consumer and not the parser

bpc = 0 carries real information: "the sink did not declare a depth", which is distinct
from "the sink declared 8". Defaulting to 8 in nvt_edid.c would erase that distinction for
every consumer of the parsed EDID. Fixing it at the point of use preserves parse fidelity and
limits the change to the one branch that misreads it.

Reproduced on

• RTX 3060 Ti (GA104), driver 595.71.05, open kernel modules, Debian 13, kernel 6.12.100
• Sink: HP Reverb G2 — EDID base byte 0x14 = 0x80 (digital, Color Bit Depth undefined),
  DisplayID extension version byte 0x12 (DisplayID 1.2)

Before:

nvidia-modeset: DPCONN> Notify Attach Begin (Head 0, pclk 428580000 raster 2980 x 1598  18 bpp)

After: 24 bpp, on every mode this sink advertises. Reproduced on every attempt, on both X11
and Wayland.

The sink confirms the change independently of the driver: its companion device reports the
bpc in use back to the host over HID, and that field goes from 06 to 08 — matching what
the same GPU negotiates for the same mode under Windows.

Reproducing without this hardware

The clamp reproduces on any EDID 1.4 digital sink you can EDID-override. Clear bits 6-4 of
base byte 0x14 (Color Bit Depth -> "undefined"), fix the base-block checksum at 0x7F,
feed it as an override, and the attach log reports 18 bpp. Restoring the bits, or applying
this patch, restores 24 bpp.

Testing

Built and booted nvidia-kernel-open-dkms 595.71.05 with this change on GA104 / kernel
6.12.100, X11 and Wayland. Other sinks on the system are unaffected: an EDID that declares a
depth does not reach the modified condition with a different result than before.

Context

Longer write-up, including a separate and still-unresolved 90 Hz failure on this same sink
that this patch does not fix:
https://forums.developer.nvidia.com/t/hp-reverb-g2-clamped-to-6-bpc-because-its-edid-leaves-color-depth-undefined-root-cause-found-two-line-patch-but-90-hz-still-fails-to-light/379240

[CLAassistant]

All committers have signed the CLA.