Skip to content

tofino_stub returns port not found when trying to apply settings to qsfp31 #271

@jgallagher

Description

@jgallagher

On main, attempting to apply settings to port qsfp31 fails. Little dummy test with curl that hits the dpd stub binary built for Linux:

# request body
% cat req.json
{"links":{"0":{"addrs":[],"params":{"autoneg":false,"kr":false,"lane":0,"speed":"Speed0G"}}}}

# posting to qsfp0, ..., qsfp30 is fine
% curl -X POST -H 'content-type: application/json' -H 'api-version: 12.0.0' --data @req.json 'http://[::1]:12224/port/qsfp0/settings'
{"links":{"0":{"params":{"lane":0,"speed":"Speed0G","fec":null,"autoneg":false,"kr":false,"tx_eq":null},"addrs":[]}}}
% curl -X POST -H 'content-type: application/json' -H 'api-version: 12.0.0' --data @req.json 'http://[::1]:12224/port/qsfp30/settings'
{"links":{"0":{"params":{"lane":0,"speed":"Speed0G","fec":null,"autoneg":false,"kr":false,"tx_eq":null},"addrs":[]}}}

# posting to qsfp31 fails
% curl -X POST -H 'content-type: application/json' -H 'api-version: 12.0.0' --data @req.json 'http://[::1]:12224/port/qsfp31/settings'
{
  "request_id": "038b79cd-d798-41d8-8a73-90dbef511cfc",
  "error_code": "invalid data: no such port",
  "message": "no such port"
}

qsfp31 definitely works in the product (@sion42x has tested it on a racklette), so this appears to be a stub-only bug. I tried tracing through what's going on, and I think there's an off-by-one disagreement in Connector numbering between PortMap (which maps an API-level PortId to a Connector) and PortData (which maps a (Connector, channel) tuple to an ASIC ID):

  • PortMap maps PortId::Rear(0..32) to connectors QSFP(1..33) (but the order is scrambled) and then maps PortId::Qsfp(0..32) to connectors Qsfp(33..65) (not scrambled: PortId::Qsfp(0) becomes connector QSFP(33), up through PortId::Qsfp(31) mapping to connector QSFP(64)).
  • PortData's map of connectors to ASIC IDs contains keys for connectors QSFP(0..64); this does not have an entry for QSFP(64) itself.

When we try to map PortId::Qsfp(31) to an ASIC ID, PortMap tells us it's connector QSFP(64), and then we get the not found error when trying to look up QSFP(64) in PortData.

I'm very unfamiliar with all of this, but AFAICT, PortMap is shared between the stub and real ASIC, so the problem is presumably in the stub's PortData. If I change this line in asic::tofino_stub::ports::init():

+++ b/asic/src/tofino_stub/ports.rs
@@ -249,12 +249,13 @@ pub fn init() -> AsicResult<PortData> {
     let eth_port = Some(CPU_PORT as u16);
     connectors.insert(Connector::CPU, PhysPort::new(Connector::CPU)?);

-    for id in 0..QSFP_PORT_COUNT {
+    for id in 1..=QSFP_PORT_COUNT {
         let connector = Connector::QSFP(id);
         let mut phys_port = PhysPort::new(connector)?;
         phys_port.media = PortMedia::Optical;

I'm able to apply settings to qsfp31. But I don't know if shifting the connector indexing like this may cause other problems somewhere else in the stub.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions