Skip to content

Add sparse edge attribute bias to graph transformer#675

Draft
yliu2-sc wants to merge 1 commit into
mainfrom
yliu2/gt-edge-attr
Draft

Add sparse edge attribute bias to graph transformer#675
yliu2-sc wants to merge 1 commit into
mainfrom
yliu2/gt-edge-attr

Conversation

@yliu2-sc

Copy link
Copy Markdown
Collaborator

Summary

Add an opt-in sparse edge-attribute attention-bias path for Graph Transformer.

This introduces edge_attr_attention_bias_mode="sparse_linear" on GraphTransformerEncoder, while preserving the default "none" behavior. When enabled, GT uses sampled edge features as sparse per-head additive attention-logit bias.

Changes

  • Add edge_attr_attention_bias_mode to GraphTransformerEncoder.
  • Support "sparse_linear" mode using positive dims from edge_type_to_feat_dim_map.
  • Add zero-initialized Linear(edge_feat_dim -> num_heads, bias=True) per positive-dim edge type.
    • Weight learns edge-feature effects.
    • Bias learns pure edge-presence effects.
  • Extend heterodata_to_graph_transformer_input(...) to return sparse edge-attr payloads:
    • pairwise_edge_attr_indices: (batch_idx, query_pos, key_pos)
    • pairwise_edge_attr_values: raw matched edge_attr rows
  • Keep edge attrs sparse until final index_put_(..., accumulate=True) into attention bias.
  • Use stored edge direction: src -> dst maps to key=src, query=dst.
  • Skip zero-dim edge types before matching.
  • Accumulate duplicate edge contributions.

Tests

  • Added transform tests for sparse edge-attr extraction, stored-direction orientation, padding exclusion, zero-dim skipping, and malformed/missing edge attrs.
  • Added encoder tests for zero-init baseline parity, learned presence/feature bias, duplicate accumulation, zero-dim skipping, and PPR compatibility.

Validation

  • ruff check passed on touched files.
  • git diff --check passed.
  • Ad hoc sparse edge-attr validation passed.

Where is the documentation for this feature?: N/A

Did you add automated tests or write a test plan?

Updated Changelog.md? NO

Ready for code review?: NO

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants