Skip to content

docs: add Generic HTTP Sink connector page and update connectors table #39

@mlevkov

Description

@mlevkov

Context

Apache Iggy 0.8.0 (release post) ships the new Generic HTTP Sink connector (apache/iggy#2925). The connector's documentation currently lives only in the upstream core/connectors/sinks/http_sink/README.md and is not surfaced on the docs site.

The site's Available Connectors table does not list it, and there is no per-connector page under /docs/connectors/sinks/ analogous to the existing Elasticsearch, Postgres, MongoDB, Quickwit, Iceberg, or Stdout pages.

Proposed Changes

A scoped 3-file PR against main:

  1. Add content/docs/connectors/sinks/http.mdx — new page sourced from the upstream README, reformatted to match the style of sibling sink pages.
  2. Edit content/docs/connectors/sinks/meta.json — insert "http" in the pages array (controls sidebar order).
  3. Edit content/docs/connectors/introduction.mdx — add the connector to the "Available Connectors" Sink row.

Naming & Style Decisions

Surface Value Rationale
File / URL slug http.mdx/docs/connectors/sinks/http Matches sibling convention (no _sink suffix).
Page title (frontmatter) HTTP Sink Matches sibling pages: Elasticsearch Sink, Postgres Sink, etc.
Available Connectors table cell Generic HTTP "HTTP" alone is ambiguous — Elasticsearch and Quickwit sinks also use HTTP transport. "Generic" disambiguates as bring-your-own-endpoint, not target-specific. Matches the upstream commit and 0.8.0 release notes.
Sidebar / table position Just before stdout Existing order is roughly databases → search/lake → utility. Generic HTTP is utility-flavored, alongside stdout.

Resulting meta.json:

"pages": ["sink", "elasticsearch", "postgres", "mongodb", "quickwit", "iceberg", "http", "stdout"]

Resulting table row:

| Sink | PostgreSQL, MongoDB, Elasticsearch, Quickwit, Apache Iceberg, Generic HTTP, Stdout |

Content Scope (Proposed)

The upstream README is ~810 lines, well above any existing connector page on the site (range: 65–190 lines). Proposed scope for http.mdx, targeting ~200 lines:

Include (essential surface):

  • Intro paragraph
  • Full Configuration options table
  • Batch Modes (individual, ndjson, json_array, raw) — unique to this connector
  • Metadata Envelope (output shape)
  • Authentication patterns (Bearer / API key / Basic)
  • Retry strategy summary + Delivery Semantics caveat (at-most-once)
  • 2–3 representative use-case configs (webhook, bulk ingestion, IoT/raw)
  • Known Limitations

Summarize with a link back to the upstream README:

  • Deployment Patterns (multi-instance, fan-out, Docker recipes)
  • Performance Considerations (connection pooling, batch-mode throughput math)

Omit:

  • "Try It" hands-on tutorial — the runtime page already covers how to run a connector
  • Testing instructions — repo-developer concern

Source-of-Truth Strategy

Upstream README.md remains canonical. The site page is a curated reference subset that links back to the upstream README for the sections we summarize. We don't mirror line-for-line because the README also serves as a contributor doc with internal source links (e.g. runtime/src/sink.rs) that don't belong on the public site. Future connector changes should update both the upstream README and this page in lockstep.

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions