Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion apps/cli-go/docs/supabase/db/pull.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Requires your local project to be linked to a remote database by running `supaba

Optionally, a new row can be inserted into the migration history table to reflect the current state of the remote database.

If no entries exist in the migration history table, the default diff engine uses `pg_dump` to capture all contents of the remote schemas you have created. Otherwise, this command will only diff schema changes against the remote database, similar to running `db diff --linked`.
If no entries exist in the migration history table, the initial pull captures the remote schema. When pg-delta is the active diff engine (the default for projects created by a recent `supabase init`), pg-delta diffs against an empty shadow and produces the full migration without a pg_dump seed. When pg-delta is disabled (`[experimental.pgdelta] enabled = false`) or you pass `--diff-engine migra`, the initial pull uses `pg_dump` to capture remote schema contents before appending a migra diff. Otherwise, this command will only diff schema changes against the remote database, similar to running `db diff --linked`.

Pass `--diff-engine pg-delta` to keep the migration-file `db pull` workflow while using pg-delta for the shadow diff step. On initial pull, pg-delta replaces `pg_dump` and produces the full migration from the shadow diff alone. Pass `--declarative` to switch to the declarative pg-delta export workflow instead.

Expand Down
27 changes: 27 additions & 0 deletions apps/cli-go/internal/db/pull/pgdelta_pull_debug.go
Original file line number Diff line number Diff line change
Expand Up @@ -111,3 +111,30 @@ func formatByteSize(size int) string {
return fmt.Sprintf("%d B", size)
}
}

func printInitialPgDeltaEmptyPullHint() {
fmt.Fprintln(os.Stderr, "Hint: initial pg-delta pulls introspect the remote catalog directly.")
fmt.Fprintln(os.Stderr, "Prefer a direct connection (db.<project-ref>.supabase.co) over the pooler, or run with PGDELTA_DEBUG=1 for a debug bundle.")
fmt.Fprintln(os.Stderr, "You can also retry once with "+utils.Aqua("--diff-engine migra")+" to use pg_dump for the initial migration.")
}

func handleInitialPgDeltaEmptyPull(
ctx context.Context,
config pgconn.Config,
options ...func(*pgx.ConnConfig),
) error {
// Always print the hint on empty initial pg-delta pull (when PGDELTA_DEBUG is off).
printInitialPgDeltaEmptyPullHint()
targetCatalog, err := exportTargetCatalog(ctx, utils.ToPostgresURL(config), "postgres", options...)
if err != nil {
// Catalog probe failed; keep the generic errInSync below.
return nil
}
summary := diff.SummarizeCatalogJSON(targetCatalog)
if summary.TotalObjects == 0 {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Don't treat built-in catalog entries as pullable objects

For an initial pg-delta pull against a project with no user-defined schema, exportTargetCatalog still returns the default/managed catalog entries (for example schema nodes), and SummarizeCatalogJSON counts any node with a schema field. That means TotalObjects can be nonzero even though there is truly nothing to pull, so this path reports the new “could not introspect” error instead of the intended generic “No schema changes found”; base the probe on pullable user objects or the same Supabase filter used by the diff.

Useful? React with 👍 / 👎.

// Remote catalog is empty too → truly nothing to pull.
return nil
}
utils.CmdSuggestion = "pg-delta returned zero SQL statements even though the remote catalog contains objects. Prefer a direct database connection over the pooler, or run with PGDELTA_DEBUG=1."
return errors.New("No schema changes found: pg-delta could not introspect the remote schema")
}
12 changes: 12 additions & 0 deletions apps/cli-go/internal/db/pull/pull.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ import (
"github.com/supabase/cli/internal/migration/new"
"github.com/supabase/cli/internal/migration/repair"
"github.com/supabase/cli/internal/utils"
"github.com/supabase/cli/internal/utils/flags"
"github.com/supabase/cli/pkg/migration"
)

Expand Down Expand Up @@ -158,6 +159,11 @@ func dumpRemoteSchema(ctx context.Context, path string, config pgconn.Config, fs
}

func diffRemoteSchema(ctx context.Context, schema []string, path string, config pgconn.Config, usePgDeltaDiff bool, differ diff.DiffFunc, fsys afero.Fs, options ...func(*pgx.ConnConfig)) error {
initialPull := len(schema) == 0

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Don't infer initial pull from an empty schema filter

A normal non-initial db pull --diff-engine pg-delta with no --schema also calls this function with schema == [], so this now treats that common path as an initial pull. If the diff is empty and the remote catalog has objects, handleInitialPgDeltaEmptyPull reports the new pg-delta introspection failure instead of the correct “No schema changes found”; pass the sync/initial state explicitly rather than deriving it from the schema filter.

Useful? React with 👍 / 👎.

if usePgDeltaDiff && initialPull && len(flags.ProjectRef) > 0 {
// Pooler→direct rewrite for pg-delta TARGET only (see ResolveDirectDbConfigForPgDelta).
config = utils.ResolveDirectDbConfigForPgDelta(config, flags.ProjectRef, utils.CurrentProfile.ProjectHost)
}
// Diff remote db (source) & shadow db (target) and write it as a new migration.
result, err := diff.DiffDatabase(ctx, schema, config, os.Stderr, fsys, differ, usePgDeltaDiff, options...)
if err != nil {
Expand All @@ -181,6 +187,12 @@ func diffRemoteSchema(ctx context.Context, schema []string, path string, config
return errors.Errorf("%w (debug bundle: %s)", errInSync, debugDir)
}
}
if usePgDeltaDiff && initialPull {
// Hint + optional introspection-specific error (see handleInitialPgDeltaEmptyPull).
if err := handleInitialPgDeltaEmptyPull(ctx, config, options...); err != nil {
return err
}
}
return errors.New(errInSync)
}
if err := utils.MkdirIfNotExistFS(fsys, filepath.Dir(path)); err != nil {
Expand Down
34 changes: 34 additions & 0 deletions apps/cli-go/internal/utils/connect.go
Original file line number Diff line number Diff line change
Expand Up @@ -277,6 +277,40 @@ func ProjectRefFromDirectDbHost(host string) (string, bool) {
return matches[2], true
}

// IsPoolerDbHost reports whether host is a Supabase connection pooler endpoint.
func IsPoolerDbHost(host string) bool {
host = strings.ToLower(host)
return host == "pooler.supabase.com" || strings.HasSuffix(host, ".pooler.supabase.com")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Honor profile-specific pooler domains

This only recognizes production *.pooler.supabase.com hosts, but the repo supports other built-in profiles whose validated pooler domains are not supabase.com (for example supabase-staging uses supabase.green, and custom profiles can set their own pooler_host). In those profiles a linked connection can legitimately resolve to a pooler host under the profile domain, but this predicate returns false, so the initial pg-delta pull skips the direct-host rewrite and retains the pooler behavior this change is trying to avoid.

Useful? React with 👍 / 👎.

}

// ResolveDirectDbConfigForPgDelta rewrites a linked pooler connection to the direct
// Supabase database host so pg-delta can introspect the full remote catalog on an
// initial migration-style db pull.
//
// Inverse of PoolerFallbackConfig (container tried direct, retry pooler): the CLI may
// have picked pooler in NewDbConfigWithPassword when the host OS cannot dial direct,
// but pg-delta runs inside Docker and is given this config as its remote TARGET only.
func ResolveDirectDbConfigForPgDelta(config pgconn.Config, projectRef, projectHost string) pgconn.Config {
if _, ok := ProjectRefFromDirectDbHost(config.Host); ok {
return config
}
if !IsPoolerDbHost(config.Host) {
return config
}
direct := config
direct.Host = GetSupabaseDbHost(projectRef)
direct.Port = 5432
direct.User = "postgres"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve the temp role when rewriting pooler URLs

When NewDbConfigWithPassword chooses the pooler because the direct host is unreachable and no SUPABASE_DB_PASSWORD was provided, initPoolerLogin stores a temporary login role/password in this config. Overwriting User with postgres while keeping that temp-role password makes the rewritten direct pg-delta target fail authentication, so the initial pull path this change is meant to fix still breaks for no-password linked users on IPv4-only host networks; preserve the resolved login role (removing only pooler routing/suffix as needed) unless the password is known to be the postgres DB password.

Useful? React with 👍 / 👎.

// Drop Supavisor tenant routing; not used on direct connections.
if len(direct.RuntimeParams) > 0 {
delete(direct.RuntimeParams, "options")
if len(direct.RuntimeParams) == 0 {
direct.RuntimeParams = nil
}
}
return direct
}

// WarnIPv6PoolerFallback prints a user-visible warning explaining that the direct
// database connection could not be used because the current environment does not
// support IPv6, and that the CLI is retrying through the IPv4 connection pooler.
Expand Down
39 changes: 39 additions & 0 deletions apps/cli-go/internal/utils/connect_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -365,6 +365,45 @@ func TestProjectRefFromDirectDbHost(t *testing.T) {
})
}

func TestIsPoolerDbHost(t *testing.T) {
assert.True(t, IsPoolerDbHost("aws-0-us-east-1.pooler.supabase.com"))
assert.True(t, IsPoolerDbHost("pooler.supabase.com"))
assert.False(t, IsPoolerDbHost("db.test.supabase.co"))
}

func TestResolveDirectDbConfigForPgDelta(t *testing.T) {
ref := apitest.RandomProjectRef()
t.Run("rewrites pooler config to direct host", func(t *testing.T) {
config := pgconn.Config{
Host: "aws-0-us-east-1.pooler.supabase.com",
Port: 6543,
User: "postgres." + ref,
Password: "secret",
Database: "postgres",
RuntimeParams: map[string]string{
"options": "reference=" + ref,
},
}
got := ResolveDirectDbConfigForPgDelta(config, ref, "supabase.co")
assert.Equal(t, GetSupabaseDbHost(ref), got.Host)
assert.Equal(t, uint16(5432), got.Port)
assert.Equal(t, "postgres", got.User)
assert.Equal(t, "secret", got.Password)
assert.Nil(t, got.RuntimeParams)
})
t.Run("leaves direct config unchanged", func(t *testing.T) {
config := pgconn.Config{
Host: GetSupabaseDbHost(ref),
Port: 5432,
User: "postgres",
Password: "secret",
Database: "postgres",
}
got := ResolveDirectDbConfigForPgDelta(config, ref, "supabase.co")
assert.Equal(t, config, got)
})
}

func TestWarnIPv6PoolerFallback(t *testing.T) {
oldStderr := os.Stderr
r, w, err := os.Pipe()
Expand Down
16 changes: 16 additions & 0 deletions apps/cli/src/legacy/commands/db/pull/pull.debug.ts
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,22 @@ export function legacySummarizeCatalogJson(catalogJson: string): LegacyCatalogSu
return { totalObjects: total, bySchema };
}

/**
* Stderr hint for empty initial pg-delta pulls. Port of Go's `printInitialPgDeltaEmptyPullHint`.
* Printed whenever pg-delta returns zero SQL on the first pull (unless PGDELTA_DEBUG handled it).
*/
export function legacyFormatInitialPgDeltaEmptyPullHint(): string {
return [
"Hint: initial pg-delta pulls introspect the remote catalog directly.",
"Prefer a direct connection (db.<project-ref>.supabase.co) over the pooler, or run with PGDELTA_DEBUG=1 for a debug bundle.",
`You can also retry once with ${legacyBold("--diff-engine migra")} to use pg_dump for the initial migration.`,
].join("\n");
}

/** Message when pg-delta sees remote catalog objects but emits zero SQL. */
export const LEGACY_PG_DELTA_INTROSPECTION_ERROR =
"No schema changes found: pg-delta could not introspect the remote schema";

/** Port of Go's `formatCatalogSummary`. */
export function legacyFormatCatalogSummary(label: string, summary: LegacyCatalogSummary): string {
if (summary.totalObjects === 0) return `${label} catalog: no objects detected`;
Expand Down
61 changes: 50 additions & 11 deletions apps/cli/src/legacy/commands/db/pull/pull.handler.ts
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,10 @@ import {
} from "../../../shared/legacy-db-config.toml-read.ts";
import type { LegacyDbConnType } from "../../../shared/legacy-db-target-flags.ts";
import { legacyMakeDir } from "../../../shared/legacy-make-dir.ts";
import { legacyToPostgresURL } from "../../../shared/legacy-postgres-url.ts";
import {
legacyHostFromPostgresURL,
legacyToPostgresURL,
} from "../../../shared/legacy-postgres-url.ts";
import { legacySchemaToCsvField } from "../../../shared/legacy-schema-flags.ts";
import { LegacyLinkedProjectCache } from "../../../telemetry/legacy-linked-project-cache.service.ts";
import { LegacyTelemetryState } from "../../../telemetry/legacy-telemetry-state.service.ts";
Expand All @@ -49,6 +52,7 @@ import { legacyStreamPgDump } from "../shared/legacy-pg-dump.run.ts";
import {
legacyEmitPoolerFallbackWarning,
legacyIsDirectLinkedHost,
legacyResolveDirectDbConfigForPgDelta,
legacyRunWithPoolerFallback,
} from "../shared/legacy-pooler-fallback.ts";
import { legacyDumpSchemaScript } from "../shared/legacy-pg-dump.scripts.ts";
Expand All @@ -64,7 +68,12 @@ import {
legacyExportCatalogPgDelta,
legacyIsPgDeltaDebugEnabled,
} from "../shared/legacy-pgdelta.ts";
import { legacySaveEmptyPgDeltaPullDebug } from "./pull.debug.ts";
import {
LEGACY_PG_DELTA_INTROSPECTION_ERROR,
legacyFormatInitialPgDeltaEmptyPullHint,
legacySaveEmptyPgDeltaPullDebug,
legacySummarizeCatalogJson,
} from "./pull.debug.ts";
import { LegacyDeclarativeSeam } from "../shared/legacy-pgdelta.seam.service.ts";
import type { LegacyDbPullFlags } from "./pull.command.ts";
import {
Expand Down Expand Up @@ -267,14 +276,16 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy
const withPoolerFallback = <A, E extends { readonly message: string }, R>(
directTarget: string,
attempt: (targetRef: string) => Effect.Effect<A, E, R>,
) =>
attempt(directTarget).pipe(
) => {
// Pooler fallback gates on the container TARGET host, not `resolved.conn.host`.
const targetHost = legacyHostFromPostgresURL(directTarget);
return attempt(directTarget).pipe(
Effect.catch((error) =>
Effect.gen(function* () {
if (
legacyIsDirectLinkedHost({
connType,
host: resolved.conn.host,
host: targetHost,
isLocal: resolved.isLocal,
projectHost: cliConfig.projectHost,
}) &&
Expand All @@ -292,14 +303,15 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy
})
.pipe(Effect.orElseSucceed(() => Option.none()));
if (Option.isSome(pooler)) {
yield* legacyEmitPoolerFallbackWarning(resolved.conn.host);
yield* legacyEmitPoolerFallbackWarning(targetHost);
return yield* attempt(connToUrl(pooler.value));
}
}
return yield* Effect.fail(error);
}),
),
);
};

const usePgDeltaDiff = legacyResolvePullDiffEngine({
engineFlagChanged: Option.isSome(flags.diffEngine),
Expand Down Expand Up @@ -447,14 +459,24 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy
}),
);
}
const initialPull = sync.kind === "missing";
// Initial pg-delta pull only: if `--linked` resolved to a pooler host, rewrite
// the pg-delta TARGET to `db.<ref>.<host>` (see legacyResolveDirectDbConfigForPgDelta).
// The CLI postgres session (`resolved.conn`) is unchanged.
const pgDeltaTargetConn =
initialPull && usePgDeltaDiff && connType === "linked" && linkedRef !== undefined
? legacyResolveDirectDbConfigForPgDelta(resolved.conn, linkedRef, cliConfig.projectHost)
: resolved.conn;
const pgDeltaTargetUrl = connToUrl(pgDeltaTargetConn);
const diffTargetUrl = initialPull && usePgDeltaDiff ? pgDeltaTargetUrl : targetUrl;
// Initial pull, migra engine (Go's `run` → `assertRemoteInSync` returns
// `errMissing`): seed the migration file with a pg_dump of the remote schema
// (`dumpRemoteSchema`, `pull.go:144-158`), then run the migra diff below as a
// second pass appended to the same file (`diffRemoteSchema(ctx, nil, …)`),
// which captures default privileges / managed schemas pg_dump can't emit.
// pg-delta initial pulls skip the dump (`pull.go:126` `if !usePgDeltaDiff`):
// they diff against an empty shadow, which already yields the full schema.
const seededFromDump = sync.kind === "missing" && !usePgDeltaDiff;
const seededFromDump = initialPull && !usePgDeltaDiff;
// Tracks whether the pg_dump seed wrote any bytes, for Go's
// `ensureMigrationWritten` (`pull.go:68,263-268`): an empty dump + empty diff
// is "in sync", a non-empty dump is a valid initial migration on its own.
Expand Down Expand Up @@ -559,7 +581,7 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy
// Native diff: shadow (baseline + local migrations) vs remote → migration SQL.
// For the initial pull (no local migrations) the schema filter is ignored,
// matching Go's `diffRemoteSchema(ctx, nil, …)`.
const diffSchema = sync.kind === "missing" ? [] : flags.schema;
const diffSchema = initialPull ? [] : flags.schema;
// Go's `DiffDatabase` emits these to stderr before provisioning + diffing
// (`internal/db/diff/diff.go:189,234-237`); the shadow seam doesn't, so the
// pull handler emits them itself to match the migration-style `db pull` output.
Expand All @@ -581,7 +603,7 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy
// Use the declarative target override when present (Go substitutes it
// for the diff target, `diff.go:196-197`); for remote pulls it's
// undefined, so this is the direct target URL as before.
const target = shadow.targetUrlOverride ?? targetUrl;
const target = shadow.targetUrlOverride ?? diffTargetUrl;
yield* output.raw(
diffSchema.length > 0
? `Diffing schemas: ${diffSchema.join(",")}\n`
Expand Down Expand Up @@ -647,8 +669,8 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy
if (diffOutcome.capture !== undefined) {
const debugDir = yield* legacySaveEmptyPgDeltaPullDebug({
ctx,
conn: resolved.conn,
targetUrl,
conn: pgDeltaTargetConn,
targetUrl: pgDeltaTargetUrl,
sourceCatalog: diffOutcome.capture.sourceCatalog,
pgDeltaStderr: diffOutcome.capture.stderr,
id: legacyFormatDebugId(yield* Clock.currentTimeMillis),
Expand All @@ -673,6 +695,23 @@ export const legacyDbPull = Effect.fn("legacy.db.pull")(function* (flags: Legacy
);
}
}
if (initialPull && usePgDeltaDiff && !legacyIsPgDeltaDebugEnabled()) {
// Shown on every empty initial pg-delta pull (unless PGDELTA_DEBUG took the
// debug-bundle path above). Helps when pooler introspection silently fails.
yield* output.raw(`${legacyFormatInitialPgDeltaEmptyPullHint()}\n`, "stderr");
// Probe the remote catalog: if objects exist but pg-delta returned no SQL,
// fail with a specific introspection error instead of generic "in sync".
const targetCatalog = yield* legacyExportCatalogPgDelta(ctx, {
targetRef: pgDeltaTargetUrl,
role: "postgres",
}).pipe(Effect.catch(() => Effect.succeed("")));
if (legacySummarizeCatalogJson(targetCatalog).totalObjects > 0) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Don't treat built-in catalog entries as pullable objects

For an initial pg-delta pull against a project with no user-defined schema, the catalog export can still contain default/managed schema entries, and legacySummarizeCatalogJson counts any node with a schema field as an object. This makes totalObjects > 0 true even when there is nothing pullable, so the handler emits the introspection-failure error instead of the intended generic “No schema changes found”; filter the catalog summary down to objects the pg-delta diff would actually emit.

Useful? React with 👍 / 👎.

return yield* Effect.fail(
new LegacyDbPullInSyncError({ message: LEGACY_PG_DELTA_INTROSPECTION_ERROR }),
);
}
// Catalog also empty → truly nothing pullable; fall through to generic error.
}
return yield* Effect.fail(
new LegacyDbPullInSyncError({ message: "No schema changes found" }),
);
Expand Down
Loading