diff --git a/website/www/site/content/en/documentation/io/managed-io.md b/website/www/site/content/en/documentation/io/managed-io.md index ec3bb90183d4..925f55aa52a5 100644 --- a/website/www/site/content/en/documentation/io/managed-io.md +++ b/website/www/site/content/en/documentation/io/managed-io.md @@ -59,7 +59,7 @@ and Beam SQL is invoked via the Managed API under the hood. Write Configuration - ICEBERG_CDC + ICEBERG table (str)
catalog_name (str)
@@ -67,17 +67,23 @@ and Beam SQL is invoked via the Managed API under the hood. config_properties (map[str, str])
drop (list[str])
filter (str)
- from_snapshot (int64)
- from_timestamp (int64)
keep (list[str])
- poll_interval_seconds (int32)
- starting_strategy (str)
- streaming (boolean)
- to_snapshot (int64)
- to_timestamp (int64)
- Unavailable + table (str)
+ autosharding (boolean)
+ catalog_name (str)
+ catalog_properties (map[str, str])
+ config_properties (map[str, str])
+ direct_write_byte_limit (int32)
+ distribution_mode (str)
+ drop (list[str])
+ keep (list[str])
+ only (str)
+ partition_fields (list[str])
+ sort_fields (list[str])
+ table_properties (map[str, str])
+ triggering_frequency_seconds (int32)
@@ -109,7 +115,7 @@ and Beam SQL is invoked via the Managed API under the hood. - ICEBERG + ICEBERG_CDC table (str)
catalog_name (str)
@@ -117,31 +123,24 @@ and Beam SQL is invoked via the Managed API under the hood. config_properties (map[str, str])
drop (list[str])
filter (str)
+ from_snapshot (int64)
+ from_timestamp (int64)
keep (list[str])
+ poll_interval_seconds (int32)
+ starting_strategy (str)
+ streaming (boolean)
+ to_snapshot (int64)
+ to_timestamp (int64)
- table (str)
- autosharding (boolean)
- catalog_name (str)
- catalog_properties (map[str, str])
- config_properties (map[str, str])
- direct_write_byte_limit (int32)
- distribution_mode (str)
- drop (list[str])
- keep (list[str])
- only (str)
- partition_fields (list[str])
- sort_fields (list[str])
- table_properties (map[str, str])
- triggering_frequency_seconds (int32)
+ Unavailable - SQLSERVER + POSTGRES jdbc_url (str)
connection_properties (str)
- disable_auto_commit (boolean)
fetch_size (int32)
location (str)
num_partitions (int32)
@@ -163,10 +162,11 @@ and Beam SQL is invoked via the Managed API under the hood. - POSTGRES + SQLSERVER jdbc_url (str)
connection_properties (str)
+ disable_auto_commit (boolean)
fetch_size (int32)
location (str)
num_partitions (int32)
@@ -238,7 +238,7 @@ and Beam SQL is invoked via the Managed API under the hood. ## Configuration Details -### `ICEBERG_CDC` Read +### `ICEBERG` Read
@@ -315,90 +315,194 @@ and Beam SQL is invoked via the Managed API under the hood. + +
- from_snapshot + keep - int64 + list[str] - Starts reading from this snapshot ID (inclusive). + A subset of column names to read exclusively. If null or empty, all columns will be read. +
+
+ +### `ICEBERG` Write + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ConfigurationTypeDescription
+ table + + str + + A fully-qualified table identifier. You may also provide a template to write to multiple dynamic destinations, for example: `dataset.my_{col1}_{col2.nested}_table`.
- from_timestamp + autosharding - int64 + boolean - Starts reading from the first snapshot (inclusive) that was created after this timestamp (in milliseconds). + Enables dynamic sharding to automatically adjust the number of parallel writers based on data volume. It handles data skew by further sub-dividing partitions into multiple shards to prevent bottlenecks during high-throughput writes. Only available with 'hash' distribution mode.
- keep + catalog_name - list[str] + str - A subset of column names to read exclusively. If null or empty, all columns will be read. + Name of the catalog containing the table.
- poll_interval_seconds + catalog_properties + + map[str, str] + + Properties used to set up the Iceberg catalog. +
+ config_properties + + map[str, str] + + Properties passed to the Hadoop Configuration. +
+ direct_write_byte_limit int32 - The interval at which to poll for new snapshots. Defaults to 60 seconds. + For a streaming pipeline, sets the limit for lifting bundles into the direct write path.
- starting_strategy + distribution_mode str - The source's starting strategy. Valid options are: "earliest" or "latest". Can be overriden by setting a starting snapshot or timestamp. Defaults to earliest for batch, and latest for streaming. + Defines distribution of write data. Supported distributions: +- none: don't shuffle rows (default) +- hash: shuffle rows by partition key before writing data
- streaming + drop - boolean + list[str] - Enables streaming reads, where source continuously polls for snapshots forever. + A list of field names to drop from the input record before writing. Is mutually exclusive with 'keep' and 'only'.
- to_snapshot + keep - int64 + list[str] - Reads up to this snapshot ID (inclusive). + A list of field names to keep in the input record. All other fields are dropped before writing. Is mutually exclusive with 'drop' and 'only'.
- to_timestamp + only - int64 + str - Reads up to the latest snapshot (inclusive) created before this timestamp (in milliseconds). + The name of a single record field that should be written. Is mutually exclusive with 'keep' and 'drop'. +
+ partition_fields + + list[str] + + Fields used to create a partition spec that is applied when tables are created. For a field 'foo', the available partition transforms are: + +- `foo` +- `truncate(foo, N)` +- `bucket(foo, N)` +- `hour(foo)` +- `day(foo)` +- `month(foo)` +- `year(foo)` +- `void(foo)` + +For more information on partition transforms, please visit https://iceberg.apache.org/spec/#partition-transforms. +
+ sort_fields + + list[str] + + Fields used to set the table's sort order, applied when the table is created. Each entry has the form ` [asc|desc] [nulls first|nulls last]`, where `` is a field name or one of the partition transforms (e.g. `bucket(col, 4)`, `day(ts)`). Direction defaults to ascending; null order defaults to nulls-first for ascending and nulls-last for descending. Note: this sets the table's declared sort order as metadata; it does not cause Beam to physically sort records before writing. +For more information on sort orders, please visit https://iceberg.apache.org/spec/#sort-orders. +
+ table_properties + + map[str, str] + + Iceberg table properties to be set on the table when it is created. +For more information on table properties, please visit https://iceberg.apache.org/docs/latest/configuration/#table-properties. +
+ triggering_frequency_seconds + + int32 + + For a streaming pipeline, sets the frequency at which snapshots are produced.
@@ -659,7 +763,7 @@ and Beam SQL is invoked via the Managed API under the hood.
-### `ICEBERG` Read +### `ICEBERG_CDC` Read
@@ -736,294 +840,279 @@ and Beam SQL is invoked via the Managed API under the hood. -
- keep + from_snapshot - list[str] + int64 - A subset of column names to read exclusively. If null or empty, all columns will be read. + Starts reading from this snapshot ID (inclusive).
-
- -### `ICEBERG` Write - -
- - - - - - +
ConfigurationTypeDescription
- table + from_timestamp - str + int64 - A fully-qualified table identifier. You may also provide a template to write to multiple dynamic destinations, for example: `dataset.my_{col1}_{col2.nested}_table`. + Starts reading from the first snapshot (inclusive) that was created after this timestamp (in milliseconds).
- autosharding + keep - boolean + list[str] - Enables dynamic sharding to automatically adjust the number of parallel writers based on data volume. It handles data skew by further sub-dividing partitions into multiple shards to prevent bottlenecks during high-throughput writes. Only available with 'hash' distribution mode. + A subset of column names to read exclusively. If null or empty, all columns will be read.
- catalog_name + poll_interval_seconds - str + int32 - Name of the catalog containing the table. + The interval at which to poll for new snapshots. Defaults to 60 seconds.
- catalog_properties + starting_strategy - map[str, str] + str - Properties used to set up the Iceberg catalog. + The source's starting strategy. Valid options are: "earliest" or "latest". Can be overriden by setting a starting snapshot or timestamp. Defaults to earliest for batch, and latest for streaming.
- config_properties + streaming - map[str, str] + boolean - Properties passed to the Hadoop Configuration. + Enables streaming reads, where source continuously polls for snapshots forever.
- direct_write_byte_limit + to_snapshot - int32 + int64 - For a streaming pipeline, sets the limit for lifting bundles into the direct write path. + Reads up to this snapshot ID (inclusive).
- distribution_mode + to_timestamp - str + int64 - Defines distribution of write data. Supported distributions: -- none: don't shuffle rows (default) -- hash: shuffle rows by partition key before writing data + Reads up to the latest snapshot (inclusive) created before this timestamp (in milliseconds).
+
+ +### `POSTGRES` Read + +
+ + + + + + -
ConfigurationTypeDescription
- drop + jdbc_url - list[str] + str - A list of field names to drop from the input record before writing. Is mutually exclusive with 'keep' and 'only'. + Connection URL for the JDBC source.
- keep + connection_properties - list[str] + str - A list of field names to keep in the input record. All other fields are dropped before writing. Is mutually exclusive with 'drop' and 'only'. + Used to set connection properties passed to the JDBC driver not already defined as standalone parameter (e.g. username and password can be set using parameters above accordingly). Format of the string must be "key1=value1;key2=value2;".
- only + fetch_size - str + int32 - The name of a single record field that should be written. Is mutually exclusive with 'keep' and 'drop'. + This method is used to override the size of the data that is going to be fetched and loaded in memory per every database call. It should ONLY be used if the default value throws memory errors.
- partition_fields + location - list[str] + str - Fields used to create a partition spec that is applied when tables are created. For a field 'foo', the available partition transforms are: - -- `foo` -- `truncate(foo, N)` -- `bucket(foo, N)` -- `hour(foo)` -- `day(foo)` -- `month(foo)` -- `year(foo)` -- `void(foo)` - -For more information on partition transforms, please visit https://iceberg.apache.org/spec/#partition-transforms. + Name of the table to read from.
- sort_fields + num_partitions - list[str] + int32 - Fields used to set the table's sort order, applied when the table is created. Each entry has the form ` [asc|desc] [nulls first|nulls last]`, where `` is a field name or one of the partition transforms (e.g. `bucket(col, 4)`, `day(ts)`). Direction defaults to ascending; null order defaults to nulls-first for ascending and nulls-last for descending. Note: this sets the table's declared sort order as metadata; it does not cause Beam to physically sort records before writing. -For more information on sort orders, please visit https://iceberg.apache.org/spec/#sort-orders. + The number of partitions
- table_properties + output_parallelization - map[str, str] + boolean - Iceberg table properties to be set on the table when it is created. -For more information on table properties, please visit https://iceberg.apache.org/docs/latest/configuration/#table-properties. + Whether to reshuffle the resulting PCollection so results are distributed to all workers.
- triggering_frequency_seconds + partition_column - int32 + str - For a streaming pipeline, sets the frequency at which snapshots are produced. + Name of a column of numeric type that will be used for partitioning.
-
- -### `SQLSERVER` Read - -
- - - - - - +
ConfigurationTypeDescription
- jdbc_url + password str - Connection URL for the JDBC source. + Password for the JDBC source.
- connection_properties + read_query str - Used to set connection properties passed to the JDBC driver not already defined as standalone parameter (e.g. username and password can be set using parameters above accordingly). Format of the string must be "key1=value1;key2=value2;". + SQL query used to query the JDBC source.
- disable_auto_commit + username - boolean + str - Whether to disable auto commit on read. Defaults to true if not provided. The need for this config varies depending on the database platform. Informix requires this to be set to false while Postgres requires this to be set to true. + Username for the JDBC source.
+
+ +### `POSTGRES` Write + +
+ + + + + + @@ -1039,24 +1128,24 @@ For more information on table properties, please visit https://iceberg.apache.or
ConfigurationTypeDescription
- fetch_size + jdbc_url - int32 + str - This method is used to override the size of the data that is going to be fetched and loaded in memory per every database call. It should ONLY be used if the default value throws memory errors. + Connection URL for the JDBC sink.
- location + autosharding - str + boolean - Name of the table to read from. + If true, enables using a dynamically determined number of shards to write.
- num_partitions + batch_size - int32 + int64 - The number of partitions + n/a
- output_parallelization + connection_properties - boolean + str - Whether to reshuffle the resulting PCollection so results are distributed to all workers. + Used to set connection properties passed to the JDBC driver not already defined as standalone parameter (e.g. username and password can be set using parameters above accordingly). Format of the string must be "key1=value1;key2=value2;".
- partition_column + location str - Name of a column of numeric type that will be used for partitioning. + Name of the table to write to.
- read_query + username str - SQL query used to query the JDBC source. + Username for the JDBC source.
- username + write_statement str - Username for the JDBC source. + SQL query used to insert records into the JDBC sink.
@@ -1162,7 +1251,7 @@ For more information on table properties, please visit https://iceberg.apache.or
-### `POSTGRES` Read +### `SQLSERVER` Read
@@ -1193,6 +1282,17 @@ For more information on table properties, please visit https://iceberg.apache.or Used to set connection properties passed to the JDBC driver not already defined as standalone parameter (e.g. username and password can be set using parameters above accordingly). Format of the string must be "key1=value1;key2=value2;". + + + + +
+ disable_auto_commit + + boolean + + Whether to disable auto commit on read. Defaults to true if not provided. The need for this config varies depending on the database platform. Informix requires this to be set to false while Postgres requires this to be set to true. +
fetch_size @@ -1284,7 +1384,7 @@ For more information on table properties, please visit https://iceberg.apache.or
-### `POSTGRES` Write +### `BIGQUERY` Read
@@ -1295,90 +1395,57 @@ For more information on table properties, please visit https://iceberg.apache.or - - - - - - - - - - - - - - -
- jdbc_url - - str - - Connection URL for the JDBC sink. -
- autosharding - - boolean - - If true, enables using a dynamically determined number of shards to write. -
- batch_size - - int64 - - n/a -
- connection_properties + kms_key str - Used to set connection properties passed to the JDBC driver not already defined as standalone parameter (e.g. username and password can be set using parameters above accordingly). Format of the string must be "key1=value1;key2=value2;". + Use this Cloud KMS key to encrypt your data
- location + query str - Name of the table to write to. + The SQL query to be executed to read from the BigQuery table.
- password + row_restriction str - Password for the JDBC source. + Read only rows that match this filter, which must be compatible with Google standard SQL. This is not supported when reading via query.
- username + fields - str + list[str] - Username for the JDBC source. + Read only the specified fields (columns) from a BigQuery table. Fields may not be returned in the order specified. If no value is specified, then all fields are returned. Example: "col1, col2, col3"
- write_statement + table str - SQL query used to insert records into the JDBC sink. + The fully-qualified name of the BigQuery table to read from. Format: [${PROJECT}:]${DATASET}.${TABLE}
@@ -1462,73 +1529,6 @@ For more information on table properties, please visit https://iceberg.apache.or
-### `BIGQUERY` Read - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
ConfigurationTypeDescription
- kms_key - - str - - Use this Cloud KMS key to encrypt your data -
- query - - str - - The SQL query to be executed to read from the BigQuery table. -
- row_restriction - - str - - Read only rows that match this filter, which must be compatible with Google standard SQL. This is not supported when reading via query. -
- fields - - list[str] - - Read only the specified fields (columns) from a BigQuery table. Fields may not be returned in the order specified. If no value is specified, then all fields are returned. Example: "col1, col2, col3" -
- table - - str - - The fully-qualified name of the BigQuery table to read from. Format: [${PROJECT}:]${DATASET}.${TABLE} -
-
- ### `MYSQL` Read