/var/log/laughingman7743.log

Releasing flink-connector-gcp 1.1.0: Bigtable write modes, checkpointed sinks, and lineage

2026-09-26T23:47:14+09:00

I have released flink-connector-gcp 1.1.0. This release adds several ways to write Bigtable rows, experimental checkpoint-owned delivery for Bigtable and Cloud Tasks, and FLIP-314 lineage metadata across all five connectors. It also extends BigQuery and Spanner filter pushdown and fixes several source and sink edge cases.

The Maven coordinates remain under io.github.flink-gcp. Use version 1.1.0 with Flink 2.2 or 2.3, and 1.1.0-1.20 with Flink 1.20. For SQL jobs, install the matching flink-sql-connector-gcp-* jar from the release downloads in Flink's lib/ directory. The supported Flink versions have not changed since 1.0.0. No public API or configuration option was removed or renamed, and the new write and delivery modes are opt-in. An existing job therefore keeps its 1.0.0 behavior when its connector dependency is upgraded.

Bigtable writes beyond upsert

Bigtable's Table API sink now selects its operation through sink.write-mode. The default remains upsert. The new choices select different operations for each input row:

Mode	Operation
`insert-if-absent`	Insert cells atomically only if the row has no stored cells.
`keep-latest`	Delete all versions of each targeted cell and write its replacement in one row operation.
`append` and `increment`	Apply read-modify-write rules to the latest stored cell values.
`aggregate`	Contribute values to typed INT64 SUM, MIN, MAX, or HLL aggregate families.
`conditional`	Use a DDL-defined predicate and mutation branches for `CheckAndMutateRow`.

The conditional mode needs the most configuration because the DDL defines a complete row command. This example changes a user's status only if the latest stored status is pending. On a match it also adds one to an aggregate counter; on a mismatch it writes an audit reason.

CREATE TABLE conditional_updates (
  row_key STRING,
  expected_status BYTES,
  new_status BYTES,
  activation_delta BIGINT,
  mismatch_reason BYTES
) WITH (
  'connector' = 'bigtable',
  'project' = 'my-project',
  'instance' = 'my-instance',
  'table' = 'users',
  'sink.app-profile-id' = 'single-cluster',
  'sink.write-mode' = 'conditional',
  'sink.conditional.row-key-column' = 'row_key',
  'sink.conditional.predicate' = 'latest-cell-value-equals',
  'sink.conditional.predicate.family' = 'profile',
  'sink.conditional.predicate.qualifier' = 'status',
  'sink.conditional.predicate.value-column' = 'expected_status',
  'sink.conditional.then.0.operation' = 'set-cell',
  'sink.conditional.then.0.family' = 'profile',
  'sink.conditional.then.0.qualifier' = 'status',
  'sink.conditional.then.0.value-column' = 'new_status',
  'sink.conditional.then.1.operation' = 'add-to-cell',
  'sink.conditional.then.1.family' = 'stats',
  'sink.conditional.then.1.qualifier' = 'activated',
  'sink.conditional.then.1.timestamp-micros' = '0',
  'sink.conditional.then.1.value-column' = 'activation_delta',
  'sink.conditional.otherwise.0.operation' = 'set-cell',
  'sink.conditional.otherwise.0.family' = 'audit',
  'sink.conditional.otherwise.0.qualifier' = 'reason',
  'sink.conditional.otherwise.0.value-column' = 'mismatch_reason'
);

INSERT INTO conditional_updates VALUES (
  'u1',
  CAST('pending' AS BYTES),
  CAST('active' AS BYTES),
  1,
  CAST('status mismatch' AS BYTES)
);

This is a write-only command-input table. Its five physical columns supply arguments to the command; they are not a description of the stored Bigtable row. sink.conditional.row-key-column selects the input column containing the row key, and each value-column names another top-level input column exactly. The CAST(... AS BYTES) expressions supply the byte values compared with or written to Bigtable.

The predicate settings compare the latest profile:status cell with expected_status. If they match, then.0 writes new_status to that cell and then.1 adds activation_delta to stats:activated. The numeric branch indexes set the order of those operations. If the predicate misses, otherwise.0 writes mismatch_reason to audit:reason instead. The explicit timestamp 0 is required for add-to-cell and addresses the aggregate cell at that timestamp.

Create the profile and audit families and an INT64 SUM stats family before running the statement. The application profile must use single-cluster routing with single-row transactions enabled. All referenced inputs must be non-null, including the value used by the branch the service does not select. The mode accepts insert-only input and has at-least-once delivery: recovery may repeat an applied request and select a different branch. The sink discards the Boolean match result. A query that needs it can use BigtableCheckAndMutateFunction on Flink 2.x, with a separate named-request configuration.

The other modes cover simpler cases. insert-if-absent checks whether the entire row has any cell before inserting. keep-latest replaces the versions of the targeted cells immediately, whereas a one-version garbage-collection rule removes old versions asynchronously. append and increment operate on raw cells and can repeat their effect after replay. For counters stored in Bigtable aggregate families, aggregate exposes SUM, MIN, MAX, and HLL contributions through SQL. The Bigtable Table reference has complete, planner-checked examples for every mode and their type and replay rules.

DataStream jobs can also send CheckAndMutateRow and ReadModifyWriteRow requests through the new single-row request sinks. When a SQL query needs the outcome of a conditional write or the cells returned by a read-modify-write request, Flink 2.x has two new asynchronous functions: BigtableCheckAndMutateFunction and BigtableReadModifyWriteFunction. These functions are unavailable in the -1.20 artifacts; the async SQL function guide shows their registration and result types.

Checkpoint-owned Bigtable delivery

Bigtable now has an experimental EXACTLY_ONCE delivery mode. Set sink.delivery-guarantee = 'exactly-once' for a Table sink using upsert, keep-latest, or aggregate. The other new write modes reject that setting. The equivalent DataStream choice is BigtableDeliveryGuarantee.EXACTLY_ONCE.

The writer holds mutations until their owning Flink checkpoint completes. A committer then applies each mutation with a retained marker that prevents a restored checkpoint from applying that mutation again. This matters for aggregate contributions: a stable cell timestamp alone does not prevent a replayed SUM contribution from being added twice. Recover from the latest completed checkpoint with all committer state; if a stop-with-savepoint produced a savepoint, follow the guide's rule for restoring it. Discarding that state after the source has checkpointed records but before their staged mutations reach Bigtable can lose those writes.

The delay is part of the mode's cost. A row becomes readable no earlier than one checkpoint interval later, followed by the time needed to commit the staged writes. Each staged mutation uses a conditional request instead of sharing a bulk MutateRows request, so the drain costs more per row than the default sink. The checkpoint-owned delivery guide describes the required marker family, transactional routing, recovery conditions, and measured latency. The mode ships as experimental; the assessment did not establish a supported workload for its performance gate.

Checkpointed Cloud Tasks creation

Cloud Tasks has a separate experimental EXACTLY_ONCE task-creation mode. It checkpoints a named task before calling CreateTask and reuses the same name and serialized task on recovery. Within the documented recovery window, this protects creation of each staged envelope against a replay. The default remains at-least-once creation.

The Table API selects the mode through sink.delivery-guarantee:

SET 'execution.runtime-mode' = 'STREAMING';
SET 'execution.checkpointing.interval' = '1 s';
SET 'execution.checkpointing.mode' = 'EXACTLY_ONCE';
SET 'execution.checkpointing.checkpoints-after-tasks-finish' = 'true';
SET 'execution.checkpointing.storage' = 'filesystem';
SET 'execution.checkpointing.dir' = 'file:///shared/flink/checkpoints/cloudtasks';
SET 'execution.checkpointing.externalized-checkpoint-retention' = 'RETAIN_ON_CANCELLATION';
SET 'execution.checkpointing.timeout' = '5 min';
SET 'restart-strategy.type' = 'fixed-delay';
SET 'restart-strategy.fixed-delay.attempts' = '3';
SET 'restart-strategy.fixed-delay.delay' = '1 s';

CREATE TABLE generated_tasks (
  payload STRING
) WITH (
  'connector' = 'datagen',
  'rows-per-second' = '10'
);

CREATE TABLE checkpointed_tasks (
  payload STRING
) WITH (
  'connector' = 'cloud-tasks',
  'project' = 'my-project',
  'location' = 'asia-northeast1',
  'queue' = 'webhooks',
  'http.url' = 'https://api.example.com/tasks',
  'format' = 'json',
  'sink.delivery-guarantee' = 'exactly-once',
  'sink.staged.max-tasks' = '1000'
);

INSERT INTO checkpointed_tasks SELECT payload FROM generated_tasks;

Replace the sample endpoint and checkpoint directory, provision the queue, and use a durable checkpoint filesystem shared by the job's processes. The queue must retain task names through the configured recovery window; the creating identity also needs permission to read queue settings for the default retention check. The synthetic datagen input shows the configuration, not event deduplication. The Cloud Tasks checkpointed-creation guide covers the recovery prerequisites and the matching DataStream options. Recovery must retain the latest checkpoint and the sink's mapped committer state. Dropping that state, including by restoring with allowNonRestoredState, can lose checkpointed tasks that have not yet been created.

This mode protects task creation, not handler execution. Cloud Tasks can dispatch a created task more than once, so the handler still needs an idempotent operation or its own deduplication record. Tasks become visible after their owning checkpoint completes; the measured DataStream runs kept up at the rates tested but added most of a checkpoint interval to visibility latency. If records already have stable task IDs, the default writer can collapse duplicate creations within the queue's name-retention window without waiting for a checkpoint.

Lineage across the connectors

All five connectors now report configured Google Cloud resources through Flink's FLIP-314 lineage metadata. For example, a BigQuery sink can identify a configured table, a Pub/Sub source can identify its subscriptions, and a Cloud Tasks sink can identify a fixed queue. These identities describe resources known when the job graph is built; a per-record destination chosen later cannot be inferred from that graph.

On Flink 2.2 and 2.3, a configured job-status listener can receive the metadata. On Flink 1.20, the vertices can be inspected directly but are not delivered to listeners automatically. The connectors provide resource metadata; an exporter or lineage service is a separate component. The lineage guide lists the resource identities and coverage limits.

Other changes

BigQuery's Table source pushes more bounded-scan filters to the service: binary values, TIME with precision 0–3, and TIMESTAMP or TIMESTAMP_LTZ with precision 0–5. Deeply nested predicates no longer risk a stack overflow during filter handling. Spanner bounded reads now push ordered FLOAT64/FLOAT8 key predicates down to the service.

Bigtable Change Streams selected-cell mode now accepts full-column deletes. Bigtable's async function now completes a result when its timeout fires between retry attempts, and Bigtable and Spanner model constructors accept immutable lists. Spanner's ChildPartitionsEvent also reports a null child as a clear argument error.

The documentation site now keeps separate 1.1 and 1.0 pages alongside Development. Use the 1.1 pages for the code in this release: Development follows main and can describe features that have not shipped. The release notes have the complete highlight list and downloads.

This is an independent open-source project. It is not affiliated with, endorsed by, or supported by the Apache Software Foundation or Google. Apache Flink, Flink, and the Flink logo are trademarks of the Apache Software Foundation.

The Cloud Tasks connector in flink-connector-gcp, part five of the connector series

2026-09-06T12:31:17+09:00

This is the final post in the flink-connector-gcp series, about the Apache Flink connectors for Google Cloud I released as 1.0.0. The release post introduces the project; the previous posts cover BigQuery, Pub/Sub, Spanner and Bigtable.

This post uses Cloud Tasks to turn enriched orders into rate-limited calls to a partner API. It covers request construction in SQL, task-id deduplication, authentication, body formats, queue sharding, failure handling and tuning, as well as the exactly-once mode planned for v1.1.0.

Documentation: https://flink-gcp.github.io/flink-connector-gcp/docs/connectors/datastream/cloudtasks/
Maven: io.github.flink-gcp:flink-connector-gcp-cloudtasks:1.0.0 (1.0.0-1.20 for Flink 1.20); for the SQL client, the flink-sql-connector-gcp-cloudtasks uber-jar from the release page goes into lib/

What Cloud Tasks does that a stream needs

Cloud Tasks queues HTTP requests for dispatch. It holds each task until the queue's pacing allows it, retries responses outside the 2xx range under the queue's policy, and can schedule tasks up to 30 days ahead. My pipelines use it to deliver work to handlers, so this connector provides a sink. It has no source for reading a queue back.

I built it for streams from fast sources such as Pub/Sub or Kafka that need to call a slower endpoint, usually a third-party API with a rate limit. Enforcing that limit in Flink requires a stateful throttle; a plain HTTP sink supplies no such control. With Cloud Tasks, the job enqueues records as they arrive and the queue paces requests, retries failures and holds the backlog. The sink therefore has no rate controls of its own. Queue settings determine dispatch speed.

I found no Flink connector for Cloud Tasks when I started, and still found none on GitHub as of 2026-09-05. Apache Beam's built-in I/O list also had no Cloud Tasks entry when checked that day. I used this repository's Pub/Sub sink as the design reference, sharing its serializer boundary and stateless writer that flushes at checkpoints.

In 1.0.0, the DataStream sink provides at-least-once task creation for HTTP and App Engine targets, fixed or per-record queue routing, opt-in named-task deduplication, and a pluggable failure policy. The SQL sink builds requests from a table: a Flink format serializes the body, while writable metadata supplies the URL, method, headers, schedule time and task id.

Enriched orders into a partner API

Order events arrive over Pub/Sub with a customer id. The SQL job looks up the customer's partner endpoint and tenant in Bigtable, then creates an HTTP task on a queue paced to the partner API's limit. The Bigtable post used a fixed target URL. Here each row chooses its URL, method, headers, schedule time and task id.

This code is copied byte for byte from the source-backed example, which CI validates through Flink's planner:

SET 'execution.checkpointing.interval' = '10 s';

CREATE TABLE incoming_orders (
  event_id    STRING,
  customer_id STRING,
  order_id    STRING,
  amount      DECIMAL(12, 2),
  dispatch_at TIMESTAMP_LTZ(6),
  proc_time AS PROCTIME()
) WITH (
  'connector' = 'pubsub',
  'project' = 'my-project',
  'subscription' = 'orders-sub',
  'format' = 'json'
);

CREATE TABLE customer_routes (
  rowkey  STRING,
  routing ROW<endpoint STRING, tenant STRING>,
  PRIMARY KEY (rowkey) NOT ENFORCED
) WITH (
  'connector' = 'bigtable',
  'project' = 'my-project',
  'instance' = 'my-instance',
  'table' = 'customer-routes',
  'lookup.async' = 'true'
);

CREATE TABLE enriched_order_tasks (
  event_id       STRING,
  order_id       STRING,
  amount         DECIMAL(12, 2),
  customer_id    STRING,
  target_url     STRING NOT NULL    METADATA FROM 'url',
  request_method STRING             METADATA FROM 'http-method',
  request_headers MAP<STRING, STRING> METADATA FROM 'headers',
  schedule_at    TIMESTAMP_LTZ(6)   METADATA FROM 'schedule-time',
  dedupe_key     STRING             METADATA FROM 'task-id'
) WITH (
  'connector' = 'cloud-tasks',
  'project' = 'my-project',
  'location' = 'asia-northeast1',
  'queue' = 'partner-api',
  'format' = 'json'
);

INSERT INTO enriched_order_tasks
SELECT e.event_id,
       e.order_id,
       e.amount,
       e.customer_id,
       r.routing.endpoint || '/orders/' || e.order_id,
       'POST',
       MAP['Content-Type', 'application/json', 'X-Tenant', r.routing.tenant],
       e.dispatch_at,
       e.event_id
FROM incoming_orders AS e
JOIN customer_routes FOR SYSTEM_TIME AS OF e.proc_time AS r
  ON e.customer_id = r.rowkey
WHERE e.event_id IS NOT NULL
  AND e.order_id IS NOT NULL
  AND r.routing.endpoint IS NOT NULL
  AND r.routing.tenant IS NOT NULL;

Physical columns form the request body. The format selected in WITH, JSON here, receives exactly the four columns from event_id through customer_id and produces the body bytes. The connector removes the five METADATA columns before serialization, so request settings cannot accidentally enter the JSON.

This separation lets the sink use JSON, CSV, Avro or raw formats with a table-supplied Content-Type. The form format described later supplies its own content type.

A non-null row URL, method or header value overrides its fixed-option counterpart. This table has no http.url, so target_url must be STRING NOT NULL; otherwise planning fails. Every row needs a URL when there is no fixed fallback. Row headers override fixed headers by case-insensitive name.

Schedule time and task id are metadata-only. schedule_at sets a time at most 30 days ahead, with null leaving the service default. The query takes it from dispatch_at, allowing an order due in an hour to be enqueued now without timer state in Flink. The queue's pacing determines when it is dispatched after that time.

The Bigtable lookup is the same kind of join as in the previous post. With this inner join, an event without a customer route produces no task. The WHERE also drops events or route rows missing any of the four required request values. The sink sees neither case, so production jobs should count these rejected inputs separately from successful dispatches.

The route table is trusted configuration. As the documentation advises, enforce an endpoint allowlist before the insert and validate order_id as one path segment before concatenating it. The connector forwards URLs without restricting target hosts or paths.

Checkpointing is required; the example chooses ten seconds. Pub/Sub acknowledges messages when a checkpoint completes. At the barrier, the Cloud Tasks sink waits for every outstanding create, including requests in retry backoff. Under the default failure policy, a completed checkpoint therefore means the service has durably accepted every preceding record except serializer skips.

Without checkpoints, that flush does not run during the stream, and outstanding creates can be lost on failure. Both ends provide at-least-once delivery, which the lookup join does not strengthen.

Successful creation means Cloud Tasks has stored the task. Dispatch and handler success are separate: the queue controls them through its retry policy, outside the sink's view. Handler delivery is at-least-once even when creation is deduplicated. The endpoint still needs an idempotent operation or its own durable business key.

Deduplicating task creation by task id

Tasks are unnamed by default, letting Cloud Tasks assign their names and create them at full speed. A replay can create a second task and call the endpoint again. For handlers that are already idempotent, I keep this at-least-once default.

The task-id metadata column, or taskIdExtractor(...) in DataStream, derives task names from row keys. Once enabled, every row must supply a non-null key. Recreating a name the service still remembers returns ALREADY_EXISTS. The sink counts this as success in tasksDeduplicated, so replay sends another create request without creating another task to dispatch.

The documentation calls this bounded effectively-once task creation. The bound comes from name retention after execution or deletion. Google's sources disagree on that duration: the REST reference says a released id can take up to 24 hours, while the v2 proto comment says about an hour. The connector advises designing against the shorter window. A scheduled task retains its id for its lifetime plus that window.

The guarantee covers creation only. Cloud Tasks still delivers the handler at least once.

A task name identifies an immutable logical task. Cloud Tasks cannot update a created task, and the sink accepts ALREADY_EXISTS without comparing its payload or schedule. If changed data should create another task, include a content or schedule version in the key. That is the role of -v1 in the documentation's external-API example.

The sink uses the key's SHA-256 digest as the task id. Google's tasks.create reference recommends hashed strings and warns that sequential ids, "for example using a timestamp", increase latency and error rates across task commands. Hashing keeps deterministic deduplication while allowing the job to supply an event id or offset without creating sequential task names. The tradeoff is readability: the business key alone does not reveal the task's console name.

Naming is opt-in because of its service-side cost. The same reference says the additional lookup causes "significantly increased latency", without quantifying it. The project's benchmark compared deterministic ids with unnamed creation on a paused queue, using the preregistered exactly-once performance gate applied to candidate modes.

The averages met the general gate, but the run-to-run throughput range was about 11 percent, above the 10 percent limit. The result remains inconclusive. For the current mode, I enable naming when duplicate-call costs justify the measured latency on that queue.

For v1.1.0, I am planning an opt-in exactly-once mode using Flink's two-phase commit (#1238). It is not implemented yet. The sink would stage tasks in checkpointed state and create them only after checkpoint completion, preserving each task's identity and contents for commit retries.

The guarantee would cover task creation within a verified recovery window; handler delivery would remain at least once. That window needs a safety margin inside a verified minimum name-retention period. Neither the one-hour nor the 24-hour description above establishes that minimum. After the recovery deadline, the job would stop creating tasks and retain pending work in Flink state for operator recovery.

The initial scope is checkpointed streaming jobs writing to one fixed, pre-provisioned queue. Shipping the mode depends on verifying the service contract and passing recovery and performance tests; the existing at-least-once mode would remain the default.

Two identities behind an authenticated call

An authenticated pipeline uses two identities. The first calls CreateTask, using application-default credentials or a service-account key file. With a key file, only the path travels in the job graph; each TaskManager reads it when its writer starts.

The second identity authenticates the HTTP request when Cloud Tasks later dispatches it. Its token settings are configured on the table or serializer. They do not authenticate Flink or grant permission to enqueue tasks.

For a Cloud Run service or function protected by IAM, the dispatch token is OIDC. The documentation provides this source-backed example:

CREATE TABLE function_tasks (
  order_id     STRING,
  amount       DECIMAL(12, 2),
  trace        MAP<STRING, STRING> METADATA FROM 'headers',
  schedule_at  TIMESTAMP_LTZ(6)    METADATA FROM 'schedule-time',
  dedupe_key   STRING              METADATA FROM 'task-id'
) WITH (
  'connector' = 'cloud-tasks',
  'project' = 'my-project',
  'location' = 'asia-northeast1',
  'queue' = 'functions',
  'http.url' = 'https://process-order-abc-an.a.run.app/tasks',
  'http.method' = 'POST',
  'http.headers.Content-Type' = 'application/json',
  'http.oidc.service-account-email' =
    'dispatcher@my-project.iam.gserviceaccount.com',
  'http.oidc.audience' = 'https://process-order-abc-an.a.run.app',
  'format' = 'json'
);

INSERT INTO function_tasks
VALUES (
  'o-42',
  CAST(19.95 AS DECIMAL(12, 2)),
  MAP['X-Trace-Id', 'trace-42'],
  CAST(CURRENT_TIMESTAMP + INTERVAL '5' MINUTE AS TIMESTAMP_LTZ(6)),
  'order-o-42'
);

The audience uses the service's stable root run.app URL. Without an explicit audience, Cloud Tasks uses the full target URL, including its path. The handler must accept whichever audience is configured.

The deployment must create the IAM bindings; the connector does not. The task creator needs enqueue permission and iam.serviceAccounts.actAs on the dispatch service account. That account must belong to the queue's project and have the target's invoker role. OIDC settings are fixed table options because SQL exposes no per-row dispatch identity.

OAuth supplies an access token for Google APIs on *.googleapis.com, the use Google generally documents for it. OIDC and OAuth occupy one protobuf oneof, so the builder rejects setting both. A partner endpoint that validates Google-issued tokens can use OIDC. An endpoint using its own credentials can receive them in a row-supplied or fixed header.

The endpoint must also be reachable. Google documents HTTP targets as endpoints with external IP addresses. One supported exception is Cloud Run with internal ingress: Cloud Tasks can reach it in the same project or VPC Service Controls perimeter through its default run.app URL. A valid token alone does not establish network reachability.

Before relying on per-row URLs, check for a queue-level URI override. Its enforcement mode defaults to always, so it can silently replace every task's URL. The sink's v2 client cannot read that field and therefore cannot guard against it. The documentation records the interaction; I check the queue configuration before using row URLs.

The connector also supports App Engine targets, although I do not use this integration in my own pipelines. In SQL, target.type = 'app-engine' selects a relative URI with optional service, version and instance routing. Tasks reach the application through Google's internal transport, so this target does not accept OIDC or OAuth options. The project's gated tests validate this support against real App Engine.

Bodies the target API expects

The target API determines the body format. SQL can use any serialization format on the classpath whose schema requirements match the physical columns. The documentation shows the resulting bytes for nested JSON, CSV with configurable delimiters, quoting and null literals, raw single-column data in a chosen charset, and binary Avro. Avro's writer schema comes from the physical columns and must be shared with the handler.

The connector passes these bytes through without inspecting them, so the table must set the appropriate Content-Type.

The module also provides form-urlencoded for application/x-www-form-urlencoded bodies. It accepts STRING and ARRAY<STRING> columns and sets its own Content-Type. Non-null columns produce fields in physical schema order; array elements produce repeated fields. Null columns and empty arrays add nothing, while a null array element fails the row.

The format does not invent encodings for numbers, booleans, nested rows or maps. SQL must cast scalars to strings to make the wire representation explicit. Quoted column names such as `items[]` can express bracket or dotted names expected by the server.

Dynamic field names, indexed arrays and arbitrary maps require a scalar function to build the complete body and the raw format to pass it through. Sending pre-encoded content through form-urlencoded would encode it again.

HTTP targets serialize bodies only for POST, PUT and PATCH; App Engine targets do so only for POST and PUT. Other methods skip body serialization. A GET request therefore takes its query string from the URL. Use the url metadata column with Flink 2.x's URL_ENCODE, or a UDF on 1.20, to escape values before concatenation. The connector does not repair an unescaped query string.

A DataStream serializer can return a complete Task proto, including a per-record dispatch deadline and multipart body, neither of which has a SQL representation. The httpTarget(url) and appEngineTarget(relativeUri) helpers bind a body SerializationSchema and provide URL, header and routing resolvers alongside fixed token settings. Per-record authorization requires the full Task serializer.

Returning null from the task serializer skips a record. Nothing is written, and the recordsSkipped counter reports the skip. The body-serialization helpers cannot skip this way: null from their wrapped Flink serializer is a serialization failure.

The task serializer must leave the name unset. The sink constructs it from the resolved queue and hashed key, keeping naming on one path.

Sharding across queues from the DataStream API

Each SQL table writes to one fixed queue. DataStream can resolve a queue for each record through the same resolver pattern used for tables and topics in the earlier connectors. Sharding across queues increases aggregate dispatch capacity when one queue's service limits are insufficient. This source-backed example keeps each customer on one queue:

CloudTasksSink.<OrderEvent>builder()
        .destinationResolver(
                (element, context) ->
                        QueueDestination.of(
                                "my-project",
                                "asia-northeast1",
                                "webhooks-"
                                        + Math.floorMod(
                                                element.customerId().hashCode(), 4)))
        .serializer(
                CloudTasksSerializationSchema.httpTarget(
                                "https://api.example.com/v1/orders")
                        .withBody(new OrderEventSchema()))
        .build();

One client serves all queues. Routing creates no per-queue client, stream or batcher to cache or evict. A destination includes its location because queues are regional.

The sink holds no per-queue state except optional per-destination metrics. Those metrics are off by default because Flink cannot unregister them: every queue used would retain counters for the task's lifetime. Any resolver cache belongs to the resolver, travels in the job graph and should also be bounded.

Every destination queue must already exist. The sink offers no create disposition because queue settings define the pacing this pipeline needs. Automatic creation would use defaults of 500 dispatches per second and 1,000 concurrent dispatches, potentially exceeding the endpoint's capacity. A deleted queue name also cannot be reused for three days, making a mistaken creation slow to undo.

Queues are infrastructure the job references. The quickstart begins by creating one with limits the endpoint can sustain.

What a failed task means

Three failure types reach failedTaskHandler: serializer rejection, an exception from the task-id extractor, and a service INVALID_ARGUMENT response, such as a malformed target, oversized body or rejected header. The default handler fails the job; alternatives can log and drop or forward to a dead-letter queue.

The shared Pub/Sub dead-letter implementation works unchanged. When serialization succeeded, its message contains the full serialized task. A consumer can call Task.parseFrom to recover the target, method, headers and authorization settings. A serialization failure produces empty message data. SQL exposes no failure-policy option and fails the job on the first routed failure.

An exhausted retry budget, missing queue or PERMISSION_DENIED fails the job without reaching a dropping handler. These failures require backpressure and restart rather than discarding a stream during an outage. A transient status anywhere in the cause chain takes precedence over INVALID_ARGUMENT, preventing instability from being classified as a dead letter.

Configuration errors also remain fatal. A null resolver result or empty extracted key would otherwise drop every record while the job appeared healthy. A serializer bug that makes every task invalid can still reach a dropping handler one record at a time. Monitor numRecordsSendErrors, which counts all tasks delivered to the handler, whether dropped or not.

A task dropped during creation never entered the queue, so the queue's retry policy never applied. Once created, a task whose handler fails is retried by Cloud Tasks outside the sink's view. App Engine has one documented distinction: a handler's 503 throttles the queue, while 429 does not trigger congestion control.

Paused and disabled queues need separate monitoring. Both accept new tasks while stopping dispatch; the paused-queue contract says it "will stop delivering tasks from it, but more tasks can still be added to it". The sink can therefore appear healthy while the backlog grows.

The knobs that matter

The writer has few options because dispatch pacing belongs to the queue. Configure maxDispatchesPerSecond, maxConcurrentDispatches and the handler retry policy there.

The sink owns create retries. The generated client sets a 20-second timeout for CreateTask and no retryable status codes; an unnamed create is non-idempotent. Failed creates are parked with a due time and retried from a subsequent write or flush.

UNAVAILABLE, DEADLINE_EXCEEDED and RESOURCE_EXHAUSTED use up to eight attempts including the first. Backoff starts at 100 ms, doubles and caps at 10 s. NOT_FOUND has a separate three-attempt budget with delays from 500 ms to 2 s. Both schedules use up to 25 percent jitter in either direction.

The shorter NOT_FOUND budget balances two cases. A queue idle for 30 days can take minutes to reactivate and return NOT_FOUND meanwhile, so that status alone does not prove the name is wrong. But a mistyped queue should fail without exhausting the longer budget for every record. Reactivation can outlast the short budget; the job's restart strategy handles that case.

Retrying DEADLINE_EXCEEDED can duplicate an unnamed task if the first create succeeded. At-least-once favors that over loss. Named-task deduplication removes this ambiguity within its retention window.

maxInFlightTasks, default 1,000, bounds outstanding creates, including parked retries. At the cap, a write yields to the task mailbox until completions free capacity.

Transport capacity can bind first. The client opens one gRPC channel by default, and one HTTP/2 channel supports about 100 concurrent streams. A subtask therefore runs about 100 concurrent creates regardless of a higher in-flight cap. Measurements on 2026-08-22 against a real paused queue reached about 210 creates per second per subtask with one channel and 1,271 with eight.

channelPoolSize allows more channels but defaults to one. Automatically sizing the pool from the in-flight cap could exceed Google's guidance of roughly 1,000 tasks per second per queue, counting creates and dispatches, as the eight-channel measurement already does. It could also violate the ramp rule of increasing traffic by no more than 50 percent every five minutes.

Increase the pool deliberately, using roughly one channel per 100 concurrent creates needed. A pipeline paced to a partner API usually does not need more channels unless ingestion bursts alone approach the queue guidance.

The project evaluated batch creation and declined it in ADR-0129. The API exists only in v2beta3 as a long-running, explicitly non-atomic operation accepting up to 100 tasks. On the same measurement day, it was no faster at the median than 100 concurrent single creates and about twice as slow at the tail.

Its failure semantics ruled it out. A batch containing existing named tasks returns one ALREADY_EXISTS for the whole batch, without per-task results, while still creating its non-duplicate tasks. A sink cannot map that outcome reliably to per-task results while treating deduplication as success. Creation therefore remains one RPC per record, with the sink managing buffering, backpressure and concurrency.

Testing without an official emulator

Google provides no official Cloud Tasks emulator. Integration tests use the MIT-licensed community aertje/cloud-tasks-emulator through testcontainers, without cloud credentials, on pull requests affecting the module. It dispatches over real HTTP, allowing a server in the test JVM to record each request's method, path, body and headers. The tests also inspect OIDC bearer JWTs for the configured account and audience.

Stored-task assertions use paused queues. Running queues delete tasks as soon as they complete, which would race inspections of the task itself. Separating these checks verifies both stored request settings and what reaches the handler.

The emulator never garbage-collects task names, so it can test ALREADY_EXISTS but not the deduplication window. It also lacks queue-level URI overrides, App Engine dispatch, OAuth tokens, failure injection and task-size enforcement. Transient retries are therefore unit-tested with a fake creator and injected clock.

The connector leaves size enforcement to the service. As of 2026-09-05, Google's create reference still specified 100 KB while its quotas page specified 1 MiB. The connector validates against neither conflicting figure, and the documentation advises staying within the smaller one.

The gated App Engine suite checks request construction and routing on paused queues, reads a queue-level routing override and observes failed handler attempts against a real App Engine Standard fixture. The fixture starts for the test class and returns to zero instances afterwards, with a scheduled sweep restoring the idle state after hard cancellation.

Closing the series

This completes the five-connector series, with each post built around a pipeline I needed. Cloud Tasks illustrates the shared design approach: expose service capabilities and add the Flink behavior around them.

The queue controls handler pacing, scheduling and retries. The sink retries only creation requests, hashes business keys for service-side deduplication, and makes INVALID_ARGUMENT eligible for failure routing. Its Flink-specific responsibility is to wait at a checkpoint until every non-skipped create is durable under the default failure policy. The SQL API expresses the request as a table.

The v1.1.0 milestone now includes the Cloud Tasks exactly-once mode alongside Bigtable's write extensions. The Cloud Tasks tracker covers the service investigation, staged writer and committer, DataStream and Table APIs, and recovery and performance validation. RPC-level metrics through the client tracer remain planned for v1.3.0. Reports about targets or body formats the SQL API cannot express will help guide further work: https://github.com/flink-gcp/flink-connector-gcp/issues

The Bigtable connector in flink-connector-gcp, part four of the connector series

2026-09-05T16:33:07+09:00

This is the fourth post in the flink-connector-gcp series, about the Apache Flink connectors for Google Cloud I released as 1.0.0. The release post introduces the project; the previous posts cover BigQuery, Pub/Sub and Spanner.

The Bigtable connector grew out of a need to enrich streams with data from Bigtable. This post follows that SQL pipeline, then covers the full-table cache, how cell timestamps affect replayed writes, change streams, and the write extensions planned for v1.1.0.

Documentation: https://flink-gcp.github.io/flink-connector-gcp/docs/connectors/datastream/bigtable/
Maven: io.github.flink-gcp:flink-connector-gcp-bigtable:1.0.0 (1.0.0-1.20 for Flink 1.20); for the SQL client, the flink-sql-connector-gcp-bigtable uber-jar from the release page goes into lib/

What existed, and what I needed

Google's flink-connector-gcp is a separate project with the same name. It publishes a Bigtable sink to Maven Central, with DataStream and Table APIs and a custom-serializer SPI. It has neither a scan source nor a lookup source. Its change-stream source has been an open pull request since March 2026, last updated in April. When I checked on 2026-09-04, its only commit in the preceding five months was a Flink version bump.

Two smaller concerns also made me hesitate to build on it. The connectors directory contains Bigtable and a BigQuery module that the README describes as catalog support for the separately maintained Dataproc connector. The examples are spread across several top-level directories. I saw no basis to expect those gaps or the layout to change, so I built my own connector.

My pipelines mostly use Bigtable as a low-latency attribute store. An event carries a user id, and a Bigtable row holds that user's current attributes. A lookup join adds those attributes to the event without exporting the table. I also wanted to consume a change stream from the same table.

I also considered Flink's HBase connector over Google's HBase-compatible client library, but its releases did not cover the Flink versions I needed. As of 2026-09-04, the newest apache/flink-connector-hbase artifacts on Maven Central were 4.0.0 builds for Flink 1.18 and 1.19, published in November 2024. There were none for 1.20 or 2.x.

I did adopt its DDL model: one plain column for the row key, one ROW per column family, and the HBase ecosystem's byte encodings for cells. An HBase connector table definition can therefore keep its schema when moved to this connector, and each connector can read tables written by the other.

The resulting module has a bounded scan source, a change-stream source and an at-least-once sink in the DataStream API. The SQL connector adds lookup joins with three cache modes and a changelog interpretation of the change stream. I also adopted Google's serializer interface shape, so porting a serializer requires changing the interface name.

An enrichment pipeline in Flink SQL

JSON events arrive over Pub/Sub with a user id. A Flink SQL job looks up each user's Bigtable row through a cache, then creates an HTTP task in Cloud Tasks for each enriched event. The next post covers Cloud Tasks in detail.

The job consists of three tables and one INSERT. This code is copied byte for byte from the source-backed example, which CI validates through Flink's planner:

SET 'execution.checkpointing.interval' = '10 s';

CREATE TABLE incoming_events (
  event_id STRING,
  user_id STRING,
  event_type STRING,
  message_id STRING METADATA FROM 'message-id' VIRTUAL,
  proc_time AS PROCTIME()
) WITH (
  'connector' = 'pubsub',
  'project' = 'my-project',
  'subscription' = 'events-sub',
  'format' = 'json'
);

CREATE TABLE user_attributes (
  rowkey STRING,
  profile ROW<tier STRING, api_path STRING>,
  PRIMARY KEY (rowkey) NOT ENFORCED
) WITH (
  'connector' = 'bigtable',
  'project' = 'my-project',
  'instance' = 'my-instance',
  'table' = 'user-attributes',
  'scan.row-prefix' = 'user#',
  'lookup.async' = 'true',
  'lookup.cache' = 'PARTIAL',
  'lookup.partial-cache.max-rows' = '10000',
  'lookup.partial-cache.expire-after-write' = '10 min'
);

CREATE TABLE api_tasks (
  event_id STRING,
  user_id STRING,
  event_type STRING,
  user_tier STRING,
  api_path STRING,
  source_message_id STRING,
  request_headers MAP<STRING, STRING> METADATA FROM 'headers'
) WITH (
  'connector' = 'cloud-tasks',
  'project' = 'my-project',
  'location' = 'asia-northeast1',
  'queue' = 'events',
  'http.url' = 'https://api.example.com/events',
  'http.method' = 'POST',
  'http.headers.Content-Type' = 'application/json',
  'format' = 'json'
);

INSERT INTO api_tasks
SELECT
  e.event_id,
  e.user_id,
  e.event_type,
  a.profile.tier,
  a.profile.api_path,
  e.message_id,
  MAP['X-Source-Message-Id', e.message_id]
FROM incoming_events AS e
JOIN user_attributes FOR SYSTEM_TIME AS OF e.proc_time AS a
  ON e.user_id = a.rowkey;

The Bigtable DDL follows the HBase connector's model. Exactly one column is not a ROW: the row key. Its type determines the key's byte encoding. A STRING key uses UTF-8; a BIGINT key uses eight big-endian bytes.

Each ROW column represents a column family, and its fields name the qualifiers, with one cell per field. Here, profile ROW<tier STRING, api_path STRING> defines the profile family with two qualifiers.

Cell encodings also follow HBase conventions, allowing the connector to read tables written by other systems that use those conventions. For keys or cells written with a different encoding, declaring BYTES or STRING is safer than decoding them as numbers.

The selected columns determine which families Bigtable returns. This query reads a.profile.tier and a.profile.api_path, so the point read includes a filter for the profile family. Families declared in the DDL but unused by the query are not transferred.

That filter also affects which rows the lookup finds. A Bigtable row exists only while it has a cell. If none of its cells belong to a family the query reads, the filter returns nothing. The inner join then produces no task for that event. The same happens when the user id falls outside user#, because scan.row-prefix bounds point reads as well as scans. A LEFT JOIN would retain these events with null attributes.

The join must use equality on the single row-key column. Unlike the Spanner example, it cannot join on composite keys or nested family fields; Flink rejects those joins during planning. A Bigtable point read addresses one atomic row key. To look up a row by two values, encode both into that key.

The example uses Flink's standard lookup cache. With lookup.async = 'true', point reads can overlap instead of blocking the operator for each event. With lookup.cache = 'PARTIAL', the cache loads entries on demand and retains them under at least one configured bound. Here the bounds are 10,000 rows and a ten-minute write expiry. A frequently accessed user that stays cached is read once per ten minutes rather than once per event, at the cost of up to ten minutes of stale attributes.

The checkpoint interval serves the Pub/Sub source: it acknowledges messages when a checkpoint completes. Both ends of this pipeline provide at-least-once delivery. The lookup join does not strengthen that guarantee.

Deployment requires three uber-jars in the SQL client's lib/, one per connector. Their dependencies are relocated so the jars can coexist.

Google's Bigtable connector also registers the bigtable identifier. Installing both Bigtable connectors as separate jars makes factory discovery fail with an ambiguity error naming that identifier. Merging them into one fat jar can instead let whichever registration survives silently control the DDL. Keep the jars separate so the conflict remains visible.

The full-table cache, and when to reach for it

The Spanner post mentioned that its lookup source rejects a FULL cache, while Bigtable offers one. The choice here depends on how large the lookup table is and how often it changes.

The three cache modes differ in where reads happen. NONE performs a point read for every event. PARTIAL reads on a cache miss and retains the result within its configured bounds.

With FULL, each lookup task loads every projected row within the configured key bounds through a bounded scan. All joins use that local copy, with no point reads. Reloads can run periodically through lookup.full-cache.periodic-reload.interval, in fixed-delay or fixed-rate mode, or at a time of day through lookup.full-cache.timed-reload.iso-time, at a configured interval in days.

A full-cache lookup is synchronous, so combining it with lookup.async = 'true' fails during planning. The scan.row-prefix and range bounds apply to the cache load just as they do to point reads.

A full cache suits a dimension table that fits comfortably in a subtask's heap, receives frequent lookups, and changes on a schedule. Consider a few hundred thousand user attributes rebuilt nightly. A timed reload after the rebuild makes every join local, with no Bigtable traffic between reloads. The cost is one copy of the projected table per lookup task, so memory use scales with lookup parallelism.

My reading of the two designs is that the difference starts with what each service's scan promises. A Spanner snapshot read runs at one timestamp in a batch transaction whose lifetime is bounded by version retention. A full cache would need to define the snapshot it holds and how successive reloads relate, a contract that source deliberately does not offer.

A Bigtable scan returns the latest cell versions as it reads each row, without a single snapshot timestamp. The cache uses the connector's bounded scan source as its loader and retains those scan semantics.

A FULL cache reload is a scan, so it can use a Data Boost application profile. Data Boost is Bigtable's read-only serverless compute and keeps the reload off the serving cluster. Point-read modes cannot use it.

Two limits matter here. Bigtable gives no guarantee for data written less than 35 minutes before a Data Boost read, so a reload may omit the newest rows. A parallel reload above 1,000 read requests per second per cluster is reported as ineligible and billed accordingly, without an error.

This project has not exercised Data Boost, which requires an Enterprise-edition instance. The gated suite checks only that the configured profile id reaches the client. I therefore consider the full-cache combination plausible but unproven.

Keeping the lookup table current

Writing the lookup table raises two questions: what happens when an update is replayed, and what happens when two updates target the same key? A second SQL job maintains the table from a stream of profile updates. It adapts the source-backed cell-timestamp example; I added the Pub/Sub source:

CREATE TABLE profile_updates (
  user_id    STRING,
  tier       STRING,
  api_path   STRING,
  updated_at TIMESTAMP_LTZ(3)
) WITH (
  'connector'    = 'pubsub',
  'project'      = 'my-project',
  'subscription' = 'profile-updates-sub',
  'format'       = 'json'
);

CREATE TABLE user_attributes_out (
  rowkey         STRING,
  profile        ROW<tier STRING, api_path STRING>,
  cell_timestamp TIMESTAMP_LTZ(6) METADATA FROM 'timestamp',
  PRIMARY KEY (rowkey) NOT ENFORCED
) WITH (
  'connector' = 'bigtable',
  'project'   = 'my-project',
  'instance'  = 'my-instance',
  'table'     = 'user-attributes',
  'sink.insert-only-input-mode' = 'insert-only'
);

INSERT INTO user_attributes_out
SELECT user_id, ROW(tier, api_path), CAST(updated_at AS TIMESTAMP_LTZ(6))
FROM profile_updates;

The sink advertises an upsert contract by default because setCell overwrites cells under the row key. A changelog DELETE removes the whole row. Declaring the row key as a primary key is optional, as with the HBase connector. On Flink 2.x, declaring it can make updating queries cheaper: a delete may carry only the key, avoiding the stateful ChangelogNormalize operator that would otherwise complete the row.

The insert-only option addresses a planning rule in Flink 2.3. A keyed upsert sink requires ON CONFLICT when the input's upsert key differs from the sink's or cannot be inferred, as with this append stream. The option narrows an insert-only input to an insert-only contract, allowing this statement to plan without the clause on Flink 1.20, 2.2 and 2.3. The physical write is unchanged: it still overwrites cells under an existing row key.

The cell_timestamp column determines what a replay does to a cell. This is an at-least-once sink, so a restart replays records written since the last completed checkpoint. With a stable explicit timestamp, replay writes the same value to the same cell version. Without one, the sink uses the writer's clock, and replay adds another version. The table's garbage-collection policy determines how long those versions remain.

The query takes its timestamp from the record's updated_at, making a replay of that cell write a no-op. Bigtable's storage model absorbs the duplicate; the sink does not track or reject replayed records. This does not cover writes using the writer's clock, non-idempotent mutations, or collisions between distinct events that target the same cell version.

On a table with the default timestamp granularity, an explicit timestamp must be millisecond-aligned or Bigtable rejects it with INVALID_ARGUMENT. The opt-in sink.cell-timestamp.truncate-to-millis drops the extra precision. An absent or null timestamp uses a millisecond-aligned writer clock.

The sink does not guarantee the order of writes to one key. Bigtable's bulk mutation contract says entries "may be applied in arbitrary order (even between entries for the same row)". Requests can also overlap in flight. Two updates to one user may therefore be applied in a different order from the order in which the job submitted them.

Before deciding whether to add ordering machinery, the project measured the service. On 2026-08-11, a campaign submitted 86,196 same-row pairs in mirrored arms against a one-node SSD instance. Request sizes ranged from 2 through 19,998 entries, and the campaign observed zero reversals. That observation does not change the service contract. The connector documents the caveat rather than adding a per-key queue for reversals the campaign did not observe.

For timestamped profile updates, a latest-cell read selects by the record-supplied cell timestamp, regardless of mutation arrival order. Two updates within the same millisecond can still target one version with no defined winner, and a row delete carries no timestamp. Those cases remain subject to the ordering caveat.

In the DataStream API, the serializer makes the same choices. It returns the client's RowMutationEntry, which can combine several cell writes and a delete into one atomic operation on a row. This example is adapted from the source-backed examples, which CI compiles:

BigtableSink.<OrderEvent>builder()
        .table(TableDestination.of("my-project", "my-instance", "orders"))
        .serializer(
                (event, context) -> {
                    long timestampMicros = event.updatedAtMillis() * 1_000;
                    RowMutationEntry entry = RowMutationEntry.create("order#" + event.id());
                    entry.setCell("cf", "status", timestampMicros, event.status());
                    entry.setCell("cf", "total", timestampMicros, event.totalCents());
                    if (event.isCancelled()) {
                        entry.deleteCells("cf", "reserved_stock");
                    }
                    return entry;
                })
        .build();

Multiplying by 1,000 converts the record's epoch milliseconds to the microseconds setCell expects. A serializer can also return null to skip a record, for example when a filter depends on the mutation being built. Nothing is written, and only the recordsSkipped counter reports the skip.

Dynamic destinations work as in the earlier posts. A resolver returns a TableDestination for each record, allowing a table per tenant or per day. The writer maintains one bulk mutation batcher per table and one client per instance.

Auto-creation requires an explicit schema: column families and their garbage-collection policies. The sink cannot infer these, so CREATE_IF_NEEDED requires table-create options naming at least one family. As in the Pub/Sub sink, creation is reactive. No admin client exists until a mutation fails with NOT_FOUND. The sink then ensures the table and its declared families exist and reapplies the failed mutations.

Garbage collection needs a deliberate choice. A family without a rule keeps Bigtable's default of collecting nothing. If the sink uses wall-clock timestamps, duplicate versions from replays can then accumulate forever. The SQL layer therefore requires a max-versions or max-age rule under create-if-needed. A typical rule for retaining only the latest cell is GcRule.union(GcRule.maxVersions(1), GcRule.maxAge(...)).

Google's connector calls timestamp-based idempotence "Exactly Once out of the box". Its writer flushes at the checkpoint barrier without a committer, and most of its built-in serializers use the Flink record timestamp for each cell. The underlying mechanism is the same, but this connector does not label it exactly-once.

Stable timestamps cannot make an increment or append idempotent. Bigtable's primitive for protecting such a write from replay is a conditional single-row mutation: apply the data only if a marker in the row says the write has not already happened.

The project benchmarked this primitive against the real service under conditions fixed in advance. Two runs were inconclusive because throughput varied too widely between repetitions. On 2026-09-05, a run that isolated each repetition in its own JVM passed: about 1.5 times the bulk baseline's throughput, at two thirds of its tail latency, with a clean replay read-back. A separate probe showed that the conditional branch also absorbed a replayed increment.

That result supports a planned exactly-once mode for v1.1.0 (#1211). It is not implemented yet. The proposal uses Flink's two-phase commit: stage records during a checkpoint interval, then apply them at checkpoint completion through conditional writes keyed on a checkpoint-id cell in each row. The intended guarantees are idempotent commit retries and no later data left visible after restoring an earlier checkpoint.

The expected costs are one conditional request per row and loss of staged records if Flink state is discarded, as with BigQuery's buffered streams. The issue's design record will settle the remaining details. Until this mode ships, BigQuery is the only connector in the family with a supported exactly-once sink mode.

Change streams without a metadata table

A Bigtable change-stream record describes one atomic row mutation as the service applied it. It contains the row key and an ordered list of entries: SetCell, DeleteCells, DeleteFamily, and the aggregate kinds AddToCell and MergeToCell. It also carries a commit timestamp, a tie breaker, the source cluster and the partition's estimated low watermark.

Unlike Spanner's change stream, it supplies neither a before image nor a complete after image of the row. That difference determines how the connector's two SQL modes interpret the stream.

The envelope mode preserves the mutation as one insert-only row, with its entries in an array in service order. This example is copied verbatim from the source-backed example:

CREATE TABLE profile_mutations (
  row_key BYTES,
  entries ARRAY<ROW<
    entry_index INT,
    kind STRING,
    family STRING,
    qualifier ROW<value_type STRING, bytes_value BYTES, long_value BIGINT>,
    `timestamp` ROW<value_type STRING, bytes_value BYTES, long_value BIGINT>,
    `value` ROW<value_type STRING, bytes_value BYTES, long_value BIGINT>,
    delete_range ROW<
      start_bound STRING,
      start_micros BIGINT,
      end_bound STRING,
      end_micros BIGINT
    >
  >>,
  mutation_type STRING NOT NULL
    METADATA FROM 'mutation-type' VIRTUAL,
  commit_timestamp TIMESTAMP_LTZ(9) NOT NULL
    METADATA FROM 'commit-timestamp' VIRTUAL,
  source_cluster_id STRING
    METADATA FROM 'source-cluster-id' VIRTUAL
) WITH (
  'connector' = 'bigtable',
  'project' = 'my-project',
  'instance' = 'my-instance',
  'table' = 'profiles',
  'scan.mode' = 'change-stream',
  'scan.change-stream.changelog-mode' = 'envelope',
  'scan.app-profile-id' = 'single-cluster-profile'
);

SELECT
  row_key,
  mutation_type,
  commit_timestamp,
  entry_index,
  kind,
  family,
  qualifier,
  entry_timestamp,
  entry_value,
  delete_range
FROM profile_mutations
CROSS JOIN UNNEST(entries) AS entry_table(
  entry_index,
  kind,
  family,
  qualifier,
  entry_timestamp,
  entry_value,
  delete_range
);

In envelope mode, a deletion is an inserted log record, just like any other mutation. There is no row image to turn into a Flink DELETE. The stream also includes garbage-collection mutations alongside application writes across all column families; mutation_type distinguishes them.

An envelope table rejects a primary key. Two mutations to the same Bigtable row are two log records, so feeding a keyed upsert sink requires the job to reconstruct rows in its own stateful operator.

The envelope is a useful history format: it records which cells changed and when. entry_index preserves each entry's position through UNNEST. These records can feed BigQuery's append path, like the history pattern in the Spanner post.

The selected-cell mode produces a keyed changelog, but requires a specific producer protocol. One configured cell must contain the complete serialized non-key part of a logical row. The source decodes it with a Flink format and emits keyed UPDATE_AFTER and DELETE rows.

The resulting changelog can feed the BigQuery CDC sink from the first post. This code is copied verbatim from the source-backed example:

SET 'execution.checkpointing.interval' = '1 min';

CREATE TABLE current_profiles (
  profile_id STRING NOT NULL,
  name STRING,
  tier STRING,
  PRIMARY KEY (profile_id) NOT ENFORCED
) WITH (
  'connector' = 'bigtable',
  'project' = 'my-project',
  'instance' = 'my-instance',
  'table' = 'profiles',
  'scan.mode' = 'change-stream',
  'scan.change-stream.changelog-mode' = 'selected-cell',
  'scan.app-profile-id' = 'single-cluster-profile',
  'scan.change-stream.selected-cell.family' = 'state',
  -- Base64 for the qualifier "current".
  'scan.change-stream.selected-cell.qualifier-base64' = 'Y3VycmVudA==',
  'scan.change-stream.selected-cell.source-cluster-id' = 'cluster-a',
  'value.format' = 'json'
);

CREATE TABLE analytics_profiles (
  profile_id STRING NOT NULL,
  name STRING,
  tier STRING,
  PRIMARY KEY (profile_id) NOT ENFORCED
) WITH (
  'connector' = 'bigquery',
  'project' = 'my-project',
  'dataset' = 'analytics',
  'table' = 'current_profiles',
  'sink.cdc.enabled' = 'true',
  'sink.create-disposition' = 'create-if-needed',
  'sink.cdc.max-staleness' = '10 min'
);

INSERT INTO analytics_profiles
SELECT profile_id, name, tier FROM current_profiles;

The producer must follow the documented replacement protocol. An upsert must atomically delete the selected column across all timestamps, or its entire family, and then set exactly one replacement cell. The same delete without a following set becomes a key-only delete. Entries for other cells produce no row.

The source cannot infer this protocol from arbitrary traffic. A standalone, repeated or out-of-order selected write fails the job, as does writing from a second cluster. The writer must replace the complete value using the required mutation shape.

Compared with the Spanner replication example, the producer supplies the row contents that Bigtable itself does not provide. This pipeline also sends BigQuery no sequence metadata, so colliding changes to one key resolve by arrival order. It is an analytics replica starting at latest; an already populated source table needs a separate initial snapshot.

Both SQL modes use the partition machinery described in the Spanner post. Bigtable change-stream partitions split and merge over time. Each open partition read is a streaming RPC carrying mutations, heartbeats and continuation tokens. The coordinator stores assigned and unassigned partitions, pending merges and resume tokens in Flink checkpoints.

No metadata table is created, so there is no additional stateful resource to provision, grant write access to or clean up. By comparison, Apache Beam's BigtableIO.readChangeStream "creates and manages a metadata table to manage the state of the connector", by default in the streamed table's instance. This quote is from the Beam Java SDK documentation, checked 2026-09-04, and describes the same arrangement as the Spanner post.

Retention handling also follows the Spanner pattern. Restoring a position outside the stream's retention fails the job by default. The explicit opt-in scan.resume-fallback.mode discards the stale token and restarts from earliest, latest or a timestamp, accepting the resulting gap.

This source emits no watermarks, so its DDL must not declare SOURCE_WATERMARK(). Each record does carry its commit timestamp as the Flink timestamp, and the partition's estimated low watermark is available as metadata. But Bigtable permits later records to have older commit timestamps, with no published finite bound on how much older. That estimate cannot support a source-watermark guarantee.

Spanner heartbeats, by contrast, guarantee that every change at or before the heartbeat timestamp has been delivered. That lets the Spanner source own its watermark. Beam's Bigtable documentation takes a different approach: it says the connector "outputs all data with an output timestamp of zero, making all data late" (checked 2026-09-04).

A Flink job can instead define its own watermark policy over the commit timestamp:

commit_timestamp TIMESTAMP_LTZ(3) NOT NULL
  METADATA FROM 'commit-timestamp' VIRTUAL,
WATERMARK FOR commit_timestamp AS commit_timestamp - INTERVAL '5' MINUTE

The five-minute delay is an example policy. Records behind that watermark are late according to the job's own choice. Downstream allowed lateness or late-data routing must account for them.

The maximum number of open partition reads is source parallelism multiplied by scan.max-concurrent-streams-per-subtask, whose default is two. This is a connector capacity bound, not a service quota. Monitor the positions of active, queued and unassigned partitions against the table's change-stream retention: once records fall outside retention, the source cannot recover them.

Change streams require a single-cluster-routing application profile. The API is marked @PublicEvolving because its record model follows a client surface the vendor is still evolving. There is no emulator option, since the Bigtable emulator implements neither change-stream RPC.

The knobs that matter

The sink exposes batch thresholds and in-flight bounds. Both count entries, one per record returned by the serializer. Bigtable's separate service limit counts mutations: at most 100,000 per batch. The client enforces that limit by flushing before another entry would exceed it, regardless of the configured entry threshold.

Batch thresholds (sink.batching.* in SQL) are unset by default, retaining the client's defaults of 100 entries, 20 MiB and a one-second timer. Lowering the entry threshold reduces the wait before a low-volume batch is sent, useful for a table-per-day job receiving only a trickle of records.

The writer's in-flight bounds control backpressure: maxInFlightEntries defaults to 1,000 and maxInFlightBytes to 64 MiB. At either cap, write() yields to the task mailbox until requests complete and the counters fall. This lets checkpoint barriers continue to run.

The client has its own flow controller, capped at 20,000 outstanding entries and 100 MiB. Reaching either limit blocks Flink's task thread, which also needs to process checkpoint barriers. Raising the writer's bounds far above the client's limits therefore shifts backpressure into a blocking client call instead of increasing effective capacity.

The connector exposes no retry knobs. The client retries transient MutateRows failures per entry, with its own backoff and a ten-minute total budget. A failure reaches the writer after the client has given up. An UNAVAILABLE at that point represents an outage that outlasted the retry budget, rather than a single slow call.

Only INVALID_ARGUMENT is eligible for row-level failure routing, where a configured handler can drop or dead-letter it. gRPC defines that status as a problem independent of system state. Under CREATE_IF_NEEDED, a NOT_FOUND can be repaired for a missing table or a declared missing family; otherwise it is fatal. All other statuses, including outages, fail the job.

Bigtable can reject an entire batch because of one bad entry. A measurement confirmed this: a good record and a bad record sent together both failed with the same status. The writer therefore resubmits each parked mutation alone to confirm a row-level rejection before invoking the handler.

The maxConsecutiveRejections limit, 100 by default, fails the job after that many confirmed rejections without an applied mutation between them. This prevents a dropping policy from silently draining a stream that the service refuses wholesale. SQL tables expose no failure-policy option and always fail the job on the first routed failure.

Scan parallelism depends on the table's tablets. Each split is a row-key range cut at boundaries Bigtable reports, and the read path cannot subdivide a tablet. A table with few tablets therefore uses few reading subtasks even when the job has higher parallelism.

Each fetch hands Flink at most 1,000 rows and targets at most 8 MiB of decoded input. Server-side filters reduce memory use before data reaches the SDK. SQL family projection is one such filter.

Testing, and what the emulator cannot show

The Bigtable emulator is useful for development, but its validation differs substantially from the real service. The differences can change failure routing. For a timestamp finer than the table's granularity or an empty row key, the service returns INVALID_ARGUMENT. For a missing column family, it returns NOT_FOUND. The emulator returns INTERNAL in these cases.

This sink treats INTERNAL as fatal but can route INVALID_ARGUMENT to a failure handler. An emulator-only test can therefore report a job failure for an error that is droppable on the service.

The emulator also models no tablets, so a scan plan has one split regardless of parallelism. It ignores application profiles and implements neither change-stream RPC. The connector page contains the full deviation tables measured against both environments.

The gated suite creates real instances for its tests. A standing one-node instance costs roughly $470 a month, so each gated class creates an instance and deletes it afterwards. A scheduled sweep reclaims instances left by runs that die before teardown.

These tests found the batch-wide rejection that led to individual confirmation of failed entries. The same-row ordering campaign also ran against the real service. It was deliberately not retained as a regression test: requiring zero reversals would turn an observation into a guarantee the service does not make.

For change-stream recovery, the run on 2026-08-12 observed all 100 seeded rows through a checkpoint, a controlled failure and recovery, without loss. It reached its bounded end time in 132.3 seconds.

What's next for this connector

The v1.1.0 milestone mainly extends Bigtable writes beyond MutateRows. Its first piece merged on 2026-09-03: a request-response runtime for the single-row transactions CheckAndMutateRow and ReadModifyWriteRow. It sits alongside the batching sink because each operation sends one request for one row and returns a response the caller needs.

The runtime follows the client's classification of both RPCs as non-idempotent. It makes one attempt under a 20-second deadline, with no retries on either side. If the request ends before the service answers, the outcome is ambiguous and the job fails: replaying an increment could apply it again.

The runtime and its options are present, but public entry points remain open work under #1174. These include per-operation sinks and functions and a Table write mode. The issue also tracks aggregate column families and an atomic keep-latest mode that deletes a cell's versions and sets its replacement in one mutation.

Response-bearing async SQL functions for Flink 2.2 and later follow. The committer-based exactly-once mode described above is currently planned as the milestone's last item.

The next post covers Cloud Tasks, the sink at the end of this pipeline and the one service for which I found no Flink connector. Reports from workloads unlike mine help decide what to build next: https://github.com/flink-gcp/flink-connector-gcp/issues

The Spanner connector in flink-connector-gcp, part three of the connector series

2026-09-04T00:16:31+09:00

This is the third post in the flink-connector-gcp series, about the Apache Flink connectors for Google Cloud I released as 1.0.0. The release post introduces the project, and the earlier posts cover BigQuery and Pub/Sub.

This post follows Spanner changes into BigQuery, both as a current-row replica and as append-only history. It also covers the move from JDBC to the native client, change-stream state, mutation writes, snapshot reads, lookup joins, and batch limits.

Documentation: https://flink-gcp.github.io/flink-connector-gcp/docs/connectors/datastream/spanner/
Maven: io.github.flink-gcp:flink-connector-gcp-spanner:1.0.0 (1.0.0-1.20 for Flink 1.20); for the SQL client, the flink-sql-connector-gcp-spanner uber-jar from the release page goes into lib/

From a JDBC dialect to the native client

I first tried to support Spanner through Flink's JDBC connector. In February 2025, I submitted a dialect and catalog upstream (apache/flink-connector-jdbc#156, FLINK-37288). I appreciate the review feedback it received, but at the time of writing it remains open and unmerged. I understand that committer time is scarce; I still needed a working pipeline.

The native Java client began as a way around that delay, but it enabled two features that could not have fit Flink's JDBC connector. The first is change streams. Spanner serves them through long-lived partition queries carrying changes, heartbeats and child-partition tokens. A source must manage concurrent queries and their lineage, starting children after their parents finish. That lifecycle fits a FLIP-27 coordinator rather than a bounded JDBC scan.

The second is snapshot partitioning. Flink's JDBC source divides a scan using a configured column and bounds, then runs separate queries without a shared snapshot. Spanner's native batch read API lets the enumerator open one batch read-only transaction and ask the service to partition it. Each server-planned partition becomes a Flink split, and every subtask rejoins the same transaction.

All subtasks therefore read one consistent snapshot. Spanner chooses the partitions from its physical storage layout, removing the need to select a split column and bounds in the job. I value having fewer ways to misconfigure a scan.

The module provides a bounded snapshot source, an unbounded change-stream source and an at-least-once mutation sink through both DataStream and Flink SQL APIs. SQL also supports primary-key lookup joins. The same code supports GoogleSQL and PostgreSQL; in the DataStream API, the database determines the dialect rather than a builder option. These examples use GoogleSQL.

Spanner to BigQuery replication in one SQL job

The pipeline promised in the Pub/Sub post connects operational data in Spanner to analytics in BigQuery. One Flink SQL job consumes a Spanner change stream and writes its upsert changelog through the BigQuery CDC sink. Current row contents flow continuously into BigQuery, with checkpoints governing progress instead of a batch export schedule. No intermediate Kafka pipeline is required.

The Spanner table needs a change stream that captures new row values. I wrote this DDL to match the Flink schema below:

CREATE TABLE Orders (
  OrderId  INT64 NOT NULL,
  Customer STRING(128),
  Status   STRING(32)
) PRIMARY KEY (OrderId);

CREATE CHANGE STREAM order_changes FOR Orders
OPTIONS (value_capture_type = 'NEW_ROW_AND_OLD_VALUES');

The Flink job uses three statements copied verbatim from the source-backed example. CI validates the site's Flink SQL through the planner and executes its GoogleSQL examples against a Spanner emulator. The source table exposes three ordering coordinates as metadata:

SET 'execution.checkpointing.interval' = '1 min';

CREATE TABLE order_changes (
  OrderId BIGINT,
  Customer STRING,
  Status STRING,
  commit_timestamp TIMESTAMP_LTZ(9) METADATA FROM 'commit-timestamp' VIRTUAL,
  record_sequence STRING METADATA FROM 'sequence' VIRTUAL,
  mod_number INT METADATA FROM 'mod-number' VIRTUAL,
  PRIMARY KEY (OrderId) NOT ENFORCED
) WITH (
  'connector' = 'spanner',
  'project' = 'my-project',
  'instance' = 'my-instance',
  'database' = 'orders-db',
  'table' = 'Orders',
  'scan.mode' = 'change-stream',
  'scan.change-stream.name' = 'order_changes',
  'scan.change-stream.changelog-mode' = 'upsert',
  'scan.startup.mode' = 'latest'
);

The BigQuery destination accepts those coordinates as one row of writable metadata:

CREATE TABLE current_orders (
  OrderId BIGINT NOT NULL,
  Customer STRING,
  Status STRING,
  change_sequence ROW<commit_timestamp TIMESTAMP_LTZ(9), record_sequence STRING, mod_number INT>
    METADATA FROM 'spanner-change-sequence',
  PRIMARY KEY (OrderId) NOT ENFORCED
) WITH (
  'connector' = 'bigquery',
  'project' = 'my-project',
  'dataset' = 'analytics',
  'table' = 'current_orders',
  'sink.cdc.enabled' = 'true',
  'sink.create-disposition' = 'create-if-needed',
  'sink.cdc.max-staleness' = '10 min'
);

The INSERT connects them:

INSERT INTO current_orders
SELECT OrderId, Customer, Status,
       ROW(commit_timestamp, record_sequence, mod_number)
FROM order_changes;

The DDL controls how the source represents changes and how BigQuery orders them.

This pipeline requires upsert mode. It emits keyed INSERT and UPDATE_AFTER rows and key-only DELETE rows, matching the BigQuery CDC sink's contract. The alternative, full, reconstructs complete retract rows and emits UPDATE_BEFORE / UPDATE_AFTER pairs. BigQuery's CDC sink rejects update-before rows. Other downstream operators, such as aggregations over a changing table, can use them.

The Spanner capture type must also match. upsert needs complete after-images, supplied by NEW_ROW and NEW_ROW_AND_OLD_VALUES. OLD_AND_NEW_VALUES contains only changed columns and cannot supply the complete row. The source validates each data-change record against the DDL before emitting rows. An incompatible capture type therefore fails deserialization instead of producing partial rows.

Checkpointing persists the change-stream position and flushes the BigQuery default stream. It therefore carries the pipeline's at-least-once guarantee. Without it, there is no saved position, and a restart begins at the configured start position again.

The commit timestamp uses TIMESTAMP_LTZ(9) to retain nanosecond precision, so this table declares no watermark. BigQuery's _CHANGE_SEQUENCE_NUMBER encodes the three source coordinates, beginning with that timestamp. Flink watermark columns support only millisecond precision; truncating here could make two changes within one millisecond equal in the first sequence section.

For jobs that prioritize event-time processing over replication fidelity, the docs instead show TIMESTAMP_LTZ(3) with SOURCE_WATERMARK(). This replication example preserves nanoseconds.

The sequence coordinates let BigQuery handle replays and out-of-order arrival. Where they order two changes, an older replay cannot overwrite the newer value. This makes an at-least-once transport usable for the replica.

They cannot create a total order that Spanner does not expose. Transactions updating disjoint column sets of one row can have equal coordinates. BigQuery resolves those ties by ingestion order.

The source starts at latest, so a fresh job captures changes committed after startup. An already populated table needs an initial snapshot followed by a handoff to a timestamp start; that bootstrap is outside this example. The result is an analytics replica of current row contents, without a byte-for-byte or transactionally consistent copy of the Spanner database.

Install flink-sql-connector-gcp-spanner and flink-sql-connector-gcp-bigquery in the SQL client's classpath. Their dependencies are relocated so both uber-jars can share one lib/.

Keeping every change instead of the latest row

The replica replaces each key's previous state. Other analytics needs that history: what changed and when, how often an order changed status, how long transitions took, or the input for a slowly changing dimension. The change stream already carries those events. To keep them, the job appends them to BigQuery instead of overwriting the previous row.

This history pattern requires a DataStream job in this connector. SQL's scan.mode = 'change-stream' exposes a changelog, but SQL expressions cannot access row kind to turn each change into a plain inserted log record. An append-only destination therefore rejects the updating query during planning.

The DataStream deserializer receives changes before the Table layer interprets them as a changelog. Each typed DataChangeRecord contains the commit timestamp, table, modification type, transaction id and record sequence, plus each mod's keys, new values and old values as normalized JSON. Emitting one row per mod makes these fields ordinary data columns.

Those rows can use BigQuery's append path from the first post: the at-least-once Storage Write API method, or FILE_LOADS for free ingestion at higher volumes. The commit timestamp is a natural time-partitioning column.

Capture semantics determine the contents. With NEW_ROW_AND_OLD_VALUES, inserts and updates include the complete new row, preserving successive states. Old values cover the changed columns, so finding what changed can use a column filter instead of a self-join with the previous row. Deletes have their own modification type and close that key's history.

Delivery remains at-least-once, so the history can contain duplicates. Each output row has an identity: server transaction id, record sequence and mod number, the mod's zero-based position in the record. Queries or a scheduled MERGE can deduplicate on that identity. This path needs neither the CDC sink nor sequence metadata because it does not overwrite earlier records.

Where the change-stream source keeps its state

Spanner divides a change stream into partitions that split and merge over time. Each partition is a streaming query whose child-partition records identify the tokens to read next. The source must track this lineage, retaining every token and waiting for all parents before reading a child. Where that state lives determines much of the connector's operational work.

This connector keeps the partition ledger entirely in Flink checkpoints. The coordinator begins with the null partition token and schedules a child only after every parent naming it has finished. It checkpoints the unfinished topology. Finished parents remain as compact proofs until their children can be scheduled, then are removed.

No metadata table is created, and progress tracking needs no write permission. The workload principal needs database read access, spanner.databases.select and the client's session permissions. A checkpoint restore brings the partition ledger back with the job.

Apache Beam's SpannerIO.readChangeStream, used by Google's Dataflow templates, instead creates a partition metadata table in a designated Spanner database. I checked this against Google's Dataflow connector documentation on 2026-09-03. That table is an additional stateful resource to provision, grant write access to and clean up.

The source uses each record's Spanner commit timestamp as its Flink event timestamp. The coordinator owns one watermark: the minimum across all unfinished partitions, including discovered, queued and running partitions. It broadcasts that value to every reader.

Quiet partitions advance through heartbeats, with a default interval of two seconds and a configurable range from one second to five minutes. They are not marked idle, because removing them from the minimum could make later records from those partitions late.

Heartbeat conversion subtracts one millisecond after truncation. A later nanosecond timestamp can fall in the same Flink millisecond, so the subtraction prevents an on-time record from being classified as late. These are the source-provided timestamps and watermarks. A job that supplies its own watermark strategy chooses that strategy's behavior instead.

The source provides at-least-once delivery. A reader checkpoints each partition's greatest consumed record timestamp and resumes inclusively at that timestamp. Several records can share it; starting after it could skip one. Records at the recovery boundary can therefore repeat.

Deduplicate on server transaction id and record sequence where uniqueness is required. Commit timestamps alone are not unique. In the earlier replication example, sequence metadata handles those boundary replays.

The change stream retains records for the period configured in its DDL. Restoring an expired partition position fails the job by default. The explicit opt-in scan.resume-fallback.mode discards the entire stale ledger and starts fresh, accepting loss of the unavailable interval. I prefer to make that availability-versus-history decision in the job configuration.

Before releasing readers, the coordinator reads stream metadata and logs its watch scope, effective retention and value-capture type. It warns about an explicit watched-column list, since newly added columns will not be included automatically, and rejects unsupported partition modes at startup.

The change-stream API is @PublicEvolving and may change in a minor release, with changes announced in release notes. The rest of the connector is frozen by the japicmp gate described in the release post.

A sink where the mutation names its own table

The sink is configured with a database. Its serializer returns a Spanner Mutation, which names the target table, so one sink can write to every table the serializer chooses. The other four connectors take destination resolvers on their builders. Spanner needs none because the mutation already carries the routing information.

The sink calls batchWriteAtLeastOnce with one mutation per mutation group. Spanner reports status per group, so each rejection identifies one input record. The sink can retry, route or fail that record individually. A plain commit would reject the entire batch for one bad row.

The Spanner client does not retry the batch-write RPC: its generated settings have an empty retryable-code set. The sink therefore owns the retry loop. It retries ABORTED, UNAVAILABLE, DEADLINE_EXCEEDED and RESOURCE_EXHAUSTED within its own budget.

A retry includes groups with transient failures and groups for which the service returned no status, as can happen when the response stream fails partway through. Groups confirmed as applied are not resent. The sink exposes this work through mutationsRetried and transient-error counters, which is less visible when sibling connectors rely on SDK retries.

The configurable failure handler normally receives only ALREADY_EXISTS, such as a replayed insert or unique-index collision, and INVALID_ARGUMENT. Schema violations fail the job by default: a NULL in a NOT NULL column, an over-long value, or a foreign-key or CHECK failure. These often indicate a faulty mapping affecting every record of that shape. Dropping records individually could conceal the mapping bug.

System state can also cause the same status family. If a database's CMEK key is disabled or unreachable, writes fail with FAILED_PRECONDITION. Routing that status to a dropping handler could discard the whole stream during a key incident.

For occasional schema-invalid input, constraintViolationPolicy(ROUTE_TO_FAILURE_HANDLER) makes those statuses eligible for dropping or dead-lettering. The default preserves records for source replay after restart. The opt-in allows progress by routing the rejected records.

Spanner's batch write has no replay protection: the service documents that a mutation may be applied more than once. The serializer's chosen operation determines the effect:

Operation	Same mutation replayed
`insertOrUpdate`, `replace`	Idempotent
`delete`	Idempotent; deleting an absent row is simply applied
`insert`	Refused with `ALREADY_EXISTS`, routed to the failure handler
`update`	Idempotent, unless the row was deleted in between: then `NOT_FOUND` fails the job

NOT_FOUND cannot distinguish a missing row from a missing table. Routing it could silently discard every record when the table name is wrong. For updates that may target absent rows, I use insertOrUpdate.

SQL selects the operation from the DDL. A declared primary key produces an upsert sink using insertOrUpdate, with deletes built from the declared key columns. A keyless table accepts only insert input and uses insert, preserving duplicate-key errors rather than assuming an unknown physical key supports upserts.

Idempotence of an individual mutation does not guarantee the latest value wins. The writer uses separate mutation groups, which Spanner may apply in an unspecified order. Successive writes to one key can therefore finish out of input order. This affects pipelines writing to Spanner; the earlier replication example writes to BigQuery.

Reading a table at one snapshot

The bounded source reads one database snapshot and finishes. It can run inside a streaming pipeline, for example to load a Spanner table for a join with an unbounded stream. Server-planned partitions all rejoin the same batch transaction, so every subtask reads the same consistent snapshot.

Snapshot reads provide at-least-once delivery with a duplicate window of one partition. Partitioned-query row order is not contractual, so the source cannot resume at a position within a partition. Checkpoints record the partitions still held by a reader, and recovery rereads them from the start.

A partition cancelled by Flink during a running job is also reread. The partitionsReread metric can therefore explain duplicates even when the job has not failed. A pipeline requiring unique rows should deduplicate downstream on the primary key.

The snapshot expires after the database's version_retention_period, one hour by default and configurable up to a week. A backfill must finish within that period or use a database whose retention was raised beforehand. A savepoint cannot resume an expired snapshot.

Two options reduce a backfill's impact on serving traffic. rpcPriority(LOW) costs nothing extra: reads stay on the instance's compute but are shed first at capacity. Data Boost, enabled with dataBoostEnabled(true), uses separate billed compute. It needs spanner.databases.useDataBoost, which is absent from roles/spanner.databaseReader, and has its own concurrency quota.

In August 2026, the gated suite exercised Data Boost end to end on a 100-processing-unit STANDARD instance, the cheapest edition. That run confirmed that a Data Boost backfill does not require an edition upgrade from STANDARD.

SQL scans push projection into the requested columns and convert predicates on consecutive primary-key columns to exact key ranges. Flink evaluates the remaining predicates. scan.index selects a secondary index, checked against live metadata at the scan snapshot. Broader filter pushdown is planned for the v1.1.0 milestone.

Lookup joins from plain SQL

Lookup joins are the read feature I use most often. An event carries an account id, and a Spanner table holds the account's current data. A processing-time temporal join enriches the event through a point read, with a cache in front, without exporting the table. The Spanner dimension table has a composite primary key:

CREATE TABLE accounts (
  region STRING(16) NOT NULL,
  account INT64 NOT NULL,
  name STRING(128)
) PRIMARY KEY (region, account);

The Flink job declares that table and joins it to events from Pub/Sub. This combines two connectors from the series. The example adapts the source-backed lookup example, validated in CI; I added the Pub/Sub event stream:

CREATE TABLE order_events (
  account   BIGINT,
  region    STRING,
  amount    INT,
  proc_time AS PROCTIME()
) WITH (
  'connector'    = 'pubsub',
  'project'      = 'my-project',
  'subscription' = 'account-events-sub',
  'format'       = 'json'
);

CREATE TABLE accounts (
  region  STRING,
  account BIGINT,
  name    STRING,
  PRIMARY KEY (region, account) NOT ENFORCED
) WITH (
  'connector' = 'spanner',
  'project'   = 'my-project',
  'instance'  = 'my-instance',
  'database'  = 'orders-db',
  'table'     = 'accounts',
  'lookup.async' = 'true',
  'lookup.cache' = 'PARTIAL',
  'lookup.partial-cache.expire-after-write' = '10 min'
);

SELECT e.region, e.account, e.amount, a.name
FROM order_events AS e
LEFT JOIN accounts FOR SYSTEM_TIME AS OF e.proc_time AS a
  ON e.account = a.account AND e.region = a.region;

The join equality must include every declared primary-key column because a point read needs the complete key. Composite keys are encoded in PRIMARY KEY declaration order, regardless of the join-predicate order. That declaration must match the physical Spanner primary key or the lookup will address a different key.

The WITH options control lookup behavior. lookup.async selects asynchronous rather than synchronous point reads. The lookup cache uses Flink's standard NONE and PARTIAL modes, with expiry, size and missing-key settings. I generally use PARTIAL with a write expiry such as the example's ten minutes. Frequently read keys stay cached, at the cost of up to that much staleness.

The source rejects FULL because a scan-backed cache would require snapshot and refresh semantics it does not provide. The next connector in the series, Bigtable, offers that mode.

lookup.max-retries retries ABORTED, DEADLINE_EXCEEDED and UNAVAILABLE. Of these, the client library retries only UNAVAILABLE, so the connector option supplies further attempts for the other two. It excludes RESOURCE_EXHAUSTED: the client already retries that status while respecting the server-requested delay, which an immediate connector retry would undermine.

Batch limits that are correctness bounds first

A request-level rejection affects every mutation in the request, so the sink must bound each batch. It flushes when any of three limits is reached. The defaults, inherited from Apache Beam, are 5,000 cells, 500 mutations and 1 MiB, all well below service limits.

maxBatchCells accounts for secondary indexes. Spanner counts one cell for the table column plus one for each index containing it. The sink reads INFORMATION_SCHEMA when the writer opens and uses that schema to estimate each mutation's cost. This requires spanner.databases.select in addition to write access.

The 5,000-cell default is sixteen times below Spanner's published 80,000 per-group figure. That headroom covers undercounting for tables created after startup or hidden from the writer's role, whose index entries cannot be included. Raising the limit consumes that margin.

maxBatchBytes protects the request-level size limit. Google's documentation can be read as specifying either 10 MiB or 100 MiB, so the gated suite tested oversized requests. The service accepted roughly 12 MiB and rejected roughly 110 MiB, naming a limit of 104,857,600 bytes, exactly 100 MiB.

The connector estimates size because the client cannot report a mutation's wire size, and the estimate generally reads low. BYTES is counted at its base64 length: Spanner wraps it in a protobuf Value, which has no bytes kind. The measurement showed 83,886,080 raw bytes becoming 111,852,884 on the wire, roughly four thirds of the input.

Raising one limit may have no effect if another still binds first. For example, maxBatchMutations cannot be reached above maxBatchCells, because every mutation costs at least one cell. Building those options logs a warning naming both values where the job's main runs. Compare bufferedCells and bufferedBytes when tuning to see which limit triggers the flush.

batchWriteTimeout bounds a complete attempt. Its default is 30 seconds, replacing the client's one-hour timeout for this RPC only. Including retries, the default worst case is 369.375 seconds: ten 30-second attempts and up to 69.375 seconds of jittered backoff.

A checkpoint may encounter that loop twice. A record-triggered flush can already be running when the barrier arrives, followed by another invocation for the checkpoint flush. Choose the checkpoint timeout to cover that work, alignment and the rest of the job.

Change-stream capacity is source parallelism multiplied by scan.max-concurrent-queries-per-subtask, default 8. This bounds the connector's open partition queries; it is not a Spanner quota.

If activeChangeStreamQueries stays at that product while queued or unassigned partition lag grows, increase parallelism or the per-subtask bound. Alert when either lag approaches stream retention, since expired records cannot be recovered. The per-subtask bound changes reader capacity. Changing parallelism requires restarting from a checkpoint or savepoint so Flink can redistribute ownership.

Testing, and what the emulator cannot show

The weekly gated real-GCP suite establishes service behavior. Spanner's emulator is useful but has specific differences. It supports the sink's BatchWrite RPC only from v1.5.31; older versions return UNIMPLEMENTED. The repository therefore pins its image separately from the other connectors' emulator bundle.

The emulator also serializes concurrent read-write transactions. In measured reads, it planned exactly two partitions for every table, ignored both partition hints, and applied a stricter partitionability check with a less useful error message than the service. The connector page records the full deviation table measured on 2026-08-10.

The sink's rejection-status table was measured against both environments, including NOT NULL violations, missing tables and replayed inserts. The gated suite asserts each result so changes on either side are detected.

Each test class creates a regional 100-processing-unit STANDARD instance and deletes it afterwards. Names include creation time, allowing the next run and a scheduled sweep to reclaim instances left by interrupted tests. Instances are not retained between runs because they bill for their entire lifetime.

The measured change-stream recovery run delivered all 5,000 unique mutations after an intentional failure following a checkpoint. It also delivered 500 repeats at the inclusive recovery boundary, matching the documented duplicate window. A subsequent savepoint restore consumed a mutation written while the job was stopped.

What's next for this connector

The v1.1.0 milestone extends bounded-scan filter pushdown. Further work is driven by reports, especially from workloads unlike mine: https://github.com/flink-gcp/flink-connector-gcp/issues.

The next post covers Bigtable: lookup joins with a full-table cache, change streams, and the write extensions planned for v1.1.0.

The Pub/Sub connector in flink-connector-gcp, part two of the connector series

2026-09-03T00:13:09+09:00

This is the second post in the flink-connector-gcp series, about the Apache Flink connectors for Google Cloud I released as 1.0.0. The release post introduces the project, and the previous post covers BigQuery.

This post follows a Pub/Sub order pipeline in Flink SQL and a DataStream sink that routes records to different topics. It also explains the source's streaming-pull design, seek-on-start behavior, and tuning for checkpoint-based acknowledgment.

Documentation: https://flink-gcp.github.io/flink-connector-gcp/docs/connectors/datastream/pubsub/
Maven: io.github.flink-gcp:flink-connector-gcp-pubsub:1.0.0 (1.0.0-1.20 for Flink 1.20); for the SQL client, the flink-sql-connector-gcp-pubsub uber-jar from the release page goes into lib/

Two Pub/Sub connectors already existed when I started. Apache's flink-connector-gcp-pubsub uses the legacy SourceFunction API over unary Pull. At the time of writing, its last main-branch commit was in November 2024. That commit added a Table API sink (FLINK-36077), but a Table API source is still missing.

Google's connector in its pubsub repository is well built on the StreamingPull API and moved to Flink 2.2 in June 2026. It remains DataStream-only and, at the time of writing, is not published to a Maven repository.

I needed complete Flink SQL support, with both source and sink tables, and message attributes queryable as columns. I also wanted a start-position table option, a policy that sends poison messages through Pub/Sub's dead-lettering, and a sink that chooses a topic for each record at runtime.

Why the source consumes through streaming pull

Apache's connector chose unary pull during its review (FLINK-9311). An earlier implementation used the high-level Subscriber, but the final design called unary Pull on the blocking gRPC stub. The review gave three reasons: backpressure blocking sourceContext.collect() naturally stops the pull loop; users avoid tuning flow control; and removing the intermediate queue reduces memory use and latency. I think that choice suited its source framework.

Two of those reasons depend on SourceFunction. Its hand-written run() loop needs queues and lock coordination to bridge an asynchronous client. In FLIP-27, Flink provides that bridge: SplitReader.fetch() is a pull loop, while SourceReaderBase owns the element queue and backpressure between the fetcher and task thread.

Flow-control tuning remains a concern. This connector exposes the subscriber settings rather than hiding them, and the tuning section explains how to size them.

Lease extension and ordering led me to the high-level client. Unary Pull returns acknowledgment ids without managing their leases. Apache's connector does not extend deadlines, so messages must be acknowledged within the subscription deadline, at most 600 seconds. Its documentation accordingly requires a much shorter checkpoint interval. The high-level client extends leases automatically, up to an hour by default, which better suits checkpoint-based acknowledgment.

The high-level client also provides sequential dispatch per ordering key. Using unary pull would require implementing that behavior separately. Streaming pull does cost more gRPC CPU. The FLINK-9311 review measured the synchronous design at 3,000 messages per second with one-second checkpoints and 20,000 with 50-millisecond checkpoints, showing its dependence on checkpoint frequency.

Each split represents one streaming-pull connection to a subscription. It stores no progress position because Pub/Sub has no offset to resume from. In default unordered mode, the plan creates max(|subscriptions|, parallelism) splits so every subscription is consumed and every subtask can work. Per-key ordering uses exactly one split per subscription.

The plan is deterministic from the subscription list, ordering mode and parallelism, and is recomputed on each start. Restoring a savepoint with a different parallelism therefore reassigns the splits. A MiniCluster test verifies rescaling in both directions.

Pub/Sub holds delivery state on the server. The source acknowledges a message when the checkpoint covering its emission completes. A failure before that leaves the message unacknowledged for redelivery, giving at-least-once delivery. Checkpoints contain no message data, so source recovery does not require retaining one.

Checkpointing is still mandatory. Without it, messages remain unacknowledged until subscriber flow control fills and consumption stalls. If messages are waiting and no checkpoint arrives within ten minutes, the reader fails the job with an error naming execution.checkpointing.interval.

The reader checks behavior because it cannot reliably inspect the configured interval. env.enableCheckpointing(...) updates the job configuration, while the reader receives the TaskManager configuration. An absent interval there does not prove checkpointing is disabled.

The order pipeline in Flink SQL

The example consumes JSON order events from a topic, uses the publish timestamp as event time, calculates hourly totals per customer, and publishes them to another topic. Pub/Sub-specific behavior is expressed through columns and options. This example assumes the subscription exists; a later section covers resource creation. The source table is:

CREATE TABLE incoming_orders (
  order_id     STRING,
  customer_id  STRING,
  amount       INT,
  message_id   STRING              METADATA FROM 'message-id'   VIRTUAL,
  publish_time TIMESTAMP_LTZ(3)    METADATA FROM 'publish-time' VIRTUAL,
  attrs        MAP<STRING, STRING> METADATA FROM 'attributes'   VIRTUAL,
  ordering_key STRING              METADATA FROM 'ordering-key' VIRTUAL,
  subscription STRING              METADATA FROM 'subscription' VIRTUAL,
  WATERMARK FOR publish_time AS publish_time - INTERVAL '5' SECOND
) WITH (
  'connector'    = 'pubsub',
  'project'      = 'my-project',
  'subscription' = 'orders-sub',
  'format'       = 'json'
);

A Pub/Sub message contains a payload, attributes and an ordering key. format decodes the payload into physical columns. The remaining fields are exposed as metadata.

publish-time records when the service received the message, truncated to milliseconds and never rounded up. It provides an event-time column for the watermark without requiring a timestamp in the payload.

attributes is always a map, empty when the message has none. Queries can filter with WHERE attrs['channel'] = 'mobile' or group by an attribute. This avoids writing a custom deserializer, one of my original requirements.

ordering-key is NULL when no key is set. Pub/Sub represents that absence as an empty string; the connector maps it to SQL null.

subscription contains the full resource name, such as projects/my-project/subscriptions/orders-sub. The configuration option accepts a bare id, but the metadata uses the API's resource-name form, suitable for joins with audit logs or Cloud Asset Inventory. Consequently, WHERE subscription = 'orders-sub' matches nothing, a distinction emphasized in the documentation.

The sink maps a column back to message attributes:

CREATE TABLE customer_totals (
  customer_id STRING,
  total       INT,
  attrs       MAP<STRING, STRING> METADATA FROM 'attributes'
) WITH (
  'connector' = 'pubsub',
  'project'   = 'my-project',
  'topic'     = 'customer-totals',
  'format'    = 'json'
);

INSERT INTO customer_totals
SELECT customer_id,
       CAST(SUM(amount) AS INT),
       MAP['window-start', CAST(window_start AS STRING)]
FROM TABLE(TUMBLE(TABLE incoming_orders, DESCRIPTOR(publish_time), INTERVAL '1' HOUR))
GROUP BY window_start, window_end, customer_id;

Both tables adapt the source-backed examples, which CI validates through Flink's planner. The combination is mine.

The window TVF makes the totals insert-only, which this sink requires. Pub/Sub cannot express retractions. An updating query such as a plain GROUP BY customer_id therefore fails during planning, preventing update and delete rows from being published as ordinary messages.

The source instead adopts the format's changelog mode. A changelog format can therefore travel over Pub/Sub, although the at-least-once transport may redeliver a retraction.

This pipeline uses event time without requesting ordered delivery. The watermark permits five seconds of lateness in this example, a bound to size for the topic. Arrival order within that bound does not change the hourly totals. Transport-level ordering addresses a narrower requirement and is covered below.

A null key or value in the attributes map fails the write. Pub/Sub supports neither, and silently removing an entry would lose data without the query knowing. Filter such entries in the query before writing.

Install the flink-sql-connector-gcp-pubsub uber-jar in Flink's lib/ or load it with ADD JAR. Its bundled dependencies are relocated so it can coexist with the BigQuery and Bigtable uber-jars.

A fan-out sink that routes on record content

Dynamic destinations were my main reason for building this sink. A resolver chooses a topic from each record's values or attributes, allowing one input stream to fan out to topics selected at runtime. The release post introduced this feature across all five connectors. On Pub/Sub, the sink is:

DataStream<OrderEvent> orders = ...;

orders.sinkTo(
        PubSubSink.<OrderEvent>builder()
                .destinationResolver(
                        (event, context) ->
                                TopicDestination.of("my-project", "orders-" + event.region()))
                .serializer(
                        PubSubSerializationSchema.payload(new OrderEventSchema())
                                .withAttributes(e -> Map.of("tenant", e.tenantId())))
                .topicCreateOptions(
                        TopicCreateOptions.builder()
                                .messageRetention(Duration.ofDays(7))
                                .build())
                .build());

This composite adapts the source-backed examples, which CI compiles. The combination is mine.

The resolver runs once per record, before serialization. Any retained field can determine the route, including message attributes preserved by a passthrough deserializer. TopicDestination holds only project and topic identity, with equality over both. It is cheap to allocate and keys the writer's publisher map.

The writer manages publishers as destinations appear and disappear. It creates an SDK publisher lazily for each active topic and retains at most 100 (maxActivePublishers). At capacity, a new topic evicts the least-recently-used publisher with no pending messages. If none is clean, the writer drains all publishers first. It never discards an in-flight message.

After a successful checkpoint flush, publishers idle for over an hour (destinationIdleTimeout) are released. A later record recreates the publisher. Releases use bounded two-phase shutdowns with overlapping waits, so several releases share one timeout budget. A shutdown that times out during eviction fails the running task, preventing abandoned gRPC channels from accumulating. At final close, teardown continues even if shutdown times out.

The writer flushes all records at each checkpoint barrier instead of storing them in Flink state. A savepoint-less redeploy can duplicate messages but cannot lose messages covered by a completed checkpoint. The next section explains how a newly routed topic is created, allowing a new region in this example to receive its stream without redeployment.

SQL sinks write to the single topic named in their DDL. A SQL job can fan out to topics known at planning time using multiple INSERT statements in a STATEMENT SET. A topic computed from record contents requires the DataStream API shown above. A Kafka-style topic metadata column was considered and declined because INSERT branching covers the SQL cases the planner can express.

Creating what is missing, on both sides

The sink creates missing topics under the default CREATE_IF_NEEDED. Creation begins only after a publish returns NOT_FOUND: the sink parks the messages, creates the topic, and republishes with bounded backoff. Existing topics incur no admin call.

A topic can use service defaults, so the disposition alone authorizes creation. TopicCreateOptions optionally adds retention, a CMEK key and storage regions.

Source creation is authorized by supplying settings for each subscription; there is no disposition option. A subscription needs a topic binding that only the pipeline author can choose. Those settings must be per subscription. Sharing one topic binding across several subscriptions would make the job consume a full copy of the topic's stream from each.

In SQL, the mapping looks like 'scan.auto-create.topics.orders-sub' = 'orders'. Its keys must match the subscription list exactly. The source creates subscriptions only: every mapped topic must already exist.

Creation is idempotent on both sides. ALREADY_EXISTS counts as success, so concurrent jobs need no coordination. Existing resources are left as they are; creation settings are neither applied nor compared.

A new subscription normally has no backlog from before its creation. To replay that history, enable topic-level retention, exposed as messageRetention in the sink's creation settings. The start-position options below then determine where to begin within the retained history.

What a start position does to a subscription

scan.startup.mode, or startPosition(...) in the DataStream API, selects the starting position. Every non-default value performs a seek:

Mode	Behavior
`continue-from-subscription` (default)	Starts wherever the subscription already is; the only mode that issues no seek
`earliest-retained`	Replays the whole retained backlog
`latest`	Discards the existing backlog
`timestamp` (with `scan.startup.timestamp-millis`)	Everything published before the instant is marked acknowledged, everything after unacknowledged

These options replace a manual gcloud pubsub subscriptions seek before submitting a backfill. A seek changes server-side subscription state shared by all consumers; there is no per-consumer offset. A job using a non-default start position should therefore own its subscription.

The enumerator performs the seek on the first start and records it in checkpointed state. Restoring that state resumes consumption without seeking again.

A redeploy without a savepoint seeks again because the record of the previous seek is gone. A job that repeatedly crashes before its first completed checkpoint also repeats the seek. latest resolves against the clock each time and is therefore not reproducible. Use timestamp for an exact boundary.

Retention determines what a backward seek can recover. Already-acknowledged messages are available only if the subscription retains acknowledged messages or the topic retains messages. With neither enabled, only unacknowledged messages can be recovered. The startup check warns about that combination.

Before assigning splits, the enumerator checks every subscription. It rejects per-key ordering on a subscription without message ordering, which would otherwise emit unordered messages. It also rejects subscriptions with exactly-once delivery enabled: their acknowledgment ids expire with the deadline and are invalidated on redelivery, while this source retains ids for a whole checkpoint interval.

All subscriptions are checked before any seek runs, so a rejected configuration cannot leave another subscription already rewound. These checks capture startup settings only. They do not detect settings changed while the job is running.

Per-key ordering, and where it stands

The connector supports per-key ordering end to end, but I expect most pipelines, including mine, to leave it disabled. Google's ordering documentation describes the costs: lower publish availability, higher end-to-end latency, a 1 MB/s publish limit per ordering key, and at most one outstanding batch per key for pull subscriptions. Checkpoint-based acknowledgment adds another constraint here: roughly one batch per key per checkpoint interval.

In my experience, many apparent ordering requirements are event-time requirements. The example's watermark handles those within its declared lateness bound without requiring ordered transport. Pub/Sub serves message fan-out rather than a log, and transport ordering is a separate choice.

For jobs that need ordering, both APIs support it on the source and sink. A source table with scan.ordering-mode = 'per-key' assigns each subscription to one subtask over one streaming-pull connection. It guarantees per-key order at the source output; downstream exchanges must partition by the key to preserve it.

On a sink table, sink.message-ordering.enabled and an ordering-key metadata column automatically route keyed rows to one writer subtask. The pair is validated during planning. The ordering documentation details the costs and conditions.

The Pub/Sub emulator cannot validate ordered consumption. The client's per-key callback serialization depends on a subscription property the emulator does not set. Callbacks can therefore arrive out of order even without Flink involved. The connector verifies ordered consumption in its weekly real-service suite.

Enabling publish ordering also changes retry behavior. With enableMessageOrdering, the SDK replaces its retry settings with unlimited retries, including for unkeyed messages. During an outage, even a configured retry timeout no longer ends the wait.

The connector rejects retryTotalTimeout and retryMaxAttempts when ordering is enabled, since the SDK would ignore them. It also adds publishProgressTimeout, a watchdog that fails the job when no publish completes within that interval. In one August 2026 measurement against an unreachable endpoint, an unordered publish failed after 591 seconds, while an ordered flush was still waiting at 700 seconds. That gap informed the watchdog's default.

Failure routing on both sides

On the source, scan.deserialization-failure-policy handles messages the format cannot decode. The default, fail, leaves the message unacknowledged and fails the job. A permanently bad message can then fail each restart. drop acknowledges and discards it, with a counter and rate-limited logging. nack returns it for redelivery so the subscription's dead-letter policy can handle it.

I wanted nack so a poison message could reach a dead-letter topic for later replay while the pipeline kept running. It uses Pub/Sub's service-side machinery without adding dead-letter code to the job. Startup rejects nack on a subscription with no dead-letter policy, since an undecodable message would otherwise keep being redelivered without failing the job.

Dead-lettering counts deliveries regardless of cause, so unrelated job restarts contribute to the same counter. The Pub/Sub service account also needs publish and subscribe grants. Without them, the service continues redelivering instead of forwarding.

For more complex handling, a DataStream deserializer receives the entire message and writes to a Collector. It can emit a bad-record variant instead of throwing, then use a side output to route it within the checkpointed pipeline.

The sink routes two failure types to its configurable handler: serializer rejections and messages individually rejected by the service with INVALID_ARGUMENT. Other failures, including outages, remain fatal. Routing an outage to a dropping handler would lose messages instead of applying backpressure and restarting.

Individual confirmation matters because Publish is an all-or-nothing batch RPC. The SDK reports the batch status for every message, and real-service measurements found no indication of which message caused it. The sink republishes the failed batch one message per request. Only messages rejected individually reach the handler; valid neighbors are published.

After 100 confirmed rejections in a row without a success, maxConsecutiveRejections fails the job even under a dropping policy. Dropping occasional anomalous records should not allow a wholly rejected stream to drain silently.

The handler can use the same PubSubDeadLetterQueue as the BigQuery post. The shared failure contract lets this implementation serve every connector. Each published element carries attributes identifying the connector, destination, error, timestamp and subtask. It does not create its topic, since an automatically created dead-letter destination may have no consumer.

Tuning around checkpoint-shaped acknowledgement

Google's documentation gives no recommended flow-control values beyond sizing them to the client machines. The SDK defaults protect memory; they are not throughput targets. The connector leaves these settings at their SDK defaults and documents a sizing rule for checkpoint-based acknowledgment. All messages received since the last completed checkpoint count against the outstanding-message limit:

flowControlMaxOutstandingElementCount ≳ peak messages/s × checkpoint interval

Below that bound, the client stops pulling before the checkpoint completes, making throughput depend on checkpoint frequency. The SDK default of 1,000 outstanding messages saturates at a hundred messages per second with a 10-second checkpoint interval. Higher limits increase both reader memory use and the number of messages replayed after failure.

maxAckExtensionPeriod, one hour by default, must also comfortably exceed the checkpoint interval. Otherwise leases expire before the covering checkpoint can acknowledge the messages, causing redelivery.

The connector adds a hard buffer budget of 10,000 messages or 64 MiB of serialized data per source reader, whichever is exceeded first. It covers cases where SDK flow control is insufficient, such as a split paused by watermark alignment or a downstream operator that stops consuming.

Measurements showed why the extra bound is needed. When a buffered message exhausts its lease-extension budget, the client stops extending it and releases its flow-control permit. Pub/Sub redelivers the message, but the connector still holds the original. Released permits allow another wave of intake, mostly duplicates, growing the buffer by roughly one flow-control window per wave. The emulator stopped after two waves; the real service continued.

The connector bounds the buffer and stops a paused split's subscriber entirely if it outgrows that bound. On resume, it reopens the subscriber. Returning the leases lets Pub/Sub redeliver to a consumer that can make progress.

The sink caps unacknowledged publishes at 1,000 messages and 64 MiB per writer subtask. Count alone is insufficient: Pub/Sub accepts messages up to 10 MiB, so 1,000 could occupy about 10 GiB. Retries retain them for up to the SDK's default 600-second timeout, or indefinitely with ordering enabled. A partial outage is therefore when retained data can peak.

The 64 MiB default is below the Java subscriber's 100 MB SDK default, which applies per client rather than per subtask. Size maxInFlightBytes so its total across sink subtasks on a TaskManager fits the heap budget. With default limits, bytes become the binding cap only above roughly 64 KiB per message.

The connector uses its own bound because the SDK flow controller blocks the task thread instead of yielding to Flink's mailbox. With ordering enabled, the SDK also leaks permits on its per-key cancellation path, eventually exhausting the budget and hanging the thread without an exception. The design record documents that behavior.

As with BigQuery, the connector documentation records measurements beside the design decisions. The configuration reference lists all options. Those experiments provide a starting point for adjusting defaults to another workload.

What's next

Near-term milestones focus on other connectors, including Bigtable write extensions in v1.1.0. Pub/Sub's roadmap is driven by workload reports on the issue tracker.

The next post covers Spanner, whose change-stream source feeds the BigQuery CDC sink from the first post. Together they provide Spanner-to-BigQuery replication in one Flink SQL job.

The BigQuery connector in flink-connector-gcp, part one of the connector series

2026-09-02T00:11:29+09:00

This is the first of five posts about flink-connector-gcp, the Apache Flink connectors for Google Cloud I released as 1.0.0. The release post explains the motivation, supported Flink versions, and testing.

This post covers BigQuery's three write methods, a multi-tenant ingestion pipeline, the bounded source, and defaults informed by measurements against the real service.

Documentation: https://flink-gcp.github.io/flink-connector-gcp/docs/connectors/datastream/bigquery/
Maven: io.github.flink-gcp:flink-connector-gcp-bigquery:1.0.0 (1.0.0-1.20 for Flink 1.20)

When I surveyed the Dataproc-maintained BigQuery connector in July 2026, it was actively developed but lacked two features I needed. Each sink wrote to one fixed table, and schema changes required a job redeployment. Dynamic destinations and schema evolution became central to this connector.

One builder, three write methods

The sink uses one builder, following the style of Beam's BigQueryIO. A call to writeMethod(...) on BigQuerySink.builder() selects one of three implementations when the job graph is built. Destination, serializer, table creation, schema updates, and failure-handler settings stay on that builder. Switching from streaming inserts to batch loads therefore changes configuration without rewriting the pipeline.

Sink<MyEvent> sink =
        BigQuerySink.<MyEvent>builder()
                .writeMethod(WriteMethod.STORAGE_API_AT_LEAST_ONCE)
                .destinationResolver(
                        (e, ctx) -> TableDestination.of("my-project", "my_dataset", e.tableName()))
                .serializer(new MyEventProtoSerializer())
                .build();

The three methods trade visibility latency against delivery guarantee and ingestion cost:

	Visibility	Delivery	Ingestion price
`STORAGE_API_AT_LEAST_ONCE`	seconds (as appends succeed)	at least once	volume-based Storage Write API pricing
`STORAGE_API_EXACTLY_ONCE`	per checkpoint	exactly once	volume-based Storage Write API pricing
`FILE_LOADS`	per checkpoint	exactly once	load jobs are free on the shared slot pool

STORAGE_API_AT_LEAST_ONCE writes through the Storage Write API's default stream. At every checkpoint, the writer flushes pending batches and waits for all in-flight appends before passing the barrier. A completed checkpoint therefore means BigQuery has acknowledged every preceding record. No sink buffer remains in Flink state.

The project evaluated FLIP-171's AsyncSinkBase for this sink and the other connectors. Its AsyncSinkWriter stores unflushed buffers in writer state rather than flushing them at the barrier. Redeploying without that state loses those buffered records. I chose to flush at the barrier so a savepoint-less redeploy risks duplicates instead of loss. The Storage Write API SDK already provides the in-flight window needed for backpressure, so the sink did not need that machinery from the base class.

STORAGE_API_EXACTLY_ONCE uses application-created buffered streams and a two-phase protocol. Writers append rows at explicit offsets, where they remain invisible. When a checkpoint completes, the committer calls FlushRows up to that checkpoint's offset.

Each writer reuses one buffered stream per active destination across checkpoints. Google's guidance discourages calling CreateWriteStream at checkpoint frequency, and the API has a quota of 10,000 calls per hour per project and region.

FlushRows is naturally idempotent: flushing an already-flushed offset returns ALREADY_EXISTS. A restarted committer can therefore retry without deterministic-ID machinery and can run at the sink's full parallelism.

The two modes have different failure risks. At-least-once keeps the sink ahead of source acknowledgment, so discarding operator state can duplicate rows but cannot lose them. Exactly-once puts visibility and source acknowledgment in the same commit phase without an atomic operation spanning both. Discarding state during a redeploy can then lose up to one checkpoint's data.

This limitation is inherent to two-phase commit; a Kafka exactly-once producer has the same risk. The sink cannot detect discarded state because a writer restored without state looks like a new job. Redeploy exactly-once jobs through savepoints. With the Flink Kubernetes Operator, use upgradeMode: savepoint or last-state, never stateless.

The later FILE_LOADS section covers its ingestion costs and checkpoint-related quotas.

A multi-tenant pipeline, end to end

The pipeline that motivated these features receives events from many tenants and writes to one BigQuery table per tenant. Tenants can appear or disappear without a job redeployment. That requires per-record routing, table creation on first write, schema evolution, and a destination for unroutable records. The sink combines all four:

Map<String, TableDestination> tablesByTenant = new HashMap<>();

BigQuerySink.<OrderEvent>builder()
        .destinationResolver(
                (event, context) -> {
                    if (!event.hasKnownTenant()) {
                        return UnroutableRecord.of(event.deadLetterPayload(), "Unknown tenant");
                    }
                    return tablesByTenant.computeIfAbsent(
                            event.tenantId(),
                            id -> TableDestination.of("my-project", "my_dataset", "orders_" + id));
                })
        .serializer(new OrderEventProtoSerializer())
        .tableCreateOptions(
                TableCreateOptions.builder()
                        .timePartitioning(TimePartitioningType.DAY, "created_at")
                        .timePartitioningExpiration(Duration.ofDays(90))
                        .clusteredFields(List.of("customer_id"))
                        .build())
        .schemaUpdateOptions(SchemaUpdateOptions.builder().allowNewFields().build())
        .failureHandler(
                FailureHandler.sendToDeadLetterQueue(
                        PubSubDeadLetterQueue.builder()
                                .topic(TopicDestination.of("my-project", "dead-letters"))
                                .build()))
        .build();

This composite adapts the source-backed examples, which CI compiles. The combination is mine.

Routing. The resolver runs once per record, before serialization, on the writer's hot path. It must be serializable, deterministic and cheap. Caching TableDestination values, as above, is the documented pattern.

Each active destination has its own writer state, including a stream writer for the Storage Write API methods. Destinations are evicted after an idle timeout, one hour by default, so tenant churn does not leave connections accumulating indefinitely.

For a record it cannot place, the resolver returns an UnroutableRecord containing a payload and reason. That result reaches the failure handler. A bare null is always fatal, preventing a drop policy from concealing a routing bug.

Table creation. With the default CREATE_IF_NEEDED, the first record for a missing table creates it. The serializer supplies the schema; tableCreateOptions(...) supplies partitioning and clustering. HTTP 409 is treated as success when parallel subtasks race to create the table.

Heavy contention can also hit BigQuery's per-table metadata-update quota and return 403 rateLimitExceeded. In August 2026, the project raced sixteen creations against one absent table and saw five rate-limited responses. The connector retries these within its recovery backoff budget, allowing concurrent creation to back off instead of failing the job immediately.

A missing table returns an unexpected status. Opening a Storage Write API stream against it gives PERMISSION_DENIED, because BigQuery masks table existence from callers who might be probing names. The permission named in the message varies by path, so the connector checks the status code, never the text.

The goccy/bigquery-emulator returns NOT_FOUND instead. Emulator tests therefore do not exercise this recovery path. In this project, auto-creation had never worked against the real service until direct measurements exposed the difference in August 2026. The documentation records dates and trial counts. That experience is why I use a weekly real-GCP suite to establish service behavior.

Schema evolution. The sink handles changes in either direction without a restart. If someone changes the destination schema through DDL, the default stream's append response reports the new schema. The writer rebuilds its connection with a fresh descriptor.

If the serializer's schema extends the table's schema, schemaUpdateOptions(...) can allow the sink to update the table. It reads the live schema, combines it with the serializer's schema, and submits an etag-conditioned update. The union only widens: new fields are appended as NULLABLE; existing fields are not removed, reordered or retyped. Concurrent unions converge, so parallel subtasks need no additional coordination.

In six of seven instrumented real-service runs, an update reached the write backend in about 35 seconds. The writer keeps retrying affected batches during propagation. One run took much longer. In a checkpointed job, the checkpoint timeout bounds that wait by triggering a restart.

Schema updates are opt-in because an unexpected field in the serializer's schema can alter a live table. Enabling allowNewFields() means trusting that schema. The connector's additive schema union does not remove fields, so unwanted additions require a separate cleanup. BigQuery supports explicit column deletion with ALTER TABLE DROP COLUMN. By default, a schema-mismatch append fails the job for investigation.

The dead letter path. PubSubDeadLetterQueue publishes failed elements to a Pub/Sub topic. Attributes identify the connector, intended destination or unresolved for routing failures, error, timestamp and subtask.

All connectors share the failure-handler contract, so one dead-letter implementation can serve a job's BigQuery, Bigtable and Pub/Sub sinks. Delivery is at-least-once. The implementation deliberately does not create the topic: a newly created dead-letter destination may have no consumer.

The same pipeline on free ingestion

Both Storage Write API methods use volume-based ingestion pricing. With FILE_LOADS, writers stage each destination's rows in Cloud Storage, using zstandard-compressed Avro by default. At every checkpoint, a committer submits BigQuery load jobs. These jobs are free on the shared slot pool, although Cloud Storage staging still costs money and shared-pool capacity is not guaranteed. Paid PIPELINE slots are available when capacity must be reserved.

Rows become visible after the checkpoint's load completes, typically minutes rather than seconds. A high-volume pipeline that accepts that latency can switch with two builder calls and a suitable checkpoint interval:

env.enableCheckpointing(300_000); // 5 minutes; see the quota arithmetic below

BigQuerySink.<OrderEvent>builder()
        .writeMethod(WriteMethod.FILE_LOADS)
        .destinationResolver(...)   // unchanged
        .serializer(serializer)     // unchanged
        .fileLoadsOptions(
                FileLoadsOptions.builder()
                        .stagingPath("gs://my-staging-bucket/flink-loads")
                        .build())
        .build();

FILE_LOADS provides exactly-once delivery through precise file references and deterministic job IDs. Each load names the exact file URIs emitted by the writers, never a bucket prefix that could include files from failed attempts. The job ID hashes the destination and file list. After a crash, a retry reattaches to the existing load job instead of submitting a second load.

Committables remain in Flink's committer state until their loads succeed. The referenced files are therefore part of the recoverable data. Use a dedicated bucket for stagingPath, with a lifecycle age longer than the longest outage the job must recover from. If the rule deletes files still referenced by a checkpoint, restoring that checkpoint leaves the loads permanently failing.

FileLoadsOptions selects the staging format. Avro is the default; Parquet is opt-in and requires parquet-avro plus a Hadoop runtime for any compression. The connector does not ship these dependencies. It checks for them when building the job graph and names any missing artifact in the client-side error.

BigQuery rejects Parquet load jobs whose schema contains a JSON column, regardless of the file contents. For such destinations, the connector overrides the format to Avro and logs that choice once per destination.

Both formats use the same derived Avro schema, so both reject INTERVAL, RANGE and BigQuery flexible column names. The tuning section includes the measurement used to evaluate Parquet's performance.

The checkpoint interval determines load-job quota use. A standard table allows 1,500 modifications per day from load, copy and query jobs combined. Each checkpoint that commits files uses at least one modification per active destination:

Checkpoint interval	Modifications per destination per day
1 min	1,440, too close to the ceiling to be viable
2 min	720
5 min	288

Multiple destinations also consume the project quota. At a 3-minute interval, one destination needs 480 load jobs per day. Two hundred active destinations use about 96,000 of the project's 100,000 daily load jobs, leaving little room for retries or other workloads.

The connector rejects intervals below 2 minutes when building the graph and warns below 5 minutes. Lowering that guard requires an explicit opt-in for short-lived jobs. Pipelines needing visibility within seconds should use the Storage Write API methods; FILE_LOADS accepts minutes of latency for free ingestion.

Reading a table back out

The Storage Read API source is a bounded FLIP-27 source. It can run inside a streaming job, for example to load a dimension table for a broadcast join, and finishes when it has read the table. There is no unbounded or CDC source, and none is planned: BigQuery exposes no changelog primitive on which to build one.

A split stores one session read stream and the number of rows already consumed. Recovery resumes the read at that offset using the API's own mechanism. Readers request another stream as soon as they finish one, so requesting more streams than subtasks can help balance the work.

BigQuery decides the actual stream count. Measurements in August 2026 produced 936 streams for a 910 GB table and always 1 for a small table, regardless of how many were requested.

BigQuery charges for bytes scanned by a read session and stores columns separately. selectedFields, the equivalent of Table API projection pushdown, avoids scanning unused columns. rowRestriction filters rows before transfer. Reading a view first executes it as a query, incurring both query-scan and result-read charges, so pruning needs to happen inside that query.

The source uses Avro on the wire. In the comparison with Arrow, constructing individual records from Arrow was 34% slower than decoding Avro directly, and the Arrow representation was 84% larger on the wire. Arrow's advantage depends on avoiding individual row materialization, while Flink requires the source to produce records. The connector therefore retained Avro.

Change data capture

With sink.cdc.enabled, the Table API sink accepts an upsert changelog and writes UPSERT and DELETE mutations through the Storage Write API's CDC support. The _CHANGE_SEQUENCE_NUMBER pseudocolumn controls ordering. It contains one to four slash-separated hexadecimal sections, compared as unsigned numbers by BigQuery.

Built-in sequence profiles derive these values from Debezium PostgreSQL and MySQL metadata, TiCDC commit timestamps, and Spanner commit-timestamp coordinates. The API remains experimental while an upstream question about exposing Debezium source metadata to sequence providers is resolved.

The worked example appears in the Spanner post: its change-stream source and this CDC sink form a replication pipeline in one Flink SQL job.

Defaults with measurements behind them

The configuration reference lists every option. Where a default needed empirical evidence, the project measured it and recorded the result beside the setting. Examples include:

Staging file roll size (16 MiB). In the August 2026 measurement, loading 769 MiB took 15.0 s with 2 MiB files, 8.3 s with 8 MiB files, and 16.9 s with 128 MiB files. Load times were lowest near 8 MiB; making files smaller did not keep improving them. The default is 16 MiB because file size also determines how much one destination can load within the 10,000-URI job limit: about 156 GiB at 16 MiB, compared with 78 GiB at 8 MiB.
Zstandard staging compression. Compression runs on the task thread and directly affects throughput. Across 2,000,000 rows, deflate took 11,436 ms and zstandard 3,182 ms. Zstandard's output was only 1.8% larger, so the choice mainly reduced CPU cost.
Parquet staging as opt-in. Parquet used 0.785x the bytes of Avro across a 64x range of file sizes. Below 256 MiB of input per load job, however, it loaded several times slower, with a sharp improvement at that threshold. A streaming checkpoint's load is usually smaller. Parquet is therefore suited to batch workloads where each destination's per-commit volume clearly exceeds 256 MiB.
maxInflightRequests at 100 where the SDK defaults to 1000. The default stream multiplexes appends over a shared pool that grows when connections appear busy. At the SDK default, a connection needs more than 200 queued requests to count as busy, so the pool rarely grows and throughput plateaus. The connector follows Google's guidance to lower the threshold and documents how to restore the SDK default.
Append batch size at 512 KiB. The default bounds memory and per-record latency. Throughput-oriented jobs can increase it toward the API's 10 MB request cap, at the cost of more buffered bytes per destination and larger retry units.

The newest option, added days before release, limits staging files finalized concurrently by one writer at a checkpoint: maxConcurrentCheckpointFinalizations, default 1 and maximum 8. In its sizing measurements, increasing sink parallelism reduced upload time by 77-85%, while writer-local close concurrency of 8 reduced close time by roughly 79-84%. Parallelism remains the first adjustment to try; local concurrency helps when more slots are unavailable.

These numbers describe measured workloads, not fixed service behavior. The documentation records the experiments so they can be repeated when a default does not suit another workload.

What's next for this connector

The v1.1.0 milestone extends the Table API source's filter pushdown. The CDC API will also continue to evolve while the Debezium-format question is resolved.

The next post covers Pub/Sub: metadata columns in both directions, seek-on-start behavior, and ordered delivery. Reports from workloads unlike mine help guide the project: https://github.com/flink-gcp/flink-connector-gcp/issues

Releasing flink-connector-gcp 1.0.0, Apache Flink connectors for BigQuery, Pub/Sub, Cloud Tasks, Bigtable, and Spanner

2026-08-31T19:06:04+09:00

I have released flink-connector-gcp 1.0.0: five independent Apache Flink connectors for Google Cloud, covering BigQuery, Cloud Pub/Sub, Cloud Tasks, Bigtable, and Spanner. Each offers DataStream and Flink SQL / Table APIs. The artifacts are available on Maven Central under io.github.flink-gcp.

Documentation: https://flink-gcp.github.io/flink-connector-gcp/
Source: https://github.com/flink-gcp/flink-connector-gcp
Release: https://github.com/flink-gcp/flink-connector-gcp/releases/tag/v1.0.0

This post explains why I built the project and what it provides. The following posts cover one connector each, with feature details, pipeline examples, and tuning.

Why I built it

The existing connectors lacked features my pipelines needed, and adding those features upstream was either slow or not an available route. These were the options for each service when I started.

Pub/Sub. Apache's flink-connector-gcp-pubsub has seen sparse activity. At the time of writing, its last main-branch commit was in November 2024. That commit added a Table API sink, but a Table API source is still missing. Google's connector in GoogleCloudPlatform/pubsub is well built on the StreamingPull API and moved to Flink 2.2 in June 2026. It remains DataStream-only and, at the time of writing, is not published to a Maven repository. Neither provides the Table API source or the full set of source and sink features I needed.

BigQuery. The Dataproc-maintained connector is actively developed, but my July 2026 survey found two gaps. Each sink writes to one fixed table, so it cannot route records to destinations chosen at runtime. It also cannot follow schema changes without redeploying the job.

Bigtable. Google's flink-connector-gcp, a separate project with the same name, provides an exactly-once Bigtable sink with Table API support. I also needed lookup joins for enrichment and a change-stream source.

Spanner. I submitted a JDBC dialect and catalog to Flink's JDBC connector (apache/flink-connector-jdbc#156, FLINK-37288). It received review feedback, which I appreciate, but has remained open and unmerged since February 2025. Committer time is scarce in a large project. I understand the delay, but needed a working pipeline, and a JDBC dialect would not have covered change streams anyway.

Cloud Tasks. I did not find an existing Flink connector at all.

Each project has its own priorities and review queue. I needed features I could not get from those projects, so I built connectors for all five services around a shared design.

What is in the box

Five connectors, each with two API surfaces:

Connector	DataStream	Flink SQL / Table API
BigQuery	source + sink	scan source (bounded), sink
Cloud Pub/Sub	source + sink	source, sink
Cloud Tasks	sink	sink
Bigtable	source + sink	scan source, lookup, sink
Spanner	source + sink	scan source, lookup, sink

Each release has two version lines: 1.0.0 for the supported Flink 2.x range and 1.0.0-1.20 for Flink 1.20 LTS. The next section explains that policy. Each connector also has a flink-sql-connector-gcp-* uber-jar for SQL, available from the release page and installed in Flink's lib/.

<dependency>
  <groupId>io.github.flink-gcp</groupId>
  <artifactId>flink-connector-gcp-bigquery</artifactId>
  <version>1.0.0</version> <!-- 1.0.0-1.20 for a Flink 1.20 job -->
</dependency>

Which Flink versions are supported, and how

The support policy follows Flink's current and previous minor releases, currently 2.2 and 2.3, plus the 1.20 long-term support line. Supporting both major versions requires two artifact lines.

One artifact covers both supported 2.x minors. The connectors compile against the oldest, 2.2, and run on the newer version. A weekly job verifies this compatibility: it builds the jars once against 2.2, then reruns the test suite against those same jars on the newer minor. If that check finds a break, the documented fallback is separate artifacts per minor, as the Kafka connector publishes. So far, it has found none.

Flink 1.20 uses the same source tree. Before choosing whether to maintain another branch, I measured the API differences affecting these connectors and found two. Each has a same-named file in small per-major source roots, selected by a Maven property at build time. All other source is shared.

The -1.20 artifacts compile that code for Flink 1.x. Binary compatibility does not extend across majors: a 2.x jar is not expected to load on a 1.20 cluster, or vice versa. The LTS line therefore has a separate compilation and version suffix.

Expanding the supported range requires an explicit change. A scheduled check reports new Flink minors and lists the required edits. Support is claimed only after the compatibility job passes. The weekly matrix also builds against the next unreleased Flink snapshot to catch upstream breaks before release.

The features I built it for

Everything below is documented with runnable examples on the documentation site.

Dynamic destinations. Every DataStream sink can derive its destination from each record: a BigQuery table, Pub/Sub topic, Cloud Tasks queue, Bigtable table, or Spanner table. A multi-tenant pipeline can route to per-tenant tables through one sink and accept new tenants without restarting.

Auto-creation. The BigQuery and Pub/Sub sinks can create a table or topic on first write. A new tenant's first record can therefore create its destination and land in it within the same job. Bigtable also supports opt-in auto-creation when a table schema is declared.

Schema evolution. The BigQuery sink can follow changes to the destination table's schema without restarting the job. Adding a column no longer requires a redeployment.

BigQuery batch ingestion from a streaming job. Alongside its at-least-once and exactly-once Storage Write API paths, the sink offers FILE_LOADS. The streaming job stages Avro files, or optionally Parquet, in Cloud Storage and commits them through load jobs at checkpoints. Load jobs have no streaming-insert cost, making this economical for high-volume pipelines that can tolerate checkpoint-aligned latency. The job retains the same sink API and dynamic destinations.

CDC into BigQuery. The BigQuery Table API sink accepts changelog streams and uses the Storage Write API's CDC support to write upserts and deletes with explicit ordering. Flink SQL can therefore materialize a changing table in BigQuery.

Change streams out of Spanner (and Bigtable). Both connectors provide change-stream sources. Combining the Spanner source with the BigQuery CDC sink makes Spanner-to-BigQuery replication a single Flink SQL job.

Pub/Sub attributes, ordering, and seek in SQL. Message attributes and ordering keys are metadata columns on both the source and sink. A source table reads attributes as MAP<STRING, STRING> and can use the publish timestamp as event time. A sink writes attributes and ordering keys from columns.

The start position is also a table option. Declaring earliest-retained, latest, or a timestamp makes the connector seek the subscription on startup, replacing a manual gcloud pubsub subscriptions seek before a backfill. Neither existing Pub/Sub connector offers a Table API source.

Lookup joins against Bigtable and Spanner. Both provide Table API lookup sources with caching options. A Flink SQL enrichment join can use FOR SYSTEM_TIME AS OF to look up a Bigtable row or query a Spanner table.

How it is tested

The test coverage matters as much as the feature list, especially where emulators behave differently from the real services.

Every pull request runs integration tests for the affected connectors against service emulators. The weekly matrix builds against every supported Flink version. A japicmp gate compares each build's public API with the published 1.0.0 release.

Documentation checks compare option and metric tables with the source. CI also compiles the Java examples, validates Flink SQL examples through the planner, and executes GoogleSQL examples against a Spanner emulator.

A weekly E2E suite also runs against real Google Cloud services. It covers behavior emulators cannot establish: BigQuery load jobs and default-stream fan-out, serializer fidelity against real column types, ordered Pub/Sub subscription delivery and dead-letter forwarding, Cloud Tasks dispatch to an App Engine handler, and Bigtable and Spanner rejection statuses.

A separate gated manual probe checks BigQuery schema evolution on the real service. It stays outside the weekly suite because schema propagation can outlast a CI runner.

Tests create and delete real instances where needed. The harness also checks that the gated test classes actually ran. Missing credentials must not produce a green result from a suite that skipped its tests.

How it was built

Claude and Codex helped write much of this repository under the support programs acknowledged below. I make the architecture decisions and give every change its final review. Two practices make that process inspectable.

First, the repository records its decisions in close to 150 architecture decision records. Changes to a settled design are expected to read and update the relevant records. This preserves the reasoning for the next contributor, whether human or model.

Second, every pull request follows a fixed review sequence. The first self-review asks whether the code does what the description says. The second asks whether the description itself is true. A different model then performs an independent review: Codex reviews Claude's work, and Claude reviews Codex's.

The independent reviewer receives the diff without the pull-request description or author notes. Its findings are hypotheses to verify. Using another model mattered more than I expected: one measured case in the decision records showed repeated same-model reviews stopping at a shallow fix, while the other model identified the underlying race condition. The pull-request review threads are public.

What's next

As of August 2026, the tracker groups planned features into these milestones:

v1.1.0 is mostly about extending Bigtable writes: check-and-mutate, read-modify-write, aggregate column families, and response-bearing writes exposed up to Flink SQL, plus wider filter pushdown for BigQuery and Spanner and release-pipeline automation.
v1.2.0 plans a sixth connector, for Firestore / Datastore.
v1.3.0 plans observability work: exporting the Google client libraries' own metrics through Flink's metric reporters, so the connector-internal RPC behavior shows up on the same dashboards as the job.

Issues and feedback are welcome on the tracker. Reports from use cases unlike mine are especially useful because they expose gaps in what I can test.

The following posts cover one connector each, with feature details, an end-to-end pipeline, and tuning guidance.

Acknowledgements

This project receives support from Anthropic through Claude and Claude for OSS, its program for open-source developers, and from OpenAI through Codex and Codex for Open Source, its program for open-source maintainers.

Building five connectors with this review process takes substantial model time. These programs made it viable as a personal project outside an employer. I am grateful to both.

A SQL injection in PyAthena's parameter formatter (CVE-2026-65321)

2026-08-03T22:54:26+09:00

PyAthena 3.35.4 fixes a SQL injection in the default parameter formatter. The bug affected DELETE and CREATE TABLE ... AS SELECT (CTAS) statements built with PyAthena's parameter substitution. An application that passed an untrusted string as a parameter to one of those statements could let that string break out of its quotes and run as SQL. All released versions up to and including 3.35.3 are affected, and the fix is in 3.35.4.

How PyAthena quotes parameters

PyAthena implements the DB-API pyformat parameter style. When an application calls cursor.execute(sql, params), PyAthena substitutes each parameter into the SQL text after quoting it, rather than sending the value to Athena as a bound parameter. Quoting a string here means wrapping it in single quotes and escaping any single quote inside it, so that the value stays one string literal. The formatter is the code that does this quoting.

Two escapers, chosen by statement prefix

The formatter carried two different escapers. One doubles a single quote, turning ' into ''; this is the ANSI SQL rule that Trino and Athena follow. The other backslash-escapes it, turning ' into \'; this is the Hive and MySQL rule. Both conventions were present because Athena runs two kinds of statement, SQL through its Trino engine and a smaller set of DDL through a Hive-derived engine, and the two engines read string literals by different rules.

The formatter picked the escaper from the start of the statement. SELECT, WITH, INSERT, UPDATE, and MERGE went to the quote-doubling escaper. Everything else went to the backslash escaper.

Why DELETE and CTAS were injectable

DELETE was missing from that first list, and a CTAS statement starts with CREATE, so both fell through to the backslash escaper. That would be harmless if Athena read \' as an escaped quote. It does not. Trino and Athena treat a backslash inside a single-quoted literal as an ordinary character, so \' is a literal backslash followed by a quote that ends the string.

The effect is easiest to see in the rendered SQL. Given the parameter value a' OR 1=1 --, the formatter produced this for a DELETE:

DELETE FROM t WHERE c = 'a\' OR 1=1 --'

Athena reads the \ as a backslash and the following ' as the end of the string literal. The OR 1=1 then runs as part of the WHERE clause, and the -- comments out the trailing quote. A predicate meant to match one row now matches every row. The same break-out works in a CTAS SELECT, where an injected UNION SELECT can copy data the caller never meant to expose into the new table.

Why it went unnoticed

The escaper selection had no test that passed a quote-containing string through DELETE or CTAS. The tests that did cover those paths used values without quotes, so the wrong escaper produced output that looked correct. An earlier fix had already moved UPDATE and MERGE onto the safe escaper, so the mismatch was understood in the abstract, but DELETE and CTAS were left behind and no test caught them.

Defaulting to the safe escaper

3.35.4 inverts the choice. The formatter now uses the quote-doubling escaper by default, and routes a statement to the backslash escaper only when it recognizes that statement as Hive DDL: CREATE TABLE or CREATE EXTERNAL TABLE that is not a CTAS, ALTER, DROP of a database, schema, or table, MSCK REPAIR, SHOW, and DESCRIBE. Leading comments are stripped before this check, so a comment cannot hide the statement type.

The default matters because the two mistakes are not symmetric. Doubling a quote in a Hive DDL statement is at worst a formatting quirk, since the Hive lexer joins adjacent string literals. Backslash-escaping a quote in a Trino statement is a break-out. When the statement type is unclear, the formatter now falls back to the escaper that cannot be exploited.

Coordinated disclosure

The issue was reported through VulnCheck, acting as CNA, on behalf of the researcher Rahul Karne. The proof of concept runs PyAthena's real formatter and executes the generated SQL against a local DuckDB database, which reproduces the break-out without touching AWS. It is tracked as CVE-2026-65321 and GitHub advisory GHSA-xwj5-g6cv-4r5c, scored CVSS 3.1 9.8 (Critical), and CVSS 4.0 9.3 in the CVE record. That score is the worst case for a library of this kind, since it assumes untrusted input reaches the vulnerable parameter directly; the real exposure of a given application depends on whether it passes untrusted values into DELETE or CTAS parameters.

What to do

Upgrade to PyAthena 3.35.4:

pip install -U 'pyathena>=3.35.4'

Until the upgrade lands, avoid passing untrusted values as parameters to DELETE or CTAS statements. Building those statements from trusted input only, or routing the untrusted value through a SELECT- or INSERT-shaped query, avoids the vulnerable escaper.

What I took from it

Two habits would have caught this earlier. The first is to test the security-relevant branch with a hostile value rather than a benign one; a single quote in a DELETE parameter would have failed loudly. The second is to make the default the safe one, so that a gap in the statement-type detection fails closed instead of open. The longer-term direction is to move parameter values off client-side escaping and onto Athena's server-side parameters, which removes this class of bug at the source.

References

Migrating a Subsonic server to Navidrome and s3fs

2023-09-20T16:07:20+09:00

This is an English translation of my 2023 setup log about replacing a Subsonic server on CentOS 7 with Navidrome on Debian 12. I first tried Mountpoint for Amazon S3, then switched to s3fs for the music library. The package versions and commands below reflect that deployment.

EC2 and Navidrome

Attach an instance role that can access the S3 bucket, using a policy like the one below. Open ports 80 and 443 in the security group, and allow SSH access from your own IP address.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::your-s3-bucket*"
            ],
            "Effect": "Allow"
        }
    ]
}

Create a user for Navidrome and run Mountpoint for S3 as that user. I gave the user a login shell to test the mount, although a login shell is probably unnecessary otherwise.

$ sudo useradd -m navidrome -s /bin/bash -d /var/lib/navidrome

Install Navidrome according to its documentation. I also installed Emacs as my editor.

https://www.navidrome.org/docs/installation/linux/

$ sudo apt update
$ sudo apt upgrade
$ sudo apt install emacs ffmpeg

$ sudo install -d -o navidrome -g navidrome /opt/navidrome
$ sudo install -d -o navidrome -g navidrome /var/lib/navidrome

$ wget https://github.com/navidrome/navidrome/releases/download/v0.49.3/navidrome_0.49.3_Linux_arm64.tar.gz
$ sudo tar -xvzf navidrome_0.49.3_Linux_arm64.tar.gz -C /opt/navidrome/
$ sudo chown -R navidrome:navidrome /opt/navidrome

Create directories for the music and data, along with the configuration file. I initially mounted the S3 bucket at the music directory with Mountpoint for S3.

$ sudo mkdir /var/lib/navidrome/{music,data}
$ sudo touch /var/lib/navidrome/navidrome.toml
$ sudo chown -R navidrome:navidrome /var/lib/navidrome

Set the required options in navidrome.toml, using the configuration reference.

https://www.navidrome.org/docs/usage/configuration-options/

MusicFolder = "/var/lib/navidrome/music"
DataFolder = "/var/lib/navidrome/data"
LogLevel = "info"
Address = "0.0.0.0"
ScanSchedule = "0 5 * * *"

Create a systemd unit for Navidrome.

[Unit]
Description=Navidrome Music Server and Streamer compatible with Subsonic/Airsonic
After=remote-fs.target network.target
AssertPathExists=/var/lib/navidrome

[Install]
WantedBy=multi-user.target

[Service]
User=navidrome
Group=navidrome
Type=simple
ExecStart=/opt/navidrome/navidrome --configfile "/var/lib/navidrome/navidrome.toml"
WorkingDirectory=/var/lib/navidrome
TimeoutStopSec=20
KillMode=process
Restart=on-failure

# See https://www.freedesktop.org/software/systemd/man/systemd.exec.html
DevicePolicy=closed
NoNewPrivileges=yes
PrivateTmp=yes
PrivateUsers=yes
ProtectControlGroups=yes
ProtectKernelModules=yes
ProtectKernelTunables=yes
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
RestrictNamespaces=yes
RestrictRealtime=yes
SystemCallFilter=~@clock @debug @module @mount @obsolete @reboot @setuid @swap
ReadWritePaths=/var/lib/navidrome

# You can uncomment the following line if you're not using the jukebox This
# will prevent navidrome from accessing any real (physical) devices
#PrivateDevices=yes

# You can change the following line to `strict` instead of `full` if you don't
# want navidrome to be able to write anything on your filesystem outside of
# /var/lib/navidrome.
ProtectSystem=full

# You can uncomment the following line if you don't have any media in /home/*.
# This will prevent navidrome from ever reading/writing anything there.
#ProtectHome=true

# You can customize some Navidrome config options by setting environment variables here. Ex:
#Environment=ND_BASEURL="/navidrome"

Save it as /etc/systemd/system/navidrome.service, then start and enable the service.

$ sudo systemctl daemon-reload
$ sudo systemctl start navidrome.service
$ sudo systemctl status navidrome.service
$ sudo systemctl enable navidrome.service

Forward port 4533 over SSH and create the Navidrome admin user.

Trying Mountpoint for S3

Mountpoint for S3 was straightforward to install using its documentation.

https://github.com/awslabs/mountpoint-s3/blob/main/doc/INSTALL.md

$ wget https://s3.amazonaws.com/mountpoint-s3-release/1.0.1/arm64/mount-s3-1.0.1-arm64.deb
$ sudo apt-get install ./mount-s3-1.0.1-arm64.deb

Create a temporary directory and try mounting the bucket.

$ mkdir mount
$ mount-s3 your-s3-bucket ./mount
$ ls ./mount

The S3 files appeared with ls. After confirming the mount worked, I unmounted it.

$ umount ./mount

To mount the bucket at startup, I added a systemd unit based on this issue.

[Unit]
Description=Mountpoint for Amazon S3 mount
Wants=network.target
AssertPathIsDirectory=/var/lib/navidrome/music

[Service]
Type=forking
User=navidrome
Group=navidrome
ExecStart=/usr/bin/mount-s3 your-s3-bucket /var/lib/navidrome/music
ExecStop=/usr/bin/fusermount -u /var/lib/navidrome/music

[Install]
WantedBy=default.target

Save it as /etc/systemd/system/mountpoint-s3.service, then start and enable it.

$ sudo systemctl daemon-reload
$ sudo systemctl start mountpoint-s3.service
$ sudo systemctl status mountpoint-s3.service
$ sudo systemctl enable mountpoint-s3.service

The mounted directory was visible to the Navidrome user, but not to other users, including root in this test.

$ ls -al /var/lib/navidrome
drwxr-xr-x  3 navidrome navidrome 4096 Sep 17 15:38 data
d?????????  ? ?         ?            ?            ? music
-rw-r--r--  1 navidrome navidrome  159 Sep 17 14:44 navidrome.toml

$ sudo ls -al /var/lib/navidrome
drwxr-xr-x  3 navidrome navidrome 4096 Sep 17 15:38 data
d?????????  ? ?         ?            ?            ? music
-rw-r--r--  1 navidrome navidrome  159 Sep 17 14:44 navidrome.toml

$ sudo -u navidrome ls -al /var/lib/navidrome
drwxr-xr-x  3 navidrome navidrome 4096 Sep 17 15:38 data
drwxr-xr-x  2 navidrome navidrome    0 Sep 17 16:26 music
-rw-r--r--  1 navidrome navidrome  159 Sep 17 14:44 navidrome.toml

I uploaded a few music files to S3, and Navidrome scanned and played them successfully. Then I uploaded the whole library from the old Subsonic server's EBS volume: 1.1 TB and 90,098 files. With that larger library, Navidrome did not read every file from the Mountpoint for S3 directory.

$ sudo du -sh /var/lib/navidrome/music/
1.1T    /var/lib/navidrome/music/
$ sudo find /var/lib/navidrome/music/ -type f -not -name cover.jpg | wc -l
90098

There were some warnings, but I could not determine the cause or collect logs useful enough for a GitHub issue. I switched to s3fs.

Switching to s3fs

https://github.com/s3fs-fuse/s3fs-fuse

$ sudo apt install s3fs
$ echo "s3fs#your-s3-bucket /var/lib/navidrome/music fuse uid=****,gid=****,iam_role=auto,endpoint=your-region,allow_other,mp_umask=022,use_cache=/tmp,nonempty 0 0" | sudo tee -a /etc/fstab
$ sudo mount -a

Navidrome scanned every file in the directory mounted with s3fs.

Nginx and HTTPS

To make Navidrome accessible from outside the instance, I configured Nginx and Let's Encrypt. I installed Nginx according to its documentation.

http://nginx.org/en/linux_packages.html#Debian

$ sudo apt install curl gnupg2 ca-certificates lsb-release debian-archive-keyring
$ curl https://nginx.org/keys/nginx_signing.key | gpg --dearmor \
  | sudo tee /usr/share/keyrings/nginx-archive-keyring.gpg >/dev/null
$ gpg --dry-run --quiet --no-keyring --import --import-options import-show /usr/share/keyrings/nginx-archive-keyring.gpg
$ echo "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] \
  http://nginx.org/packages/debian `lsb_release -cs` nginx" \
  | sudo tee /etc/apt/sources.list.d/nginx.list
$ echo -e "Package: *\nPin: origin nginx.org\nPin: release o=nginx\nPin-Priority: 900\n" \
  | sudo tee /etc/apt/preferences.d/99nginx
$ sudo apt update
$ sudo apt install nginx

Install Certbot and request a certificate. Because I manage DNS with Route 53, I used the certbot-dns-route53 plugin for the DNS challenge.

$ sudo apt install certbot python3-certbot-dns-route53
$ certbot certonly \
 --dns-route53 \
 -d foo.bar \
 -d *.foo.bar \
 -m foo@bar \
 --agree-tos

The EC2 instance also needs permissions to update the relevant Route 53 hosted zone.

{
    "Version": "2012-10-17",
    "Id": "certbot-dns-route53",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "route53:ListHostedZones",
                "route53:GetChange"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect" : "Allow",
            "Action" : [
                "route53:ChangeResourceRecordSets"
            ],
            "Resource" : [
                "arn:aws:route53:::hostedzone/YOURHOSTEDZONEID"
            ]
        }
    ]
}

The certificate needs periodic renewal, so I enabled Certbot's systemd timer. With the Route 53 plugin, renewal did not require a separate script.

$ sudo systemctl enable certbot.timer

I modified /lib/systemd/system/certbot.service as shown below to reload Nginx after certificate renewal.

[Unit]
Description=Certbot
Documentation=file:///usr/share/doc/python-certbot-doc/html/index.html
Documentation=https://certbot.eff.org/docs
[Service]
Type=oneshot
ExecStart=/usr/bin/certbot -q renew --no-random-sleep-on-renew --post-hook "/usr/sbin/nginx -t && /usr/bin/systemctl reload nginx"
PrivateTmp=true

Configure the certificate in Nginx and proxy requests to Navidrome. This was my proxy configuration.

upstream navidrome {
    server 127.0.0.1:4533;
}

server {
    listen       80;
    server_name  foo.bar;
    rewrite ^ https://$server_name$request_uri? permanent;
}

server {
    listen       443 ssl http2;
    server_name  foo.bar;

    access_log  /var/log/nginx/foo.bar_access.log  ltsv;

    ssl_certificate /etc/letsencrypt/live/foo.bar/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/foo.bar/privkey.pem;

    location / {
        proxy_set_header       Host               $host;
        proxy_set_header       X-Real-IP          $remote_addr;
        proxy_set_header       X-Forwarded-Proto  https;
        proxy_set_header       X-Forwarded-Host   $host;
        proxy_set_header       X-Forwarded-Server $host;
        proxy_set_header       X-Forwarded-For    $proxy_add_x_forwarded_for;

        proxy_pass             http://navidrome;
        proxy_redirect         http:// https://;

        proxy_connect_timeout  30;
        proxy_send_timeout     60;
        proxy_read_timeout     60;
    }
}

$ sudo systemctl start nginx.service
$ sudo systemctl status nginx.service
$ sudo systemctl enable nginx.service

A firewall was not installed by default on this instance. Install and configure one if needed.

$ sudo apt install ufw
$ sudo ufw allow 22/tcp
$ sudo ufw allow 80/tcp
$ sudo ufw allow 443/tcp
$ sudo ufw allow 4533/tcp
$ sudo systemctl start ufw
$ sudo systemctl enable ufw
$ sudo ufw enable

Finally, I allocated an Elastic IP, associated it with the instance, and configured the Route 53 record.

I could connect from an iPhone using these Subsonic clients:

The old Subsonic server used 1.5 TB of sc1 EBS storage. Moving the library to S3 removed that fixed volume size from the server. I hoped Mountpoint for S3 would become a workable option for this library later.

Now I can keep buying music on Bandcamp.