publish_timeout on ArrakisSink (default 30 s): a block whose
delivery is not confirmed in time counts as a publish failure
instead of stalling for the producer's own message timeout (5
minutes by default); 0 waits indefinitely
ArrakisSink no longer keeps a failed pipeline alive: its publish
thread is a daemon, so a failure elsewhere in the pipeline (which
never delivers EOS to the sink) lets the process exit instead of
waiting on the thread until the supervisor kills it
At EOS the sink drains its backlog only while publishing works; once
a block fails the rest is dropped with a logged count, so a dead
broker cannot hold the pipeline open for the length of the backlog
ArrakisSource.fast_forward (default True) passes through to the
arrakis client's live-stream skip-ahead, so a source stuck behind
its latency contract recovers with a bounded gap; False opts out
ArrakisSink catches arrakis.PublishError in the publish thread:
the failed block is logged at ERROR and dropped so healthy
partitions keep flowing, and the pipeline aborts only after
max_publish_failures consecutive failures (default 10, 0 disables)
Require arrakis >= 0.19 for PublishError and the live-stream
skip-ahead
The publish thread drains the queue at EOS before exiting; it
previously exited as soon as EOS was marked, dropping however many
queued blocks thread scheduling left behind
ArrakisSource and ArrakisSink declare thread_safe = True,
opting in to threaded pad dispatch under Pipeline.run(threaded=...)
summary_interval on ArrakisSource and ArrakisSink (default
900 s): periodic INFO summary of blocks received or queued and how
far the newest block lags real time (plus the publish backlog for
the sink), doubling as a liveness heartbeat; 0 disables
backlog_warn on ArrakisSink (default 10 blocks): warn once when
the publish queue grows past the threshold, i.e. publishing is not
keeping up, and log recovery once it drains below half of it
ArrakisSource and ArrakisSink now log at INFO: the publisher
registered at startup, the first block received or queued, and the
periodic summary, with per-block detail at DEBUG (the stream request
itself is logged by the arrakis client when it opens). Previously
neither element logged anything
ArrakisSink logs the exception when its publish thread fails,
before surfacing it to the pipeline
The sgn-arrakis CLI configures the sgn logger (the parent of
every element's logger) rather than the nonexistent sgn-ts, and
defaults LOG_LEVEL to INFO instead of DEBUG
Declare gpstime as a direct dependency, used for the summaries'
lag behind real time
ArrakisSource is resilient to server interruptions: stream errors
reconnect with capped exponential backoff, live streams bridge outages
with gaps, and bounded streams resume from the last delivered timestamp
Update minimum arrakis version to 0.14.1: the reconnect handling
relies on the client raising stream errors on the next read (older
clients masked outages as gap-filled blocks until stream close) and on
gRPC keepalives surfacing dead connections as errors instead of
indefinite hangs
Update minimum sgn-ts version to 0.13, which introduces the
TSResourceSource resource hook API the source now streams through