05 Workflows
A subscription streaming platform
Sign-in, the home screen, a checkout and a play request from the web, phone and TV apps, each through its own backend for frontend, with Kafka carrying the events, change data, heartbeats, activity and logs behind them.
The problem
A subscription service lets only paying households stream, on a limited number of screens, from devices as different as a browser and a TV remote. Payments confirm late and out of order, content keys must stop with the session, and every login, heartbeat and log line has to reach the systems that act on it without slowing the stream.
Step by step
Enter at the edge
A browser, a phone app or a TV app sends every request over HTTPS to the cloud load balancer, which spreads it across the ingress gateway's Envoy pods in every zone; TLS ends at the gateway. Envoy's rate-limit service counts each caller's requests in Redis, shared by every gateway pod, and answers 429 with Retry-After past the limit. The gateway then routes by host, so the web, app and TV hosts each reach their own backend for frontend (BFF).
What can go wrong: A lost zone takes its gateway pods with it; health checks steer traffic to the other zones, and every service runs as a Helm release with replicas across zones.
Sign in through every BFF
Each BFF is a confidential OIDC client of the identity service, so no token lives in a browser or an app. The web and phone apps send the user to Identity's hosted sign-in page, password then MFA, and an authorisation code with PKCE comes back to the BFF. The TV, with only a remote, asks Identity for a device code and shows a short user code and a QR link; the user approves it on a phone, and the TV BFF polls until Identity answers.
What can go wrong: Credential stuffing; the gateway limits attempts per IP, account and device in Redis, and Identity alerts the user to every new device.
Record the login
Identity checks the password hash and MFA factors in its Postgres, sends the MFA code or a new-device alert through the email and SMS gateways, and in one transaction registers the device and writes login.succeeded to its outbox. It returns an access JWT, which carries the profile's maturity level, and a rotating refresh token. The BFF keeps both in Redis under a session ID and gives the device only that session: an HttpOnly cookie on the web, a keychain token on the phone, a device token on the TV.
Build the home screen
Each BFF asks Recommendations for the profile's rows, sized for its screen. Recommendations reads the precomputed candidate rows from Redis, keeps the titles the plan and region allow, as Entitlements reports them, and the profile's maturity level permits, and fills in each title from the Catalog. The BFF shapes the response for its client: a server-rendered page for the web, lean JSON for the app, rows built for a remote on the TV.
Open one web checkout
Subscribe opens a checkout in Billing, which writes a pending row to Postgres whose ID becomes the idempotency key for every charge attempt. The web shows the checkout page; the app opens it in the phone's browser and the TV shows it as a QR code the phone scans, so every client pays through one web checkout, outside app-store billing. The card goes only into the payment provider's hosted fields, and Billing sees a token.
Charge once, confirm by webhook
Billing creates the subscription at the payment provider with the checkout ID as the idempotency key, so a retried request cannot charge twice; when the bank asks for 3-D Secure, the browser completes the challenge with the provider. The provider's signed webhook, not the browser's redirect, marks the invoice paid. A RabbitMQ timer fires 30 minutes after the checkout opens: if no webhook has arrived, Billing asks the provider for the payment's state, then activates or expires the checkout.
What can go wrong: A forged or replayed webhook; Billing verifies the provider's signature and stores each webhook ID, dropping one it has already handled.
Publish through the outbox
In one Postgres transaction Billing marks the subscription active and the invoice paid and writes subscription.activated to the outbox table. Debezium reads the outbox from the write-ahead log and publishes the event to billing.events, keyed by household so each household's events stay in order. The commit and its event cannot part: there is no moment where one is written and the other lost.
What can go wrong: Kafka down; the rows wait in the outbox and the replication slot holds the log, so Debezium publishes them, late but complete, when Kafka is back.
Grant the plan, send the receipt
Entitlements consumes subscription.activated, grants the plan's titles and screen limit to the household, and refreshes its copy in Redis, so the next play request passes. Notifications consumes the same event and sends the receipt and a welcome by email and push. Each reads in its own consumer group at its own pace; a send that keeps failing moves to a retry topic, then a dead-letter topic, instead of holding up its partition.
Ask to play
Press play reaches the Playback API through the client's BFF, with the access JWT. Playback checks the title's rating and region window in the Catalog against the maturity level in the token's claims, and asks Entitlements whether the household's plan covers the title here and now.
Open a play session
Playback opens a session in Play sessions, which counts the household's live streams in Redis, each with an expiry, and refuses one past the plan's screen limit. Playback returns a grant through the BFF: a signed CDN URL and a license token bound to the session.
What can go wrong: A stream left open on a lost device; with no heartbeat its Redis entry expires within minutes and frees the screen.
Stream and license
The player fetches the manifest and the segments from the CDN with the signed URL; a cache miss fetches the encrypted renditions from origin storage. It sends the license token to DRM licenses, which checks with Play sessions that the session is still live, then gets the title's content key from the key service. Licenses are short-lived, Widevine or PlayReady in browsers and FairPlay on Apple devices, so a revoked session stops at the next renewal.
Process the heartbeats
Every 30 to 60 seconds the player's heartbeat reaches Play sessions through the BFF, which keeps the session alive and publishes play.heartbeats. Flink reads them in event time, with watermarks for late arrivals, and writes quality of experience per minute to ClickHouse: start-up time and rebuffering by CDN, region and device. Reports turns the same history into watch minutes per title for royalties.
Catch abuse as it happens
The fraud detector consumes the login, device and play events: a new device, travel too fast to be real, one account streaming from many homes. It publishes risk.actions; Identity revokes the refresh-token family, Play sessions ends the flagged streams, and the BFFs ask for MFA on the next request, so a stolen login stops working within minutes.
Track what people watch
The apps batch impressions, searches, plays and seeks to their BFF, which publishes them to activity topics keyed by profile. Recommendations reads recent activity to re-rank rows such as Continue watching, and Flink groups the rest into viewing sessions for analysts and the nightly training that rebuilds each profile's candidate rows in Redis.
Watch the platform
An OpenTelemetry agent on every node collects each pod's logs and spans and publishes them to Kafka, which absorbs bursts; ClickHouse stores them for search and trace views. Prometheus scrapes every pod, Kafka's consumer lag and Flink's business metrics, such as payment declines and failed play starts; Alertmanager pages the on-call when an SLO burns too fast, and Grafana draws the dashboards.
Weak points and compensations
Weak point Three BFFs call the same services, so a change to a service's API reaches three codebases.
Compensation Each service publishes a versioned contract, and consumer-driven contract tests run every BFF against it before a release; a breaking change ships as a new version beside the old one.
Weak point Kafka carries events, change data, activity and logs, so one busy use can slow the rest.
Compensation Each use has its own topics, partitions and retention, and logs and spans run under a client quota, so a burst of logs cannot delay a grant.
Weak point A payment confirmed by webhook can reach Billing minutes after the user paid.
Compensation The checkout page follows Billing's state, the RabbitMQ timer asks the provider when no webhook arrives, and a push on activation tells every signed-in device as soon as the plan is granted.
Weak point Playback trusts the maturity level in the access token, which can be out of date for the token's lifetime.
Compensation Access tokens live five minutes, and changing a profile's maturity level revokes its refresh tokens, so the next refresh carries the new level.
Trade-offs
- A BFF per client gets each device's sign-in and payloads right, at the cost of three services that call the same backends.
- One web checkout for every client keeps a single payment path and avoids app-store fees, but phone and TV users leave the app to pay.
- Kafka as the one spine gives every consumer the same replayable history, but every team now depends on its partitions, retention and upgrades.