google cloud pubsub - Multiple app instances consuming all messages from Pub/Sub topic

Question

What's the right way to scale out app (in my case k8s deployment) having each instance receiving all topic messages? Should I create a new subscription on application start up? Wouldn't it be expensive?

question from:https://stackoverflow.com/questions/65902921/multiple-app-instances-consuming-all-messages-from-pub-sub-topic

Answer 1

If every instance needs to receive all messages, then yes, you will need to create a subscription per instance. Note that there is a limit of 10,000 subscriptions per topic. You will pay for delivery of messages to each subscription, so the total cost will be approximately (total throughput)*(1 + # of subscriptions), since you pay once for the publish and then for delivery for each subscription.

You would also want to make sure to delete subscriptions once each instance stops. You could rely on subscription expiration, but if you are using push subscriptions this may not work as the publishing of messages will be enough to keep the subscriptions alive. In that case, if instances can stop unexpectedly, you may have to have a cleanup job that periodically goes through and deletes old subscriptions.

Keep in mind that the instances will only receive messages published after the subscription was created.