Is there a public Pub/Sub Lite topic?

Featured in many Google Cloud quickstarts and tutorials, this public Pub/Sub topic projects/pubsub-public-data/topics/taxirides-realtime allows anyone to attach a subscription to it and listen to fake taxi rides data in real time. Does such a public topic exist for Pub/Sub Lite too? In this blog post, I will explain why having a public Pub/Sub Lite topic is infeasible, how you can still share a Pub/Sub Lite topic resource with others, and finally, what you can do to route the fake taxi rides data from the Pub/Sub public topic to your Pub/Sub Lite topic.

Before I delve into Pub/Sub Lite, let me go over how to create a subscription to a Pub/Sub topic. Assuming I have a project bamboo-copilot-311421. I can create a subscription to the public Pub/Sub topic like this:

gcloud pubsub subscriptions create bobcat \
  --project=bamboo-copilot-311421 \
  --topic=projects/pubsub-public-data/topics/taxirides-realtime

The reason you can do it too is because this public topic resource has an IAM binding that specifically grants allUsers the role roles/pubsub.subscriber, which includes the permission pubsub.topics.attachSubscription.

After the subscription is created, I can pull messages from this subscription with:

gcloud pubsub subscriptions pull bobcat \
  --project=bamboo-copilot-311421 \
  --limit=1 \
  --auto-ack=true

I can pull messages from this subscription because the account that I’m signed in or my work email in this case has the project.owner role, which includes the permission pubsub.subscriptions.consume.

If you wonder which gcloud account and project you are currently using on your machine, try:

gcloud config list

How do you see which roles are bound to your account?

export PROJECT=$(gcloud config get-value project)
export EMAIL=$(gcloud config get-value account)
gcloud projects get-iam-policy $PROJECT \
  --format="table(bindings.members,bindings.role)" \
  --flatten="bindings[].members" \
  --filter="bindings.members=user:$EMAIL"

Attaching a subscription to an existing Pub/Sub Lite topic follows similar steps. Assuming my project ID bamboo-copilot-311421 has the project number 831370759181. The gcloud command to create a subscription to my owned Pub/Sub Lite topic looks like:

gcloud pubsub lite-subscriptions create starfish \
  --project=bamboo-copilot-311421 \
  --topic=projects/831370759181/locations/us-east1-b/topics/fish \
  --zone=us-east1-b

Using a similar command, you won’t be able to attach a subscription to this topic in your project because Pub/Sub Lite does not share topic and subscription resources across projects. The best you can do is being allowed to create a Pub/Sub Lite subscription in my project. This would require me to add some IAM bindings for your account:

gcloud projects add-iam-policy-binding bamboo-copilot-311421 \
  --member='user:$YOUR-EMAIL' \
  --role='roles/browser'
gcloud projects add-iam-policy-binding bamboo-copilot-311421 \
  --member='user:$YOUR-EMAIL' \
  --role='roles/pubsublite.admin' \
  --condition-from-file='conditions.json'

where conditions.json looks like:

{
  "title": "fish",
  "description": "Allow creating subscriptions to a specific Pub/Sub topic",
  "expression":
     "(resource.type == 'pubsublite.googleapis.com/Topic' && 
     resource.name == 'projects/831370759181/locations/us-central1-b/topics/fish')"
}

I use the --condition-from-file flag here to add an IAM condition because it can limit to which Pub/Sub Lite topic I intend to open access. There are more parameters which you can set in IAM conditions.

For --member, a service account works too. Service accounts are the recommended way for Google Cloud client libraries to authenticate with Google Cloud services. The authentication is done via a service account key file pointed to by the environment variable GOOGLE_APPLICATION_CREDENTIALS.

Instead of using the predefined role roles/pubsublite.admin, a custom role with a smaller set of permissions for creating subscriptions and consuming messages would also work.

Generally, editing IAM conditions in Cloud Console is much easier than using gcloud commands in the terminal.

To have a public Pub/Sub Lite topic that everyone can attach subscriptions to, a project owner must add everyone’s accounts — emails or service accounts — in his or her project and subject everyone to slightly different IAM conditions based on different subscription names. Otherwise, people could end up accessing other’s subscriptions and consuming messages from those. Moreover, everyone must share the subscribing throughput in the project (max at 128 MiB/s) and the subscribing throughput per Pub/Sub Lite topic partition (4-32 MiB/s). Other hard limits — only 1,000 subscriptions are allowed for a Lite topic in a project — also apply. Billing-wise, the public Pub/Sub Lite topic’s project owner will be billed for everyone’s Pub/Sub Lite subscriptions and traffic, which will inevitably add up. As a result of all of the above, a truly public Pub/Sub Lite topic is not feasible.

What if you are interested in having your own streaming Pub/Sub Lite source just like the public Pub/Sub topic? I can think of a scenario where you already have code that tests against Pub/Sub, and you want to see how Pub/Sub Lite performs in comparison.

Well, setting one up is not too hard. You can stand up a Dataflow pipeline that constantly publishes messages from the Pub/Sub public topic to a Pub/Sub Lite topic. You just need a Pub/Sub Lite topic.

When you are ready, head over to Dataflow Console to create a template.

Under “Dataflow templates”, select “Custom Template”. Then provide the path to a template file which I have already made public: gs://pubsub-public-data/templates/pubsublite-public-topic-template.json. Provide your Pub/Sub Lite topic name as well. You may need to enable the Dataflow Flex Template service if you haven’t already done so.

Click “Run Job”.

It takes Dataflow a few moments to get the pipeline ready. Once it is running, you should see messages arriving in your Pub/Sub topic by visiting the Lite topic details page or try to pull a few messages from its subscription.

You can check out the code and the commands I used to create this custom template on GitHub.

Hope I have made it clear in this blog article why Pub/Sub Lite doesn’t have a public topic and how you can still share your Pub/Sub Lite topic with others. Also hope that you find the Dataflow template useful for your testing cases. Thank you for reading!