GSP1298

Overview

Imagine you are a data engineer for Cymbal Gaming. You and the development team are building a new e-sports game, "Galactic Grand Prix." The game produces real-time data based on heads-up games between two players from different teams. For example, two players will compete in one event. A winner is determined, and then points are awarded to the winning player and team. You are tasked with building a solution to handle this streamed data, leveraging a Dataflow pipeline to ingest data from Pub/Sub, transform it using Python code, and store the results in BigQuery tables. These tables will then be used to visualize the results as player and team dashboards.

You have read that Pub/Sub, Dataflow and BigQuery could be used for this type of use case. You have also learned that Gemini can help you along the way, for example, if you get stuck with writing a new query you could use Code Assist to help review and debug your code. It could even help you with suggestions to resolve issues. Using these features will help you to be more independent in your work and, perhaps, even more efficient. However, you are not sure how to get started.

When you start the lab, the environment will contain the resources shown in the following diagram.

By the end of the lab, you will have used the architecture to perform several tasks.

The following table provides a detailed explanation of each task in relation to the lab architecture.

Numbered Task	Detail
1.	Configure environment variables in Cloud Shell.
2.	Create the cloud resources: You will use Cloud Shell commands to create the Pub/Sub topic and subscription, and the BigQuery dataset.
3.	Retrieve the Python files from the repo: In this task, you retrieve the Python files from the repo and configure them for your project.
4.	Generate synthetic data and run the pipeline: With the Python files configured for your project, you will then run the files to generate synthetic data and initiate the pipeline. The esports-simulation.py file uses Python to generate continuous Pub/Sub messages for game events. You can then run the esports-pipeline.py file. Running this file launches a Dataflow pipeline that ingests the messages, transform them, and record the results to the raw_events, player score updates and team score updates tables in BigQuery.
5.	Verify the results in BigQuery: Now that you have consumed and transformed the data, and written the records to the tables in BigQuery, it is time to verify the results. You will run queries on each table to analyze and view the results.

Objectives

In this lab, you learn how to:

• Configure environment variables.
• Create the cloud resources.
• Retrieve Python files from the repo and modify them for your project.
• Generate synthetic data and run the pipeline.
• Verify the results in BigQuery.

Finally, you will have time to reflect on what you have learned in this lab and consider how you could address your own use cases with streaming data, and answer questions in your Lab Journal.

Setup and requirements

Before you click the Start Lab button

Read these instructions. Labs are timed and you cannot pause them. The timer, which starts when you click Start Lab, shows how long Google Cloud resources are made available to you.

This hands-on lab lets you do the lab activities in a real cloud environment, not in a simulation or demo environment. It does so by giving you new, temporary credentials you use to sign in and access Google Cloud for the duration of the lab.

To complete this lab, you need:

• Access to a standard internet browser (Chrome browser recommended).

Note: Use an Incognito (recommended) or private browser window to run this lab. This prevents conflicts between your personal account and the student account, which may cause extra charges incurred to your personal account.

• Time to complete the lab—remember, once you start, you cannot pause a lab.

Note: Use only the student account for this lab. If you use a different Google Cloud account, you may incur charges to that account.

How to start your lab and sign in to the Google Cloud console

1. Click the Start Lab button. If you need to pay for the lab, a dialog opens for you to select your payment method. On the left is the Lab Details pane with the following:

   • The Open Google Cloud console button
   • Time remaining
   • The temporary credentials that you must use for this lab
   • Other information, if needed, to step through this lab

2. Click Open Google Cloud console (or right-click and select Open Link in Incognito Window if you are running the Chrome browser).

   The lab spins up resources, and then opens another tab that shows the Sign in page.

   Tip: Arrange the tabs in separate windows, side-by-side.

   Note: If you see the Choose an account dialog, click Use Another Account.

3. If necessary, copy the Username below and paste it into the Sign in dialog.

   {{{user_0.username | "Username"}}}

   You can also find the Username in the Lab Details pane.

4. Click Next.

5. Copy the Password below and paste it into the Welcome dialog.

   {{{user_0.password | "Password"}}}

   You can also find the Password in the Lab Details pane.

6. Click Next.

   Important: You must use the credentials the lab provides you. Do not use your Google Cloud account credentials. Note: Using your own Google Cloud account for this lab may incur extra charges.

7. Click through the subsequent pages:

   • Accept the terms and conditions.
   • Do not add recovery options or two-factor authentication (because this is a temporary account).
   • Do not sign up for free trials.

After a few moments, the Google Cloud console opens in this tab.

Note: To access Google Cloud products and services, click the Navigation menu or type the service or product name in the Search field.

Task 1. Configure environment variables in Cloud Shell

In this task, you configure environment variables in Cloud Shell.

1. Open Cloud Shell.

2. You need to authorize use, as this is the first time you use Cloud Shell in this lab. To do this, click Authorize on the pop-up window.

3. Run the following commands.

   export PROJECT_ID="{{{project_0.project_id|set at lab start}}}" export REGION="{{{project_0.default_region|set at lab start}}}" export BUCKET_NAME="{{{project_0.project_id|set at lab start}}}-bucket"

4. Run the commands below to confirm the variables are stored.

   echo ${PROJECT_ID} echo ${REGION} echo ${BUCKET_NAME}

Task 2. Create the cloud resources

In this task, you will create the cloud resources including the Pub/Sub topic and subscription, and the BigQuery dataset using Cloud Shell.

Create the cloud resources

While remaining in Cloud Shell:

1. Run the following command to create the Pub/Sub topic.

   gcloud pubsub topics create esports_events_topic

2. Run the following command to create the Pub/Sub subscription.

   gcloud pubsub subscriptions create esports_events_topic-sub --topic=esports_events_topic

3. Run the following command to create the BigQuery dataset.

   bq --location=US mk --dataset esports_analytics

Confirm creation of resources

1. At the top of the console, there is a search function. Enter Pub/Sub here, and click Pub/Sub from the options listed. You see Pub/Sub topics listed, including the esports_events_topic.

2. Click on esports_events_topic, which is the topic ID. You see the Subscriptions tab appear for the topic, and the esports_events_topic-sub subscription is listed. This confirms you successfully created the topic and the subscription.

3. Return to the search function at the top of the console. Enter BigQuery here, and click BigQuery from the options listed. You see the Welcome to BigQuery Studio page appears with a pop-up for Welcome to BigQuery in the Cloud Console.

4. Click Done.

5. Within the Explorer pane, expand your project.

6. Notice at the bottom of the list, you see the esports_analytics dataset. This confirms you successfully created the BigQuery dataset.

Click Check my progress to verify the objective. Create the cloud resources

Task 3. Retrieve the Python files and configure them.

You will now use wget in Cloud Shell to retrieve the Python files from a public Cloud Storage bucket. Once retrieved you will open the files in Cloud Shell Editor and use the Code Assist feature of Gemini to explain how each file works. Finally you will use the Cloud Shell Editor to configure the files with details for your project and Cloud Storage bucket.

Retrieve the Python files

1. Return to Cloud Shell.

2. Run the following command to return to the home directory.

   cd ~

3. Create the esports directory and navigate to it.

   mkdir esports cd esports

4. Retrive the Python files using the wget commands below.

   wget https://storage.googleapis.com/spls/gsp1298/esports-simulation.py wget https://storage.googleapis.com/spls/gsp1298/esports-pipeline.py

   You see confirmation in the terminal that the files are downloaded.

Use code assist to explain what Python file does

1. Click on the Open Editor button in Cloud Shell. You see the Cloud Shell Editor open. You also see the Gemini Code Assist pane.

2. Close the Walkthrough tab.

3. In the Explorer pane, expand the esports folder. You see the Python files displayed.

4. Open the esports-simulation.py file.

5. At the top right of the file, click the arrow next to Gemini .

6. Click on Select Gemini Code Assist Project, to select the project to use for Gemini. From the list, select Project ID.

7. Within the Gemini Code Assist pane, at the bottom, you see a place to enter a prompt, where you see "Ask Gemini." Enter the following prompt:

   Review the code in the file esports-simulation.py. Explain what this code does.

You see that Code Assist explains what the code does in detail. At a high level the code will simulate an esports match of events and publishes these to a Google Cloud Pub/Sub topic.

8. Now use the same process to explain what the esports-pipeline.py file does with this prompt.

   Review the code in the file esports-pipeline.py. Explain what this code does.

Time to reflect

1. Considering your data and use cases, think of how you would use Python to generate synthetic data or collect data from your applications, and then publish this data as Pub/Sub messages to a topic in your own development workflow. Record your responses in your lab journal.

Note: Gemini was used to assist the developer who created the esports-simulation.py file. Consider using Gemini to generate synthetic streaming data with Python for your projects, and help explain and troubleshoot your code with Code Assist.

2. Using your own words, what does the esports-pipeline.py file do?

3. Considering your data and use cases, think of how you would use a Dataflow pipeline to consume Pub/Sub messages, to transform the data within them and store the reults in BigQuery.

Click Check my progress to verify the objective. Retrieve the Python files and configure them

Configure the files for your project and bucket

While still in the esports-pipeline.py file:

1. Around line 11, set the PROJECT_ID variable to your Project_ID. You can find this at the top of the Google Cloud Console or simply replace the line with the code below.

   PROJECT_ID = "{{{project_0.project_id|set at lab start}}}"

2. Around line 18, set the GCS_TEMP_LOCATION variable to your Cloud Storage bucket name (the bucket name provided to you at launch). The resulting line should look something like:

   GCS_TEMP_LOCATION = "gs://{{{project_0.project_id|set at lab start}}}-bucket/temp"

3. Around line 19, set the REGION variable to your default lab region. You can find this at the lab panel details or simply replace the line with the code below.

   REGION = "{{{project_0.default_region|set at lab start}}}"

4. Save the file.

5. Return to the esports-simulation.py file.

6. Around line 10, set the PROJECT_ID variable to your Project_ID, as with the other file, simply replace the line with the code below.

   PROJECT_ID = "{{{project_0.project_id|set at lab start}}}" Note: If you do not modify the files for your project, you will encounter errors at run-time in the Cloud Shell when you run these files in the next task.

7. Save the file.

Task 4. Generate synthetic data and run the pipeline

In this task, you will run the Python files to generate the synthetic data and run the pipeline.

Install the dependencies and run the simulator

1. Return to Cloud Shell terminal.

2. Confirm you are in the home directory.

   cd ~

3. Navigate to the esports directory.

   cd esports

4. Use pip to install the dependencies, specifically the Python library for Pub/Sub.

   pip install google-cloud-pubsub

5. Run the simulator. This script will run continuously, sending events to your Pub/Sub topic.

   python3 esports-simulation.py

   You will see output indicating that events are being published in the terminal. Keep this terminal open and running.

Confirm messages are published to the topic in the Pub/Sub console

1. Use the search function at the top of the console. This time, search for Pub/Sub, and click Pub/Sub in the list.

2. Click on esports_events_topic, which is the topic ID. You see the Subscriptions tab appear for the topic, and the esports_events_topic-sub subscription is listed.

3. Click on esports_events_topic-sub, which is the Subscription ID.

4. On the esports_events_topic-sub subscription detail page, click on the MESSAGES tab. You see a message indicating, "Click Pull to view messages and temporarily delay message delivery to other subscribers."

5. Click PULL. If you see messages listed here, this confirms that messages generated by the esports-simulation.py file are being received by Pub/Sub. Feel free to explore one of the messages, using the view contents button.

Attempt to Run the Dataflow pipeline

1. Open a new Cloud Shell tab. Click the + icon on the Cloud Shell terminal bar to open a second terminal.

2. Confirm you are in the home directory.

   cd ~

3. Navigate to the esports directory.

   cd esports

4. Install the Python Dependencies for Dataflow (apache-beam). In the new terminal, set up a virtual environment and install the required libraries.

   python3 -m venv df-env source df-env/bin/activate pip install apache-beam[gcp]

5. Attempt to launch the Dataflow pipeline. Run the pipeline script using the command below. Make sure you've set your PROJECT_ID and BUCKET_NAME environment variables in this new terminal as well.

   export PROJECT_ID="{{{project_0.project_id|set at lab start}}}" export BUCKET_NAME="{{{project_0.project_id|set at lab start}}}-bucket" python3 esports-pipeline.py \ --project=$PROJECT_ID \ --region={{{project_0.default_region | Region}}} \ --runner=DataflowRunner \ --streaming \ --temp_location=gs://$BUCKET_NAME/temp \ --job_name=esports-leaderboard-pipeline

   This command submits the job to the Dataflow service. It will take a few minutes for the job to start up and begin processing data.

   However, at this point you will see a warning and an errors like these below:

   WARNING:apache_beam.options.pipeline_options:Bucket specified in staging_location has soft-delete policy enabled. To avoid being billed for unnecessary storage costs, turn off the soft delete feature on buckets that your Dataflow jobs use for temporary and staging storage. For more information, see https://cloud.google.com/storage/docs/use-soft-delete#remove-soft-delete-policy. ERROR:apache_beam.runners.dataflow.dataflow_runner:2025-07-11T14:50:27.572Z: JOB_MESSAGE_ERROR: The Dataflow service agent cannot access the worker service account. Ensure that the Dataflow API is enabled for your project. In addition, verify that for the worker service account, the Dataflow service agent principal has the 'Cloud Dataflow Service Agent' role. To grant the role, see https://cloud.google.com/iam/docs/manage-access-service-accounts#view-access. On the Service Accounts page, select the worker service account, open the Permissions tab, and select 'Include Google-provided role grants' to verify roles. To learn more about service accounts, see https://cloud.google.com/dataflow/docs/concepts/security-and-permissions#permissions

   What do these mean? First, let's start with the WARNING.

   The warning indicates that the bucket used in this lab, -bucket, must have the Soft Delete protection feature disabled. We recommend you disable it, and we will provide instuctions below to disable it.

   Regarding the ERROR... Now, let's address the ERROR... What does it mean?

   There are two possible causes for this error.

   1. The Dataflow API is not enabled. However, at the launch of your lab, we enabled this for you. You can confirm this by going to APIs & Services.

   2. The actual cause of the error is you need to add the Cloud Dataflow Service Agent role to the service account for the Cloud Compute. Below, we will include insturctions to complete this task.

Disabling Soft Delete feature on the Cloud Storage bucket

1. Use the search function at the top of the console. Search for Cloud Storage, and click the Cloud Storage option in the list. You see the Overview page appears.

2. Click Buckets. You see the Buckets page appears and your bucket, -bucket is displayed in the list.

3. Click on -bucket. You see the details page appears for the bucket.

4. Click on Protection. You see a list of options appears, with Soft delete policy (for data recovery) included.

5. Click Disable. You see a pop-up appears for Disable soft delete policy.

6. Click Confirm. Soft delete policy is now disabled.

Adding the Cloud Dataflow Service Agent role

1. Use the search function at the top of the console to go to IAM. You see the IAM page appears.

2. Click Service Accounts. You see the Service Accounts list appears. One of the service accounts listed is the one for Compute Engine. Your service account for the Compute Engine service would look similar to the one below:

655017706949-compute@developer.gserviceaccount.com

3. To add the Cloud Dataflow Service Agent role to this service account, begin by clicking Actions next to it.

4. From the list of options provided, click Manage permissions.

5. Under Manage service account permissions, click Manage access. Notice the Editor role is already included.

6. Click + Add another role.

7. Use the Select a Role option. Search for and select the Cloud Dataflow Service Agent role.

8. Click Save.

9. Confirm the role is added.

Run the Dataflow pipeline

1. Return to Cloud Shell.

2. Confirm you are on the second terminal tab, the one for running the Dataflow pipeline (esports-pipeline.py).

3. Run the following command, once again.

   export PROJECT_ID="{{{project_0.project_id|set at lab start}}}" export BUCKET_NAME="{{{project_0.project_id|set at lab start}}}-bucket" python3 esports-pipeline.py \ --project=$PROJECT_ID \ --region={{{project_0.default_region | Region}}} \ --runner=DataflowRunner \ --streaming \ --temp_location=gs://$BUCKET_NAME/temp \ --job_name=esports-leaderboard-pipeline

   This time you should see a message indicating, "Autoscaling is enabled for Dataflow Streaming Engine. Workers will scale between 1 and 100 unless maxNumWorkers is specified."

   This means that you have overcome the soft delete warning, and the Cloud Dataflow Service Agent error.

Access the Dataflow Service and observe the Pipeline

1. Use the search function at the top of the console to search for and select the Dataflow service. You see the Jobs page appears, with two Jobs of the same name. One is running and the other failed.

2. Click on the running job. You see the job graph appears. At the bottom of the page, you see the Logs pane.

3. Expand the Logs pane. You see tabs for JOB LOGS and WORKER LOGS.

4. Click WORKER LOGS.

5. Review the logs provided here. If you see logs for indicating the raw_events, player_score_updates and/or team_score_updates tables created, you can move on to the next task.

   Note: It may take 5 to 7 minutes to see these logs for the tables being created.

Click Check my progress to verify the objective. Generate synthetic data and run the pipeline

Task 5. Verify the results in BigQuery

In this task, you will run SQL queries to verify the results of the processed messages. To accomplish this, you will create two views, and then query the views to display the player and team leaderboards. You will also take time to reflect on how Looker could be used to enhance the leaderboard visualizations.

Create the views in BigQuery

1. Return to BigQuery.

2. In the explorer, expand your project and select the esports_analytics dataset.

3. Click the + (plus) to create a new query.

4. In the query tab provided, enter the following query.

   -- Query 1: Create the Player Leaderboard View -- This view finds the most recent score for each player and ranks them. CREATE OR REPLACE VIEW `esports_analytics.player_leaderboard_live` AS SELECT -- Use the RANK() window function to calculate the rank in real-time RANK() OVER (ORDER BY total_score DESC) as rank, player_id, total_score, last_updated FROM ( -- This subquery gets only the single most recent score for each player SELECT player_id, total_score, last_updated, ROW_NUMBER() OVER (PARTITION BY player_id ORDER BY last_updated DESC) as rn FROM `esports_analytics.player_score_updates` ) WHERE rn = 1; -- Query 2: Create the Team Leaderboard View -- This view finds the most recent score for each team and ranks them. CREATE OR REPLACE VIEW `esports_analytics.team_leaderboard_live` AS SELECT -- Use the RANK() window function to calculate the rank in real-time RANK() OVER (ORDER BY total_wins DESC) as rank, team_id, total_wins, last_updated FROM ( -- This subquery gets only the single most recent score for each team SELECT team_id, total_wins, last_updated, ROW_NUMBER() OVER (PARTITION BY team_id ORDER BY last_updated DESC) as rn FROM `esports_analytics.team_score_updates` ) WHERE rn = 1;

   This query creates two views: one for the player leaderboard and the other for the team leaderboard. It will find the most recent scores for the player and the team, and rank them accordingly.

Run a query to display the player leaderboard

1. Click the plus to create a new query.

2. In the query tab provided, enter the following query.

   SELECT * FROM `esports_analytics.player_leaderboard_live` ORDER BY rank;

Run a query to display the team leadeboard

1. Click the plus to create a new query.

2. In the query tab provided, enter the following query.

   SELECT * FROM `esports_analytics.team_leaderboard_live` ORDER BY rank;

Time to reflect

Using your journal, answer the following questions:

1. Which player is currently in the lead?
2. Which team is currently in the lead?

Click Check my progress to verify the objective. Verify the results in BigQuery

Congratulations!

You have generated Google Cloud resources to support your esports Dataflow pipeline, including a Pub/Sub topic and subscription, a BigQuery Dataset, tables and views, and the pipeline itself using simulated data generated from Python scripts. You also used Code Assist to help you explain the code in these scripts. You are becoming more confident with Google Cloud each day and can use Gemini to supplement your knowledge and skills with Data Engineering workflows.

Next steps / learn more

• Generate a SQL query
• Explain SQL queries
• Gemini Models
• Generative AI
• Write better prompts for Gemini for Google Cloud

Google Cloud training and certification

...helps you make the most of Google Cloud technologies. Our classes include technical skills and best practices to help you get up to speed quickly and continue your learning journey. We offer fundamental to advanced level training, with on-demand, live, and virtual options to suit your busy schedule. Certifications help you validate and prove your skill and expertise in Google Cloud technologies.

Manual last updated on August 11, 2025

Lab last tested on August 11, 2025

Copyright 2025 Google LLC. All rights reserved. Google and the Google logo are trademarks of Google LLC. All other company and product names may be trademarks of the respective companies with which they are associated.