Serverless for Apache Spark を使用して BigQuery を読み込む のレビュー

Maho Noguchi · 1年以上前にレビュー済み

Kabundu Edward · 1年以上前にレビュー済み

As a beginner in the world of Date Engineering on GCP, I struggle to understand the point of Dataproc in this lab. Also, I wanted to let you new that during the spark execute command I got the following error a couple of times. After the third try, it worked: ERROR: (gcloud.beta.dataproc.batches.submit.pyspark) Batch job is FAILED. Detail: Insufficient 'CPUS' quota. Requested 12.0, available 11.0. Your resource request exceeds your available quota. See https://cloud.google.com/compute/resource-usage. Use https://cloud.google.com/docs/quotas/view-manage#requesting_higher_quota to request additional quota. Running auto diagnostics on the batch. It may take few minutes before diagnostics output is available. Please check diagnostics output by running 'gcloud dataproc batches describe' command.

Toaiari Andrea · 1年以上前にレビュー済み

Wu Beini · 1年以上前にレビュー済み

Restrepo Castrillón Ronald Alberto · 1年以上前にレビュー済み

Berti Antonio · 1年以上前にレビュー済み

I see opportunities to improve this quick lab: 1) The lab is missing one step which is granting permissions to get the dataproc cluster. I've got the following error message in the VM while executing the batch job: ```` ERROR: (gcloud.beta.dataproc.batches.submit.pyspark) Batch job is FAILED. Detail: Multiple Errors: - Failed to fetch cluster for batch - Permission 'dataproc.clusters.get' denied on resource '//dataproc.googleapis.com/projects/qwiklabs-gcp-00-c467d66f6efc/regions/us-east4/clusters/srvls-batch-1b1a8482-374f-4c44-83d7-6bc417531bed' (or it may not exist). ``` Fortunatelly I was able to figure it out through IAM permissions configuration, but the lab does not provide that guidance. 2) It can be out of scope for a lab, but I think somehow it lacks the explanation for what use cases this solution would be preferred over other ones.

Lauro Caio · 1年以上前にレビュー済み

na

Nayak Lipsita · 1年以上前にレビュー済み

castiblanco julian · 1年以上前にレビュー済み

Lafarie William · 1年以上前にレビュー済み

"Download these files, click that button, and stuff will happen". That's a summary for this lab. I never worked with Spark before, and I learnt nothing from this lab. It doesn't force you to review what you downloaded, or write you're own transformation (this is an ETL module after all). Additionally, it returned an error the first time I executed the Spark code. With no changes at all, it ran ok the second time.

Baquero Pau · 1年以上前にレビュー済み

faced that max-workers on region lower that submit config

Lopoukhov Alexander · 1年以上前にレビュー済み

Rathi Gaurang · 1年以上前にレビュー済み

Abdelaziz Mohamed · 1年以上前にレビュー済み

srl Virtualmente Italia · 1年以上前にレビュー済み

Sharan Satyam · 1年以上前にレビュー済み

Patnala Priyanka · 1年以上前にレビュー済み

nice

ANWARDEEN NIVASH · 1年以上前にレビュー済み

Festa Gaetano · 1年以上前にレビュー済み

GOOGLEUSER nikhil vihar · 1年以上前にレビュー済み

Mallaiah Yogesh · 1年以上前にレビュー済み

GOOGLEUSER nikhil vihar · 1年以上前にレビュー済み

Rodriguez Humberto · 1年以上前にレビュー済み

HERNANDEZ QUEZADA CESAR · 1年以上前にレビュー済み

Gasparotto Filho Norberto · 1年以上前にレビュー済み