리뷰 Dataflow를 사용한 서버리스 데이터 처리 - Dataflow를 사용한 일괄 분석 파이프라인(Python)개

Ferdie O. · 4시간 전에 리뷰됨

spent two hours getting message not sufficient workers/resources, in zone/region, but restricted from selecting another.

Ferdie O. · 5시간 전에 리뷰됨

EZHUMALAI A. · 13시간 전에 리뷰됨

Vignesh T. · 14시간 전에 리뷰됨

The dataflow jobs failed with "Startup of the worker pool in us-east1 failed to bring up any of the desired 1 workers. This is likely a quota issue or a Compute Engine stockout. The service will retry." There was also an SSL Certificate error to the GCS bucket which I solved by "gcloud auth application-default login" and clicking the link and pasting the code from the link. Thirdly, part 2 of the lab required dill imports which I installed with the following command. pip install apache-beam[dill]

Sayed Fawad Ali S. · 1일 전에 리뷰됨

spent two hours getting message not sufficient workers/resources, in zone/region, but restricted from selecting another.

Ferdie O. · 2일 전에 리뷰됨

Had many errors running the pipeline due to missing certificates. Fixed it by adding: export GCE_METADATA_MTLS_MODE=none

Martin H. · 4일 전에 리뷰됨

vicente b. · 4일 전에 리뷰됨

The lab is very cool, but in all of the course labs I am hitting the problem that I cannot run dataflow jobs in the specified regions and zones because no workers are available in those zones (though I am running the jobs multiple times). It completely ruins the learning experience, because I am not able to finish any lab though they are very well prepared. Please, add the option to switch to different regions and zones or keep some compute quota for Qwik labs.

Jan K. · 5일 전에 리뷰됨

ZONE_RESOURCE_POOL_EXHAUSTED getting this error frequently

Ashok K. · 6일 전에 리뷰됨

Gustavo L. · 6일 전에 리뷰됨

Allan L. · 7일 전에 리뷰됨

Harsh A. · 8일 전에 리뷰됨

Guilherme A. · 13일 전에 리뷰됨

Wayne F. · 13일 전에 리뷰됨

Luis Antonio C. · 14일 전에 리뷰됨

Luis Antonio C. · 14일 전에 리뷰됨

already submitted feedback for this lab, with the issues that i've encountered

Rafael D. · 14일 전에 리뷰됨

WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1017)'))': /computeMetadata/v1/instance/service-accounts/default/?recursive=true Traceback (most recent call last): File "/home/jupyter/training-data-analyst/quests/dataflow_python/3_Batch_Analytics/solution/batch_user_traffic_pipeline.py", line 99, in <module> run() File "/home/jupyter/training-data-analyst/quests/dataflow_python/3_Batch_Analytics/solution/batch_user_traffic_pipeline.py", line 78, in run (p | 'ReadFromGCS' >> beam.io.ReadFromText(known_args.input_path) File "/home/jupyter/training-data-analyst/quests/dataflow_python/3_Batch_Analytics/lab/df-env/lib/python3.10/site-packages/apache_beam/io/textio.py", line 808, in __init__ self._source = self._source_class( File "/home/jupyter/training-data-analyst/quests/dataflow_python/3_Batch_Analytics/lab/df-env/lib/python3.10/site-packages/apache_beam/io/textio.py", line 144, in __init__ super().__init__( File "/home/jupyter/training-data-analyst/quests/dataflow_python/3_Batch_Analytics/lab/df-env/lib/python3.10/site-packages/apache_beam/io/filebasedsource.py", line 127, in __init__ self._validate() File "/home/jupyter/training-data-analyst/quests/dataflow_python/3_Batch_Analytics/lab/df-env/lib/python3.10/site-packages/apache_beam/options/value_provider.py", line 193, in _f return fnc(self, *args, **kwargs) File "/home/jupyter/training-data-analyst/quests/dataflow_python/3_Batch_Analytics/lab/df-env/lib/python3.10/site-packages/apache_beam/io/filebasedsource.py", line 190, in _validate match_result = FileSystems.match([pattern], limits=[1])[0] File "/home/jupyter/training-data-analyst/quests/dataflow_python/3_Batch_Analytics/lab/df-env/lib/python3.10/site-packages/apache_beam/io/filesystems.py", line 240, in match return filesystem.match(patterns, limits) File "/home/jupyter/training-data-analyst/quests/dataflow_python/3_Batch_Analytics/lab/df-env/lib/python3.10/site-packages/apache_beam/io/filesystem.py", line 779, in match raise BeamIOError("Match operation failed", exceptions) apache_beam.io.filesystem.BeamIOError: Match operation failed with exceptions {'gs://qwiklabs-gcp-04-497cbdcab972/events.json': RefreshError(TransportError("Failed to retrieve https://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true from the Google Compute Engine metadata service. Compute Engine Metadata server unavailable. Last exception: HTTPSConnectionPool(host='metadata.google.internal', port=443): Max retries exceeded with url: /computeMetadata/v1/instance/service-accounts/default/?recursive=true (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1017)')))"))}

Mykola O. · 15일 전에 리뷰됨

Mara Malina F. · 15일 전에 리뷰됨

Daniela L. · 16일 전에 리뷰됨

Gabriela C. · 16일 전에 리뷰됨

Luis Antonio C. · 19일 전에 리뷰됨

Leighton C. · 19일 전에 리뷰됨

There are some issues with running the dataflow; first, the version of a library has to be downgraded: pip install "google-auth==2.43.0" due to error: https://github.com/googleapis/google-cloud-python/issues/16090. Secondly, one has to specify the machine type and in the part B additionally have to set worker zone: options.view_as(beam.options.pipeline_options.WorkerOptions).machine_type = "e2-standard-4" options.view_as(beam.options.pipeline_options.WorkerOptions).worker_zone = "us-west1-c"

Przemyslaw S. · 20일 전에 리뷰됨