Is there anyone who uses a dataflow pipeline in the Asia-northeast3(Seoul) Region?
There are 15 pipelines spread over 00:00 ~ 08:00 KST every day, and ZONE_RESOURCE_POOL_EXHAUSTED error frequently occurs, like 4 days a week nowadays.
I’m wondering if it truly is a lack of resources in Seoul, Korea, or if it’s derived from the infrastructure setting of mine.
All the experiences regarding the error would be appreciated even if it has not occurred in the same region.
Hello @Bo_Hyun_Hwang,
If you look at the Dataflow Logs or Log Explorer, you should see which resource is concerned. Also, looking at the quotas pages while using Dataflow would instantly reveal which resource is being overused, but I don’t think that would be easy considering your flow runs during the night.
This error happened to me when my Dataflow was trying to autoscale and/or reach its maximum (allowed) number of workers.
In your case, I would say the problem is coming from a “max workers” or “autoscaling” configuration in your Dataflow. It could also come from your GCP project recently using more resources from your pool, so your Dataflow cannot run as it used to.
You can check your quotas and adapt your needs, perhaps by changing the machine type used or the region your Dataflow is running if the reached quotas aren’t global.