Deadlock due to race condition in download thread when using Bigquery Storage API

There is a race condition in the download thread and the main thread when using Bigquery Storage API to fetch data.

https://github.com/googleapis/python-bigquery/blob/a76af359525ef3c49c958663f81fd24c9d35e1e7/google/cloud/bigquery/_pandas_helpers.py#L797-L801

https://github.com/googleapis/python-bigquery/blob/a76af359525ef3c49c958663f81fd24c9d35e1e7/google/cloud/bigquery/_pandas_helpers.py#L934-L942

When the download thread is blocked on `worker_queue.put(item)`, if the main thread exit, causing the pool to shut down, the download thread will be stuck. This behavior prevents the program from exiting.

	for page in rowstream.pages:
	if download_state.done:
	return
	item = page_to_item(page)
	worker_queue.put(item)

	finally:
	# No need for a lock because reading/replacing a variable is
	# defined to be an atomic operation in the Python language
	# definition (enforced by the global interpreter lock).
	download_state.done = True

	# Shutdown all background threads, now that they should know to
	# exit early.
	pool.shutdown(wait=True)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Deadlock due to race condition in download thread when using Bigquery Storage API #2032

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Deadlock due to race condition in download thread when using Bigquery Storage API #2032

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions