KEMBAR78
gh-133171: Make the executor_list thread-safe by aisk · Pull Request #140038 · python/cpython · GitHub
Skip to content

Conversation

@aisk
Copy link
Contributor

@aisk aisk commented Oct 13, 2025

The idea is simple and naive: just add a PyMutex to the PyInterpreterState for the executor_list under the nogil build, and lock or unlock the mutex in all functions that operate on the executor_list.

Since I'm not very familiar with the related code, there might be a better way to do this, so any feedback would be appreciated!

@bedevere-app bedevere-app bot mentioned this pull request Oct 13, 2025
7 tasks
@aisk aisk added the skip news label Oct 13, 2025
@aisk aisk changed the title gh-133171: Add lock on operations to executor_list gh-133171: Make the executor_list thread-safe Oct 13, 2025
Copy link
Member

@ZeroIntensity ZeroIntensity left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not entirely sure that a mutex is safe here, because we're locking code paths that call Py_DECREF. @Fidget-Spinner, can these paths escape?

#endif

// Executor list lock macros for thread-safe access to executor linked lists
#define EXECUTOR_LIST_LOCK(interp) \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The point of FT_MUTEX_LOCK is to prevent macros like this. If you're really against repeating interp->executor_list_lock several times, you can define a macro for that instead.

@Fidget-Spinner
Copy link
Member

In theory, Py_DECREF(executor) could clear the (future) constants pool that we plan to add, which would hold arbitrary constant objects, which can escape.

@corona10
Copy link
Member

corona10 commented Oct 15, 2025

I am not sure if making executor_list operations thread-safe at this point is good idea.
There is no way to verify that we make the thread-safe until we actually integrate the whole pipeline. I do believe that we have to do more things before we start to work on this task.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants