Python C API - Is it thread safe?
NickName:DenverCoder9 Ask DateTime:2017-02-02T23:38:50

Python C API - Is it thread safe?

I have a C extension that is called from my multithreaded Python application. I use a static variable i somewhere in a C function, and I have a few i++ statements later on that can be run from different Python threads (that variable is only used in my C code though, I don't yield it to Python).

For some reason I haven't met any race condition so far, but I wonder if it's just luck...

I don't have any thread-related C code (no Py_BEGIN_ALLOW_THREADS or anything).

I know that the GIL only guarantees single bytecode instructions to be atomic and thread-safe, thus statements as i+=1 in Python are not thread-safe.

But I don't know about a i++ instruction in a C extension. Any help ?

Copyright Notice:Content Author:「DenverCoder9」,Reproduced under the CC 4.0 BY-SA copyright license with a link to the original source and this disclaimer.
Link to original article:https://stackoverflow.com/questions/42006337/python-c-api-is-it-thread-safe

Answers
DavidW 2017-02-03T11:36:35

Python will not release the GIL when you are running C code (unless you either tell it to or cause the execution of Python code - see the warning note at the bottom!). It only releases the GIL just before a bytecode instruction (not during) and from the interpreter's point of view running a C function is part of executing the CALL_FUNCTION bytecode.* (Unfortunately I can't find a reference for this paragraph currently, but I'm almost certain it's right)\nTherefore, unless you do anything specific your C code will be the only thread running and thus any operation you do in it should be thread safe.\nIf you specifically want to release the GIL - for example because you're doing a long calculation which doesn't interfere with Python, reading from a file, or sleeping while waiting for something else to happen - then the easiest way is to do Py_BEGIN_ALLOW_THREADS then Py_END_ALLOW_THREADS when you want to get it back. During this block you cannot use most Python API functions and it's your responsibility to ensure thread safety in C. The easiest way to do this is to only use local variables and not read or write any global state.\nIf you've already got a C thread running without the GIL (thread A) then simply holding the GIL in thread B does not guarantee that thread A won't modify C global variables. To be safe you need to ensure that you never modify global state without some kind of locking mechanism (either the Python GIL or a C mechanism) in all your C functions.\n\nAdditional thought\n* One place where the GIL can be released in C code is if the C code calls something that causes Python code to be executed. This might be through using PyObject_Call. A less obvious place would be if Py_DECREF caused a destructor to be executed. You'd have the GIL back by the time your C code resumed, but you could no longer guarantee that global objects were unchanged. This obvious doesn't affect simple C like x++.\n\nBelated Edit:\nIt should be emphasised that it's really, really, really easy to cause the execution of Python code. For this reason you shouldn't use the GIL in place of a mutex or actual locking mechanism. You should only consider it for operations that are really atomic (i.e. a single C API call) or entirely on non-Python C objects. You won't lose the GIL unexpected while executing C Code, but a lot of C API calls may release the GIL, do something else, and then regain the GIL before returning to your C code.\nThe purpose the GIL is to make sure that the Python internals don't get corrupted. The GIL will continue to serve this purpose within an extension module. However race conditions that involve valid Python objects arranged in ways you don't expect are still available to you. For example:\nPySequence_SetItem(some_list, 0, some_item);\nPyObject* item = PySequence_GetItem(some_list, 0);\nassert(item == some_item); // may not be true \n// the destructor of the previous contents of item 0 may have released the GIL\n",


More about “Python C API - Is it thread safe?” related questions

Python C API - Is it thread safe?

I have a C extension that is called from my multithreaded Python application. I use a static variable i somewhere in a C function, and I have a few i++ statements later on that can be run from diff...

Show Detail

python google api thread safe

I am trying to download files from google drive in thread safe way. I have this two classes: APIConnector, GSuiteUserManager. I build this classes in order to make my google api client thread safe.

Show Detail

Is google drive api for c# thread safe?

I'm using google drive API in a multi-thread C# application. I would like to know if the Google dot net client library it's thread-safe or not. Also I would like to know what's more right: creat...

Show Detail

Embedding Python thread safe

I'm trying to use Python in a module for an analysis software of vehicle bus systems. For this I have to embed Python in a thread safe manner, since there can be multiple instances of the module w...

Show Detail

Thread safe client lib on app engine (python)

I found a little sample code about bigquery insert in one of Google's git repositories. https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/appengine/standard/bigquery/main.py ...

Show Detail

Is Python list slicing thread safe?

Coming from Are lists thread-safe?, I need to know if specifically list slicing is thread safe. It's not clear to me from the linked article What kinds of global value mutation are thread-safe?. B...

Show Detail

Is this python code thread-safe?

I am trying to make my chunk of code non-thread-safe, in order to toy with some exceptions that I want to add on later. This is my python code: from time import sleep from decimal import * from

Show Detail

Is C read() Thread Safe?

I'm writing a program where multiple threads might read from a file simultaneously. No threads are writing to the file, but they might each copy its contents to separate memory segments. To do imp...

Show Detail

Why is reading not thread-safe?

I was wondering why reading from memory is not thread-safe. Among what I've seen so far, inter alia this question, reading from memory does not appear to be thread-safe. I've been coding in Python...

Show Detail

Is the official client for the YouTube API thread-safe?

I'm using the official Python client for the YouTube API (called gdata.youtube.service.) My question: Is it thread-safe? For example, I want to do both GetUserFavoritesFeed and GetYouTubeUserFeed ...

Show Detail