Skip to content

Commit 0c4ca17

Browse files
itamaroencukou
andauthored
gh-157384: Document and test PyImport_CreateModuleFromInitfunc() semantics (GH-157389)
Document that single-phase init modules are added to sys.modules by the call itself (unlike multi-phase init modules), and that the spec name identifies the module, so a later call with the same name and a different init function reuses the cached module without calling the new init function. Add regression tests for both behaviors. Co-authored-by: Petr Viktorin <encukou@gmail.com>
1 parent 29fd3b8 commit 0c4ca17

4 files changed

Lines changed: 139 additions & 18 deletions

File tree

‎Doc/c-api/extension-modules.rst‎

Lines changed: 28 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -313,16 +313,32 @@ This is typically done using :c:func:`PyModule_Create` and functions like
313313
Single-phase initialization differs from the :ref:`default <multi-phase-initialization>`
314314
in the following ways:
315315
316-
* Single-phase modules are, or rather *contain*, “singletons”.
316+
* Single-phase modules with :c:member:`PyModuleDef.m_size` set to -1 are,
317+
or rather *contain*, “singletons”.
317318
318-
When the module is first initialized, Python saves the contents of
319+
When such a module is first initialized, Python saves the contents of
319320
the module's ``__dict__`` (that is, typically, the module's functions and
320321
types).
321322
322323
For subsequent imports, Python does not call the initialization function
323324
again.
324325
Instead, it creates a new module object with a new ``__dict__``, and copies
325326
the saved contents to it.
327+
Modules are matched to the saved contents by a (*module_name*, *file_name*)
328+
tuple where:
329+
330+
- *module_name* is the module's full dotted name, taken from
331+
:py:attr:`ModuleSpec.name <importlib.machinery.ModuleSpec.name>`
332+
(or the equivalent argument of the importing function).
333+
Note that names that the initialization function uses to create the module
334+
(for example, :c:member:`PyModuleDef.m_name`) are not taken into account.
335+
- *file_name* is the extension's file name.
336+
It is unset for modules not loaded from a file, such as built-ins and
337+
modules added with :c:func:`PyImport_CreateModuleFromInitfunc` or
338+
:c:func:`PyImport_AppendInittab`.
339+
340+
As part of the first initialization, Python also adds the module to
341+
:data:`sys.modules` under its name.
326342
For example, given a single-phase module ``_testsinglephase``
327343
[#testsinglephase]_ that defines a function ``sum`` and an exception class
328344
``error``:
@@ -344,6 +360,16 @@ in the following ways:
344360
345361
The exact behavior should be considered a CPython implementation detail.
346362

363+
* For single-phase modules with non-negative :c:member:`PyModuleDef.m_size`,
364+
Python may cache a pointer to the initialization function under the
365+
(*module_name*, *file_name*) key as for ``m_size == -1``.
366+
367+
Subsequent imports that specify a different initialization function
368+
(for example, using :c:func:`PyImport_CreateModuleFromInitfunc`)
369+
may call either the cached function or the provided "new" one.
370+
371+
The exact behavior should again be considered a CPython implementation detail.
372+
347373
* To work around the fact that ``PyInit_modulename`` does not take a *spec*
348374
argument, some state of the import machinery is saved and applied to the
349375
first suitable module created during the ``PyInit_modulename`` call.

‎Doc/c-api/import.rst‎

Lines changed: 22 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -408,7 +408,7 @@ Importing Modules
408408
409409
*spec* must be a :class:`~importlib.machinery.ModuleSpec` object.
410410
411-
*initfunc* must be an :ref:`initialization function <extension-export-hook>`,
411+
*initfunc* must be an :ref:`initialization function <extension-pyinit>`,
412412
the same as for :c:func:`PyImport_AppendInittab`.
413413
414414
On success, create and return a module object.
@@ -417,10 +417,27 @@ Importing Modules
417417
(Custom importers should do this in their
418418
:py:meth:`~importlib.abc.Loader.exec_module` method.)
419419
420-
If the spec's name is dotted and *initfunc* creates the module using only
421-
the last component of the name, the module's :attr:`~module.__name__`
422-
is set to the full name from the spec.
423-
424420
On error, return NULL with an exception set.
425421
422+
For modules using :ref:`legacy single-phase initialization <single-phase-initialization>`,
423+
most of the import process is done in this function, rather than in
424+
extra steps that importers typically take after
425+
:py:meth:`~importlib.abc.Loader.create_module`.
426+
Specifically:
427+
428+
- The module is added to :data:`sys.modules` under the *spec*'s
429+
:py:attr:`~importlib.machinery.ModuleSpec.name` as part of initialization.
430+
- If a single-phase module was previously imported under the
431+
same :py:attr:`~importlib.machinery.ModuleSpec.name` by this function
432+
or as a built-in module (for example, one registered with
433+
:c:func:`PyImport_AppendInittab`), Python may reuse the previous module's
434+
contents instead of calling *initfunc* again.
435+
- The module will be fully initialized by *initfunc*.
436+
Calling :c:func:`PyModule_Exec` on it will have no effect, though it is
437+
still recommended, in case the module later switches to
438+
:ref:`multi-phase initialization <multi-phase-initialization>`.
439+
- Also, if the *spec*'s :py:attr:`!name` contains dots, this name is used
440+
for prepending a parent package to the new module's
441+
:attr:`~module.__name__`.
442+
426443
.. versionadded:: 3.15

‎Lib/test/test_embed.py‎

Lines changed: 21 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -273,16 +273,27 @@ def test_create_module_from_initfunc(self):
273273
out, err = self.run_embedded_interpreter("test_create_module_from_initfunc")
274274
self.assertEqual(self._nogil_filtered_err(err, "embedded_ext"), "")
275275
self.assertEqual(out,
276-
"<module 'my_test_extension' (static-extension)>\n"
277-
"my_test_extension.executed='yes'\n"
278-
"my_test_extension.exec_slot_ran='yes'\n"
279-
"<module 'embedded_ext' (static-extension)>\n"
280-
"embedded_ext.executed='yes'\n"
281-
"<module 'sp_pkg.sp_submod' (static-extension)>\n"
282-
"sp_pkg.sp_submod.__name__='sp_pkg.sp_submod'\n"
283-
"sys.modules[\"sp_pkg.sp_submod\"] is sp_pkg.sp_submod=True\n"
284-
"\"sp_submod\" in sys.modules=False\n"
285-
)
276+
# multi-phase init: not in sys.modules until importlib adds it
277+
"created my_test_extension: in sys.modules=False\n"
278+
"<module 'my_test_extension' (static-extension)>\n"
279+
"my_test_extension.executed='yes'\n"
280+
"my_test_extension.exec_slot_ran='yes'\n"
281+
# single-phase init: added to sys.modules by the init function
282+
"created embedded_ext: in sys.modules=True\n"
283+
"<module 'embedded_ext' (static-extension)>\n"
284+
"embedded_ext.executed='yes'\n"
285+
# same name, different initfuncs: the cached module is
286+
# returned and the second initfunc is never called
287+
"a.which='A' b.which='A' a is b=True\n"
288+
"create_static_module.initfunc_calls()=(1, 0)\n"
289+
# Static extension
290+
"created sp_pkg: in sys.modules=False\n"
291+
"created sp_pkg.sp_submod: in sys.modules=True\n"
292+
"<module 'sp_pkg.sp_submod' (static-extension)>\n"
293+
"sp_pkg.sp_submod.__name__='sp_pkg.sp_submod'\n"
294+
"sys.modules[\"sp_pkg.sp_submod\"] is sp_pkg.sp_submod=True\n"
295+
"\"sp_submod\" in sys.modules=False\n"
296+
)
286297

287298
def test_inittab_submodule_multiphase(self):
288299
out, err = self.run_embedded_interpreter("test_inittab_submodule_multiphase")

‎Programs/_testembed.c‎

Lines changed: 68 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2374,6 +2374,58 @@ static int test_repeated_init_and_inittab(void)
23742374
return 0;
23752375
}
23762376

2377+
// Two different single-phase init functions for the same module name.
2378+
static int cmfi_initfunc_a_calls = 0;
2379+
static int cmfi_initfunc_b_calls = 0;
2380+
2381+
static PyModuleDef cmfi_same_name_a_def = {
2382+
PyModuleDef_HEAD_INIT,
2383+
.m_name = "same_name",
2384+
.m_size = -1,
2385+
};
2386+
2387+
static PyModuleDef cmfi_same_name_b_def = {
2388+
PyModuleDef_HEAD_INIT,
2389+
.m_name = "same_name",
2390+
.m_size = -1,
2391+
};
2392+
2393+
static PyObject*
2394+
PyInit_cmfi_same_name_a(void)
2395+
{
2396+
cmfi_initfunc_a_calls++;
2397+
PyObject *mod = PyModule_Create(&cmfi_same_name_a_def);
2398+
if (mod == NULL || PyModule_AddStringConstant(mod, "which", "A") < 0) {
2399+
Py_XDECREF(mod);
2400+
return NULL;
2401+
}
2402+
return mod;
2403+
}
2404+
2405+
static PyObject*
2406+
PyInit_cmfi_same_name_b(void)
2407+
{
2408+
cmfi_initfunc_b_calls++;
2409+
PyObject *mod = PyModule_Create(&cmfi_same_name_b_def);
2410+
if (mod == NULL || PyModule_AddStringConstant(mod, "which", "B") < 0) {
2411+
Py_XDECREF(mod);
2412+
return NULL;
2413+
}
2414+
return mod;
2415+
}
2416+
2417+
static PyObject*
2418+
create_same_name_b(PyObject* self, PyObject* spec)
2419+
{
2420+
return PyImport_CreateModuleFromInitfunc(spec, PyInit_cmfi_same_name_b);
2421+
}
2422+
2423+
static PyObject*
2424+
initfunc_calls(PyObject* self, PyObject* Py_UNUSED(args))
2425+
{
2426+
return Py_BuildValue("(ii)", cmfi_initfunc_a_calls, cmfi_initfunc_b_calls);
2427+
}
2428+
23772429
// A single-phase init submodule that only knows its short name,
23782430
// like pybind11's PYBIND11_MODULE(sp_submod, m) would.
23792431
static PyModuleDef cmfi_sp_submod_def = {
@@ -2423,6 +2475,10 @@ create_module(PyObject* self, PyObject* spec)
24232475
Py_DECREF(name);
24242476
return PyImport_CreateModuleFromInitfunc(spec, PyInit_embedded_ext);
24252477
}
2478+
if (PyUnicode_EqualToUTF8(name, "same_name")) {
2479+
Py_DECREF(name);
2480+
return PyImport_CreateModuleFromInitfunc(spec, PyInit_cmfi_same_name_a);
2481+
}
24262482
if (PyUnicode_EqualToUTF8(name, "sp_pkg")) {
24272483
Py_DECREF(name);
24282484
return PyImport_CreateModuleFromInitfunc(spec, PyInit_cmfi_sp_pkg);
@@ -2447,6 +2503,8 @@ exec_module(PyObject* self, PyObject* mod)
24472503

24482504
static PyMethodDef create_static_module_methods[] = {
24492505
{"create_module", create_module, METH_O, NULL},
2506+
{"create_same_name_b", create_same_name_b, METH_O, NULL},
2507+
{"initfunc_calls", initfunc_calls, METH_NOARGS, NULL},
24502508
{"exec_module", exec_module, METH_O, NULL},
24512509
{NULL}
24522510
};
@@ -2478,6 +2536,12 @@ test_create_module_from_initfunc(void)
24782536
L"import embedded_ext;"
24792537
L"print(embedded_ext);"
24802538
L"print(f'{embedded_ext.executed=}');"
2539+
// Same name, different initfuncs: the first one wins
2540+
L"spec = spec_from_loader('same_name', StaticExtensionImporter);"
2541+
L"a = create_static_module.create_module(spec);"
2542+
L"b = create_static_module.create_same_name_b(spec);"
2543+
L"print(f'{a.which=} {b.which=} {a is b=}');"
2544+
L"print(f'{create_static_module.initfunc_calls()=}');"
24812545
// Single-phase submodule whose init only knows its short name
24822546
L"import sp_pkg.sp_submod;"
24832547
L"print(sp_pkg.sp_submod);"
@@ -2510,7 +2574,10 @@ test_create_module_from_initfunc(void)
25102574
" return None\n"
25112575
" @staticmethod\n"
25122576
" def create_module(spec):\n"
2513-
" return create_static_module.create_module(spec)\n"
2577+
" mod = create_static_module.create_module(spec)\n"
2578+
" print(f'created {spec.name}: '\n"
2579+
" f'in sys.modules={spec.name in sys.modules}')\n"
2580+
" return mod\n"
25142581
" @staticmethod\n"
25152582
" def exec_module(module):\n"
25162583
" create_static_module.exec_module(module)\n"

0 commit comments

Comments
 (0)