Skip to content

Customize generation and compiling of frozen modules #425

@1ndahous3

Description

@1ndahous3

First of all, thanks for the cool project, I finally embedded python in my application using CMake!

I embed python in a static and isolated configuration, without using modules on the filesystem. After setting all compile time and runtime defines I got stuck with an error on initialization (same as in #339):

Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
...
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00002008 (most recent call first):
  <no Python frame>

While in most cases this means problems with the path to the core "*.py" libraries, in my case it means that the "encodings" module is not frozen (in memory).

After a couple of days of fighting and deep research I came up with a solution and figured out what was going on.

All frozen python modules must be generated in two places:

  1. cpython/Python/frozen.c
  2. deepfreeze/deepfreeze.c and frozen_modules/*.h headers (for frozen.c or custom lists like struct _frozen *_PyImport_FrozenModules[]).

The files from the second case are generated in this project by hardcoded statements in python-cmake-buildsystem/blob/master/cmake/libpython/CMakeLists.txt (COMMAND _freeze_importlib, LIBPYTHON_FROZEN_SOURCES, COMMAND _bootstrap_python, LIBPYTHON_DEEPFREEZE_SOURCES, ...).

But to add frozen modules to the global lists in frozen.c the script cpython/Tools/build/freeze_modules.py is used.

I also found that the "encodings" module is non-frozen not by accident and can be restored (see: https://github.com/python/cpython/blob/8ac7613dc8b8f82253d7c0e2b6ef6ed703a0a1ee/Tools/build/freeze_modules.py#L56).

So, putting it all together, I created a helper CMakeLists.txt in my project that does all the magic for my static runtime, and everything works: CMakeLists.txt

I think it would be great to implement such patches and generation here, in python-cmake-buildsystem project to not patch the project that patches CPython :D

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions