python: Fully destroy the intepreter on re-evaluation #6227

damz · 2025-09-24T19:38:18Z

I have seen a crash in the current code when importing a file that is not part of a package, like:

├── foo
│   ├── bar
│   │   └── baz.py
│   └── __init__.py
└── scad.py

scad.py:

import os
import sys

if os.getcwd() not in sys.path:
    sys.path.append(os.getcwd())

import foo.bar.baz

Would crash openscad on re-evaluation of the scad.py file.

The crashdump is in initPython, somewhere where it tries to empty the sys.modules dict:

#0  0x00007fa843b7e12e PyUnicode_AsEncodedString (libpython3.13.so.1.0 + 0x17e12e)
#1  0x0000000000d2f093 _Z10initPythonRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEd (/home/damz/Downloads/openscad/build/openscad + 0x92f0>
#2  0x0000000000f0b40b _ZN10MainWindow13parseDocumentEP15EditorInterface (/home/damz/Downloads/openscad/build/openscad + 0xb0b40b)

Proposed solution

Instead of trying to manually reinit the environment, which is messy and doesn't reverts all the side-effects that could have happened (like modifying sys.path above), tear down the interpreter with Py_FinalizeEx (available since Python 3.6, released in 2016).

Note that the patch is messy but trivial: it is literally just removing a bunch of code from initPython. Look at it with "Hide whitespace" enabled in the diff view (or git show -w in the CLI).

gsohler · 2025-09-24T20:05:22Z

would be awesome if you managed to really reinitizlize python without a segfault.
PS: most nasty libs are numpy, scipy and ctypes ...

damz · 2025-09-24T20:10:30Z

What I can say is that this seems to work in basic tests. I haven't tried numpy at all.

damz · 2025-09-24T20:14:49Z

Yeah, numpy fails on re-evaluation with:

RuntimeError: CPU dispatcher tracer already initlized

(which is numpy/numpy#28271)

damz · 2025-09-24T20:19:55Z

The C documentation for PyDict_Next clearly states that you cannot mutate a dictionary while iterating it:

The dictionary p should not be mutated during iteration. It is safe to modify the values of the keys as you iterate over the dictionary, but only so long as the set of keys does not change.

damz · 2025-09-24T20:27:04Z

Sub-interpreters would be an option, but these are Python 3.12+ (released 2023). Any thoughts about adding that restriction to more recent versions of Python?

t-paul · 2025-09-24T21:56:47Z

Sub-interpreters would be an option, but these are Python 3.12+ (released 2023). Any thoughts about adding that restriction to more recent versions of Python?

Yes, fine by me. It will make some builds a bit more difficult and we probably need to disable python for older distributions. But if that means we get clean environments, that's worth it in my view.

coryrc · 2025-10-09T21:41:25Z

I noticed some build environments are newer, but the appimage builder image (manually uploaded by Torsten to Docker, I'm not sure from where) has something older than Python 3.13 (because my draft PR failed only for appimage), so you'll need to update at least that. But, you know, perfect enemy of the good and all.

t-paul · 2025-10-09T22:00:25Z

Yep, correct. This will need to build a newer Python for the AppImage build-env. https://github.com/openscad/openscad/wiki/Build-Matrix - but if this will allow clean restarts, it's very much worth the extra effort in my view.

This builds a string that is not used anywhere, as far as I can tell.

damz · 2025-10-14T01:29:20Z

I reworked this based on sub-interpreters.

It turns out that sub-interpreters have been there forever, it is sub-interpreters with restrictions and independent GILs that are new to Python 3.12.

damz · 2025-10-14T02:41:03Z

Marked as draft because this makes the situation with numpy worse. Now it doesn't load at all, not even once.

t-paul · 2025-10-14T09:24:13Z

Right, a query for Py_mod_multiple_interpreters shows a lot of Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED.

So is actually forking a separate process the only option? I suppose not supporting numpy would be a huge drawback for the specific use in OpenSCAD. Looking at numpy/numpy#24755 and numpy/numpy#28271 there's ongoing work towards sub-interpreters but it also sounds like there's still some way to go.

MarcelKilgus · 2025-10-21T06:12:53Z

Just randomly stumbled upon this issue. I have done the same thing for work one or two years ago and I can attest that it is simply impossible to initialize Python cleanly a second time in the same process (because of popular 3rd party libraries, Python itself is fine). Basically numpy people say nobody uses sub-interpreters, so no need to support them. And nobody uses sub-interpreters because numpy doesn't work in them, Catch 22.

With Python 3.14 sub-interpreters came into the focus once more as they are now available within Python itself (previously one had to use the C API). Maybe that finally moves the goalpost so far that numpy has to get its act together 🤷‍♂️

t-paul · 2025-10-21T08:23:04Z

I did a simple test and forking and running python against 2 different numpy versions (one installed via Debian, the other via venv) and that worked. Now the question would be what other problems will that drag in. It certainly makes other things more difficult, but as the forked process has all the other stuff too, maybe it's just a matter of getting the result back into the main process?

gsohler · 2025-10-21T08:54:04Z

I like the forking option as it removes the need to do python-re-initialization . I believe we can even use the SAME memory for parent and child and just need to ensure proper synchronization of data. But I am not sure how that works with the Windows platform. Does Windows API support "forking" ?

…

On Tue, Oct 21, 2025 at 10:23 AM Torsten Paul ***@***.***> wrote: *t-paul* left a comment (openscad/openscad#6227) <#6227 (comment)> I did a simple test and forking and running python against 2 different numpy versions (one installed via Debian, the other via venv) and that worked. Now the question would be what other problems will that drag in. It certainly makes other things more difficult, but as the forked process has all the other stuff too, maybe it's just a matter of getting the result back into the main process? — Reply to this email directly, view it on GitHub <#6227 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACCO4MVMNQHREHQRRJZP4E33YXUP5AVCNFSM6AAAAACHND74Z2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTIMRVGM3TOMBQGE> . You are receiving this because you commented.Message ID: ***@***.***>

damz force-pushed the pr/python-crash branch from c8579cb to f0056c2 Compare September 24, 2025 19:51

kintel added the Comp: Python label Sep 24, 2025

damz added 2 commits October 13, 2025 18:26

python: remove deadcode

bb7aaab

This builds a string that is not used anywhere, as far as I can tell.

python: Use sub-interpreters to isolate script executions

54ef1e1

damz force-pushed the pr/python-crash branch from f0056c2 to 54ef1e1 Compare October 14, 2025 01:27

damz marked this pull request as draft October 14, 2025 01:38

Uh oh!

python: Fully destroy the intepreter on re-evaluation #6227

Are you sure you want to change the base?

python: Fully destroy the intepreter on re-evaluation #6227

Uh oh!

Conversation

damz commented Sep 24, 2025

Proposed solution

Uh oh!

gsohler commented Sep 24, 2025

Uh oh!

damz commented Sep 24, 2025

Uh oh!

damz commented Sep 24, 2025

Uh oh!

damz commented Sep 24, 2025

Uh oh!

damz commented Sep 24, 2025

Uh oh!

t-paul commented Sep 24, 2025

Uh oh!

coryrc commented Oct 9, 2025

Uh oh!

t-paul commented Oct 9, 2025

Uh oh!

damz commented Oct 14, 2025

Uh oh!

damz commented Oct 14, 2025

Uh oh!

t-paul commented Oct 14, 2025

Uh oh!

MarcelKilgus commented Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

t-paul commented Oct 21, 2025

Uh oh!

gsohler commented Oct 21, 2025 via email

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

MarcelKilgus commented Oct 21, 2025 •

edited

Loading