I faced the exact same issue when implementing dictionaries. There was no way to refer to the virtual machine from within the Eq trait in that case. You need to be able to call python code, for this you require a virtualmachine. Next to that, there is another issue, the python code can fail with an exception. How do you handle this exception?

The documentation for weakref states:
"Exceptions raised by the callback will be noted on the standard error output, but cannot be propagated; they are handled in exactly the same way as exceptions raised from an object’s __del__() method.". So exceptions isn't a problem as they can't be propagated out of the drop method.

I have gone with approach 3 of, putting objects that are about to be deleted into a queue and processing them after every frame and del statement. This is the simplest approach to get working.

Codecov Report

Merging #488 into master will decrease coverage by 4.16%.
The diff coverage is 58.79%.

@@            Coverage Diff             @@
##           master     #488      +/-   ##
==========================================
- Coverage   57.38%   53.21%   -4.17%     
==========================================
  Files          67       68       +1     
  Lines       12300    13167     +867     
  Branches     2906     3198     +292     
==========================================
- Hits         7058     7007      -51     
- Misses       3457     4349     +892     
- Partials     1785     1811      +26

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fcea845...350bc30. Read the comment docs.

The documentation for weakref states:
"Exceptions raised by the callback will be noted on the standard error output, but cannot be propagated; they are handled in exactly the same way as exceptions raised from an object’s del() method.". So exceptions isn't a problem as they can't be propagated out of the drop method.

I think it is fair to deviate here from cpython. This is pretty unacceptable to silently take up an error. At least the interpreter should crash in some clean way. Maybe by propagating the exception to top loop?

For exceptions like SystemExit and KeyboardInterrupt they need to be propagated, so that they can tell the vm to shutdown.

A consequence of propagating an exception from __del__ is that any exception can come from any function, making it difficult to reason about what exceptions a method could raise. Another problem in CPython is that the tracebacks holds references to all frames therefore all local variables including the deleted reference. Also it makes observing when garbage collection implementation calls __del__ easier, making it possibly harder to change it's behavior in the future.

Shutting the whole vm down due to an exception in __del__ is a significant deviation as it will mean Python programs will be less resilient to errors. Normally catching exceptions within a main loop protects the program from crashing, and can recover from normal exceptions.

A correction is that sys.excepthook is actually what is called to handle the uncaught exception from __del__, allowing the ability to override this handling. Note it's not passed a traceback due to the problem I mentioned earlier about holding references to the deleted object.

This is now a bit outdated with the recent changes to the payload, so I will close this, and reimplement it in a series of a couple of PRs, the first of which is #636.