I'm confused why the stack changes are needed here:

1. For the main unix thread, its allocated stack is probably megabytes long (and might even automatically grow), so MICROPY_STACK_CHECK_MARGIN shouldn't make any difference for the main thread.
2. ports/unix/mpthreadport.c has:

    // ensure there is enough stack to include a stack-overflow margin
    if (*stack_size < 2 * THREAD_STACK_OVERFLOW_MARGIN) {
        *stack_size = 2 * THREAD_STACK_OVERFLOW_MARGIN;
    }                                                        

which is 16k, so I don't think the change to tests/thread/thread_stacksize1.py is needed (is 16k really not enough?).
3. ports/unix/mpthreadport.c has:

    // adjust stack_size to provide room to recover from hitting the limit
    *stack_size -= THREAD_STACK_OVERFLOW_MARGIN;

and that's already 8k, so I don't think MICROPY_STACK_CHECK_MARGIN is needed.

It's a bit hard to follow the logic of the stack limit for threads. So maybe we should either get rid of THREAD_STACK_OVERFLOW_MARGIN and use MICROPY_STACK_CHECK_MARGIN instead. Or just increase THREAD_STACK_OVERFLOW_MARGIN if it really needs to be increased for sanitizer builds.

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (master@255d74b). Learn more about missing BASE report.

@@            Coverage Diff            @@
##             master   #17735   +/-   ##
=========================================
  Coverage          ?   98.38%           
=========================================
  Files             ?      171           
  Lines             ?    22283           
  Branches          ?        0           
=========================================
  Hits              ?    21924           
  Misses            ?      359           
  Partials          ?        0           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

I'm confused why the stack changes are needed here:

This PR with the cherry-picked commit also hit "RuntimeError: maximum recursion depth exceeded" when testing longlong+UBSan, so you're right that it's not the right fix for this case.

It's a bit hard to follow the logic of the stack limit for threads. So maybe we should either get rid of THREAD_STACK_OVERFLOW_MARGIN and use MICROPY_STACK_CHECK_MARGIN instead. Or just increase THREAD_STACK_OVERFLOW_MARGIN if it really needs to be increased for sanitizer builds.

Seems reasonable, I'll have a look along these lines.

This PR with the cherry-picked commit also hit "RuntimeError: maximum recursion depth exceeded" when testing longlong+UBSan

So maybe there just needs to be more stack overall, rather than a change in stack margin? Maybe ports/unix/mpthreadport.c:mp_thread_create just needs to have a larger minimum stack size for sanitizer builds?