tests: various improvements for Octoprobe (WIP) #17782

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Draft

dpgeorge wants to merge 17 commits into micropython:master from dpgeorge:tests-improvements-for-octoprobe

+123 −31

Member

dpgeorge commented Jul 29, 2025 •

edited

Loading

Summary

This is a collection of various small improvements to the test suite to get Octoprobe to pass more of the tests.

Once fully working, these changes will be made into separate PRs.

Testing

To be tested by Octoprobe: https://reports.octoprobe.org/

Test report prior to this PR: https://reports.octoprobe.org/github_selfhosted_testrun_194/octoprobe_summary_report.html

Test report with this PR: https://reports.octoprobe.org/github_selfhosted_testrun_199/octoprobe_summary_report.html

dpgeorge added the tests label

codecov bot commented Jul 29, 2025 •

edited

Loading

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.38%. Comparing base (69ead7d) to head (8920560).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master   #17782   +/-   ##
=======================================
  Coverage   98.38%   98.38%           
=======================================
  Files         171      171           
  Lines       22283    22283           
=======================================
  Hits        21924    21924           
  Misses        359      359

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

dpgeorge force-pushed the tests-improvements-for-octoprobe branch from 18ad8c0 to 5958518 Compare

July 31, 2025 02:27

hmaerki mentioned this pull request

LOLIN_C3_MINI / RUN-TESTS_STANDARD / invalid pin octoprobe/testbed_micropython#34

Open

github-actions bot commented Jul 31, 2025 •

edited

Loading

Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

dpgeorge force-pushed the tests-improvements-for-octoprobe branch from c399b93 to ca374fa Compare

July 31, 2025 13:16

Contributor

hmaerki commented Jul 31, 2025 •

edited

Loading

@dpgeorge: Two easy to reproduce bugs which could be added to ths PR

dpgeorge force-pushed the tests-improvements-for-octoprobe branch from ca374fa to aee37bb Compare

August 1, 2025 02:45

dpgeorge added 17 commits

August 1, 2025 14:04


          tests/extmod/machine_spi_rate.py: Support samd with default SPI.

05c93c9

Signed-off-by: Damien George <damien@micropython.org>


          tests/extmod: Support ESP32-C3 targets.

e90886c

Some boards define ESP32C3, some define ESP32-C3.

Probably it should just use default pins, eg SPI(1).

Signed-off-by: Damien George <damien@micropython.org>


          tests/extmod/machine_spi_rate.py: Use default pins on ESP32.

e9f18a8

Signed-off-by: Damien George <damien@micropython.org>


          tests/run-tests.py: Run tests-with-regex-output as normal tests.

9dd250d

Some tests (currently given by the `special_tests` list) have output which
must be mached via a regex, because it can change from run to run (eg the
address of an object is printed).  These tests are currently classified as
`is_special` in the test runner, which means they get special treatment.
In particular they don't set the emitter as specified by `args.emit`.  That
means these tests do not run via .mpy or using the native emitter, even if
those options are given on the command line.

This commit fixes that by considering `is_special` as different to
`tests_with_regex_output`.  The former is used for things like target
feature detection (which are not really tests) and when extra command line
options need to be passed to the unix micropython executable.  The latter
(now called `tests_with_regex_output`) are specifically for tests that have
output to be matched via regex.

The `thread_exc2.py` test now needs to be excluded when running using the
native emitter, because the native emitter doesn't print traceback info.
And the `sys_settrace_cov.py` test needs to be excluded because set-trace
output is different with the native emitter.

Signed-off-by: Damien George <damien@micropython.org>


          esp8266/main: Decrease usable stack by 64 bytes.

e15347b

To get `micropython/extreme_exc.py` to pass when run via-mpy.

Signed-off-by: Damien George <damien@micropython.org>


          tests/run-perfbench.py: Skip bm_hexiom on small targets.

da2a4df

Signed-off-by: Damien George <damien@micropython.org>


          tests/net_inet/tls_num_errors.py: Support running on unix port.

9628ebf

Signed-off-by: Damien George <damien@micropython.org>


          tests/net_inet: Skip tests on axTLS when necessary.

ed330fe

Signed-off-by: Damien George <damien@micropython.org>


          tests/perf_bench: Skip tests when vfs doesn't exist.

ac5d831

Eg for ESP8266-512K.

Signed-off-by: Damien George <damien@micropython.org>


          tests/float/string_format_modulo2_intbig.py: Get working on esp8266.

e6a10e7

In --via-mpy --emit native mode.

Signed-off-by: Damien George <damien@micropython.org>


          tests/extmod/vfs_blockdev_invalid.py: Skip if no RAM to mkfs.

911af74

On ESP8266 this test can fail with --via-mpy.

Signed-off-by: Damien George <damien@micropython.org>


          tests/net_hosted/ssl_verify_callback.py: Skip if necessary.

5454cad

Signed-off-by: Damien George <damien@micropython.org>


          tests/float/string_format_modulo2_intbig.py: Make CPython compatible.

839e6fc

Signed-off-by: Damien George <damien@micropython.org>


          py/persistentcode: Only select hard-fp arch if compiler uses hard ABI.

347ac41

Because RP2350 uses float-abi=softfp!  Maybe we should change that somehow?

Could also use `defined(__SOFTFP__)` to detect soft (all software fp) or
softfp (hardware for calculations but soft/integer for passing arguments).

Signed-off-by: Damien George <damien@micropython.org>


          tests/run-perfbench.py: Skip tests that have a MemoryError.

fa9efe4

Signed-off-by: Damien George <damien@micropython.org>


          tests/run-tests.py: Skip various tests on small SAMD targets.

20b1309

Tested on ADAFRUIT_ITSYBITSY_M0_EXPRESS.

Signed-off-by: Damien George <damien@micropython.org>


          tests/run-tests.py: Add extra logic to skip thumb2 fp tests.

On RP2350 these must now be skipped with via-mpy, due to mpy-cross not
knowing the correct architecture.

Signed-off-by: Damien George <damien@micropython.org>

dpgeorge force-pushed the tests-improvements-for-octoprobe branch from aee37bb to 8920560 Compare

August 1, 2025 04:05

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

tests