Skip to content

extmod/zephyr_ble: Add Zephyr BLE host stack with RP2 port integration.#19

Draft
andrewleech wants to merge 139 commits intoreview/zephyr-ble-corefrom
pr/zephyr-ble-core
Draft

extmod/zephyr_ble: Add Zephyr BLE host stack with RP2 port integration.#19
andrewleech wants to merge 139 commits intoreview/zephyr-ble-corefrom
pr/zephyr-ble-core

Conversation

@andrewleech
Copy link
Copy Markdown
Owner

Summary

This adds the Zephyr BLE host as a third backend for modbluetooth, integrated as a port-agnostic extmod. The Zephyr host code is compiled against a HAL shim layer (extmod/zephyr_ble/hal/) that replaces Zephyr's kernel primitives (semaphores, work queues, timers, FIFOs, memory slabs) with MicroPython-compatible implementations. This means the Zephyr BLE host runs cooperatively on the main MicroPython task without requiring Zephyr RTOS itself.

This was motivated by limitations in the existing stacks — NimBLE doesn't have an active BLE pre-qualification and is missing some newer BLE features, while BTstack's MicroPython integration lacks pairing/bonding and L2CAP channel support. The Zephyr BLE host stack has active qualification, full feature coverage, and is under active development by multiple silicon vendors.

The RP2 port is the first integration, providing two build variants for Pico W and Pico 2 W:

  • zephyr_ble — cooperative polling from the main task (preferred)
  • zephyr_ble_freertos — HCI processing on a dedicated FreeRTOS task

Also included: a gap_unpair() API addition across all BLE backends, micropython-lib updates for aioble robustness, and bond key persistence via Python secret store callbacks.

flowchart TD
    subgraph "extmod/zephyr_ble (port-agnostic)"
        MOD[modbluetooth_zephyr.c] --> HAL[HAL shim layer]
        HAL --> WORK[work queues]
        HAL --> SEM[semaphores]
        HAL --> TIMER[timers]
        HAL --> FIFO[FIFOs]
        HAL --> H4[HCI H4 transport]
    end

    subgraph "Port integration (e.g. RP2)"
        HCI_DRV[mpzephyrport_rp2.c<br/>CYW43 HCI driver] --> H4
        POLL[main loop polling] --> WORK
    end

    subgraph "Zephyr BLE host (lib/zephyr)"
        HOST[hci_core / gatt / att / smp / l2cap]
    end

    HOST --> MOD
    HCI_DRV --> HOST
Loading

Testing

All 12 BLE multitests passing on Pico W and Pico 2 W (both zephyr_ble variant) with PYBD (NimBLE) as central:
ble_gap_advertise, ble_gap_connect, ble_characteristic, ble_gap_pair, ble_gap_pair_bond, ble_subscribe, ble_irq_calls, ble_gattc_discover_services, ble_l2cap, perf_gatt_notify, perf_l2cap, ble_gap_unpair.

Performance: ~24ms/notification (GATT), ~2184 B/s (L2CAP) on Pico W; ~25ms/notification, ~7956 B/s on Pico 2 W.

Not tested: zephyr_ble_freertos variant on Pico 2 W.

Trade-offs and Alternatives

The HAL shim layer is substantial (~3K lines) because it reimplements Zephyr kernel primitives. The alternative would be running actual Zephyr RTOS, but that would limit this to the Zephyr port only. The shim approach allows any MicroPython port with an HCI transport to use the Zephyr BLE host.

The lib/zephyr submodule adds the full Zephyr BLE host source. Only the host stack files are compiled — no kernel, no drivers, no board support. The submodule is pinned to a specific commit with two small patches (wrapper files for gatt.c and conn.c to expose static internals needed for clean teardown).

MTU is compile-time only (CONFIG_BT_L2CAP_TX_MTU=512); runtime ble.config(mtu=X) is not supported. ble_mtu.py test is skipped.

Generative AI

I used generative AI tools when creating this PR, but a human has checked the code and is responsible for the description above.

@andrewleech andrewleech force-pushed the pr/zephyr-ble-core branch 10 times, most recently from 577fa84 to eb4a81a Compare March 11, 2026 12:08
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Mar 11, 2026

Code size report:

Reference:  docs/esp32: Replace 'esptool.py' by 'esptool' in command line example. [e4920d6]
Comparison: extmod/zephyr_ble: Fix build issues from L2CAP absorption. [merge of 795da9c]
  mpy-cross:   -32 -0.008% 
   bare-arm:    +8 +0.014% 
minimal x86:   +42 +0.022% 
   unix x64:  +104 +0.012% standard[incl +32(data)]
      stm32: +7112 +1.803% PYBV10[incl +396(bss)]
      esp32:  +664 +0.038% ESP32_GENERIC
     mimxrt:   -16 -0.004% TEENSY40
        rp2:  +256 +0.028% RPI_PICO_W
       samd:  +100 +0.036% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:  +355 +0.078% VIRT_RV32

@andrewleech
Copy link
Copy Markdown
Owner Author

/review

@mpy-reviewer
Copy link
Copy Markdown

mpy-reviewer Bot commented Mar 13, 2026

Review failed. Retry with /review.

@andrewleech andrewleech force-pushed the pr/zephyr-ble-core branch 3 times, most recently from ffe840c to 673bd99 Compare March 16, 2026 06:34
This commit updates the documentation for the `re` library, officially
documenting non-capturing grouping rules (ie. "(?:...)").

The documentation mistakenly marked that feature as not supported, but
is is indeed supported in the current iteration of the regex library.

This closes micropython#18900.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Mar 17, 2026

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.45%. Comparing base (e4920d6) to head (c197acc).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@                   Coverage Diff                   @@
##           review/zephyr-ble-core      #19   +/-   ##
=======================================================
  Coverage                   98.45%   98.45%           
=======================================================
  Files                         175      175           
  Lines                       22635    22635           
=======================================================
  Hits                        22286    22286           
  Misses                        349      349           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

This reverts commit 046013a.

Looks like since the latest round of GitHub Actions updates, the
Cache LRU algorithm is working as designed again.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
Both the overall IRQ line and the per-channel IRQ, for good measure.

Otherwise, soft reset will remove the handler before the finaliser for the
DMA object(s) run and trigger IRQs if the channel is still active.

Closes micropython#18765

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
Small tweak to avoid changes in other targets' lockfiles from printing
warnings when building esp32 port.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
Not currently building, and too many versions to concurrently support.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
Also rename the prefix from can to pyb_can, in anticipation
of machine.CAN tests.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
Closes micropython#18922

Signed-off-by: Angus Gratton <angus@redyak.com.au>
The function arguments mean totally different things for Classic vs FDCAN
hardware, but the argument name wasn't particularly clear for either.

This commit shouldn't really change the binary firmware at all.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
Simplifies the pattern of an optional arg which can be a list of at
least a certain length, otherwise one is lazily initialised.

Modify pyb.CAN and ESP-NOW APIs to use the helper. Note this changes
the return type of pyb.CAN.recv() from tuple to list.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
jepler and others added 29 commits April 15, 2026 16:01
Otherwise, a very deeply nested regular expression like

    re.compile("(" * 65536)

can exhaust the host stack during the compile phase. This turns
that into a `RuntimeError: maximum recursion depth exceeded`
instead.

This crash was found via fuzzing.

Signed-off-by: Jeff Epler <jepler@unpythonic.net>
This commit shortens the amount of space taken by the DMA control fields
table, and explicitly marks it as `const`.

The DMA fields info table used a full-size QSTR index value, and 9 bits
of numeric information.  Given that the QSTR index could be converted
into a `qstr_short_t`, there is no fields spill outside a machine word
boundary - albeit with having 7 unused bits but there isn't much that
can be done for that.  The effective structure size for each entry is
halved, from 8 bytes down to 4.

Also, the structure is only read from, yet it was not marked as `const`.
Marking the structure as constant did not help reduce the final size but
at least correctly signals the compiler that no write accesses are
possible.

This shrinks the RPI_PICO/RPI_PICO_W build by 56 bytes, with a similar
size reduction for RPI_PICO2/RPI_PICO2_W.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit refactors shared ROMFS definitions appearing in some Arm
boards configuration, moving them into the QEMU port's Makefile.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit reserves a memory area to mount ROMFS partitions into, and
lets the port Makefile know the memory area details.

A 4 MiB segment is allocated at 0x8061_0000, which is inside the
emulated machine's DRAM segment.  The virt board requires the image
loaded in its DRAM segment to be contiguous, so the ROMFS segment is
placed right after the stack area.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit reserves a memory area to mount ROMFS partitions into, and
lets the port Makefile know the memory area details.

A 4 MiB segment is allocated at 0x8062_0000, which is inside the
emulated machine's DRAM segment.  The virt board requires the image
loaded in its DRAM segment to be contiguous, so the ROMFS segment is
placed right after the stack area.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit lets the MPS2_AN385 machine definition load a ROMFS
partition right at the beginning of the 16MiB PSRAM area.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit lets the SABRELITE machine definition load a ROMFS
partition at the very end of the RAM space.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit renames the "tests/frozen" directory into "tests/assets" to
make it more explicit that it does contain files that are needed for
other tests to function.

Right now there's only a single pre-compiled module being used for
miscellaneous tests, but it will soon hold ROMFS test data as well.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit adds a new test to make sure ROMFS files are mounted and
read correctly, to be run as part of the CI process.

The changes also include the source binary files that have been used to
create the pre-baked ROMFS partition image used in the test, along with
a Makefile to allow recreating said file.

The CI test ROMFS image is mounted only if no other ROMFS partition is
mounted in slot 0.  The specific test is executed only if there actually
is a ROMFS partition mounted and if the partition is identified as the
one used to run tests on.  This allows for user images to be mounted and
for a successful test run if that is the case.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
The esp32 port has the machine Counter and Encoder classes implemented in
Python, requiring a `machine.py` that extends the built-in machine module.

That previously used `__getattr__()` to delegate lookups to the built-in,
but that means any failed lookup raises an `AttributeError` instead of an
`ImportError`.  This means (among other things) that certain tests like
CAN and I2CTarget would fail because they couldn't skip the test correctly.

This commit improves the situation by using `from machine import *` instead
of `__getattr__()`, which puts all the built-in functions/classes/constants
directly in the `machine.py` global namespace.  That means an `ImportError`
is now correctly raised for attributes that don't exist.

Although this takes up a bit more RAM, it's now a lot faster to import from
the machine module: what used to take around 100us to lookup a name now
takes only 5us.

Signed-off-by: Damien George <damien@micropython.org>
Enable machine.CAN if CAN1 or CAN2 or both are enabled.

Signed-off-by: iabdalkader <i.abdalkader@gmail.com>
usbd_conf.c unconditionally includes shared/tinyusb/mp_usbd.h which
pulls in tusb.h when MICROPY_HW_ENABLE_USBDEV is set. The
BUILDING_MBOOT guard that disables TinyUSB comes after the include,
so mboot builds fail with missing tusb.h. Guard the include with
!BUILDING_MBOOT.

Also add forward declaration of mp_usbd_ll_init() next to the
MICROPY_HW_TINYUSB_LL_INIT macro in mpconfigboard_common.h, since
the function is used in an inline function in mp_usbd.h but only
declared in the port-specific usbd_conf.h.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Convert the mimxrt port from the old `MICROPY_EVENT_POLL_HOOK` macro to use
the new `mp_event_wait_xxx()` functions in conjunction with
`MICROPY_INTERNAL_WFE`.

This change should be functionally equivalent to the existing behaivour
because `mp_event_wait_ms()` and `mp_event_wait_indefinite()` are equal to
`mp_handle_pending(MP_HANDLE_PENDING_CALLBACKS_AND_EXCEPTIONS); __WFE()`,
which is what `MICROPY_EVENT_POLL_HOOK` was.

Signed-off-by: Damien George <damien@micropython.org>
Convert the samd port from the old `MICROPY_EVENT_POLL_HOOK` macro to use
the new `mp_event_wait_xxx()` functions in conjunction with
`MICROPY_INTERNAL_WFE`.

This change should be functionally equivalent to the existing behaivour
because `mp_event_wait_ms()` is equal to
`mp_handle_pending(MP_HANDLE_PENDING_CALLBACKS_AND_EXCEPTIONS); __WFE()`,
which is what `MICROPY_EVENT_POLL_HOOK` was.

Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Switch L2CAP CoC from recv+alloc_buf to the seg_recv API, which gives the
application per-PDU callbacks with manual credit control.  The old path
issued one credit per SDU, forcing the peer to wait for the full SDU to
be delivered before sending the next one.  With seg_recv, credits are
issued one per non-last PDU (allowing the peer to pipeline all K-frames
of a single SDU) and one credit per SDU from recvinto() (for the first
PDU of the next SDU), keeping at most one assembled SDU buffered.

Work around a Zephyr bug in l2cap_chan_seg_recv_rx_init() which leaves
rx.mps at zero for seg_recv channels (unlike l2cap_chan_rx_init for the
normal path), causing immediate channel disconnect on the first received
PDU.  Set rx.mps = BT_L2CAP_RX_MTU in l2cap_create_channel() and use
bt_l2cap_chan_give_credits() in accept/connect paths, matching the
pattern from Zephyr's credits_seg_recv test.

Also enable Data Length Extension (DLE) so the controller can negotiate
251-byte PDU payloads, reducing per-PDU overhead.

TX pipeline: allow up to L2CAP_SDU_BUF_COUNT-1 SDUs in flight concurrently
(tracked via tx_in_flight counter) rather than stalling after every send.

On nRF52840 dongle (PCA10059) with PYBD (NimBLE) as central:
  perf_l2cap.py before: ~2,184 B/s
  perf_l2cap.py after:  ~11,518 B/s  (5.3x improvement)
  All 11 BLE multitests pass.

Signed-off-by: Andrew Leech <andrew@alelec.net>
Replace single-SDU L2CAP accumulation buffer with a FIFO that
holds multiple SDUs.  Deep initial credit window (fills rx_buf)
allows the peer to pipeline SDUs without per-SDU credit
round-trips, which is critical for Z2Z throughput where each
credit round-trip costs 2+ connection intervals.

Add deferred L2CAP recv notification (rx_notify_pending) to
avoid re-entrancy between seg_recv_cb and Python IRQ handlers.
Each port's port_run_task must call flush_recv_notify() after
work_process completes.

Disable DLE auto-negotiation (CONFIG_BT_AUTO_DATA_LEN_UPDATE 0)
for CYW43 compatibility — CYW43 disconnects with "Instant
Passed" (0x16) when DLE is negotiated.

Add l2cap_status_cb TX kick via bt_tx_irq_raise() to unblock
queued SDUs when credits arrive.

Run codeformat.py on extmod files.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Add ZEPHYR_BLE_POLL_INTERVAL_MS define (default 128ms)
matching NimBLE convention. IRQ-driven ports use poll_now()
for immediate processing; this is a fallback for timer
housekeeping.

Change CONFIG_BT_AUTO_DATA_LEN_UPDATE to #ifndef guard so
ports with capable controllers can override via CFLAGS.

Move random data generation out of the timed window in
perf_l2cap.py and use getrandbits(8) for faster generation.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Cleanup unused functions, macros and debug helpers that were never called
or only conditionally compiled behind disabled feature flags. Removes dead
registry system, PSA crypto stubs, LIFO operations, and various unused
helper functions and inlines from kernel/device/config headers. Also
deletes gatt_pragma.h which is unreferenced.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Simplify HAL shim code:
- Remove competing weak mp_bluetooth_zephyr_hci_uart_wfi from sem.c
  (poll.c has the real one that calls port_run_task).
- Protect k_sem_take/give fast paths with MICROPY_PY_BLUETOOTH_ENTER/EXIT
  to prevent torn read-modify-write if HCI RX calls k_sem_give concurrently.
- Delegate k_lifo_get to k_queue_get (dequeue from head is identical
  for both FIFO and LIFO; only insertion differs).
- Replace runtime debug_enabled volatile in fifo.c with compile-time
  ZEPHYR_BLE_DEBUG guard matching sem.c and work.c patterns.
- Extract ensure_sys_work_q_init() helper for the lazy init check that
  was duplicated in k_work_submit, k_work_schedule, k_work_reschedule.
- Remove conflicting MICROPY_PY_BLUETOOTH_ENTER/EXIT no-op fallback from
  zephyr_ble_atomic.h (modbluetooth.h provides the correct default).

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Fix conn_handle sentinel and L2CAP credit flow:
- Fix 4 call sites comparing conn_handle against 0xFFFF instead of 0xFF.
  mp_bt_zephyr_conn_to_handle() returns uint8_t 0xFF which widens to
  0x00FF, so the 0xFFFF guards were dead code.
- Remove leftover L2CAP debug macro redefinition.
- Fire _IRQ_L2CAP_SEND_READY from status_cb when TX credits arrive.
  Previously only sent_cb fired SEND_READY, so Python never got unblocked
  after stalling on zero TX credits (e.g. NimBLE gives only 2 initial).
- Check tx.credits in stall detection, not just buffer pool count.
  Without this, Python queues SDUs faster than they can transmit when
  the peer gives few initial credits.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Address review findings:
- Rename global malloc/free to zephyr_ble_gatt_malloc/free with
  preprocessor redirect in gatt_wrapper.c. Add weak global aliases
  for -nostdlib builds where no libc malloc/free exists.
- Add __attribute__((weak)) to strtoul so libc wins when linked.
- Remove dead branch in k_current_get() (both paths returned same).
- Use named errno constants in crypto stubs (-EINVAL, -ENOSYS, -EIO).
- Rename ZEPHYR_BLE_POLL_MAX_TIMEOUT_MS to distinct per-file names.
- Delete stale README.md (CLAUDE.local.md is the living design doc).
- Zero ECC private key in bt_pub_key_hci_disrupted().
- Make ble_state volatile for callback safety.
- Guard le_dbg with ZEPHYR_BLE_DEBUG.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Add k_sem_reset to cooperative semaphore path (was only in FreeRTOS path).
Guard byte-order macros with #ifndef to avoid -Werror redefinition
conflicts with real zephyr/sys/byteorder.h on embedded ports.
Mark L2CAP callback variables as __attribute__((unused)) for non-debug
builds.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.