extmod/zephyr_ble: Add Zephyr BLE host stack with RP2 port integration.#19
Draft
andrewleech wants to merge 139 commits intoreview/zephyr-ble-corefrom
Draft
extmod/zephyr_ble: Add Zephyr BLE host stack with RP2 port integration.#19andrewleech wants to merge 139 commits intoreview/zephyr-ble-corefrom
andrewleech wants to merge 139 commits intoreview/zephyr-ble-corefrom
Conversation
577fa84 to
eb4a81a
Compare
|
Code size report: |
Owner
Author
|
/review |
|
Review failed. Retry with |
ffe840c to
673bd99
Compare
This commit updates the documentation for the `re` library, officially documenting non-capturing grouping rules (ie. "(?:...)"). The documentation mistakenly marked that feature as not supported, but is is indeed supported in the current iteration of the regex library. This closes micropython#18900. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## review/zephyr-ble-core #19 +/- ##
=======================================================
Coverage 98.45% 98.45%
=======================================================
Files 175 175
Lines 22635 22635
=======================================================
Hits 22286 22286
Misses 349 349 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
b3df7e5 to
c197acc
Compare
This reverts commit 046013a. Looks like since the latest round of GitHub Actions updates, the Cache LRU algorithm is working as designed again. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>
Both the overall IRQ line and the per-channel IRQ, for good measure. Otherwise, soft reset will remove the handler before the finaliser for the DMA object(s) run and trigger IRQs if the channel is still active. Closes micropython#18765 This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>
Small tweak to avoid changes in other targets' lockfiles from printing warnings when building esp32 port. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>
Not currently building, and too many versions to concurrently support. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>
This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>
Also rename the prefix from can to pyb_can, in anticipation of machine.CAN tests. Signed-off-by: Angus Gratton <angus@redyak.com.au>
Closes micropython#18922 Signed-off-by: Angus Gratton <angus@redyak.com.au>
The function arguments mean totally different things for Classic vs FDCAN hardware, but the argument name wasn't particularly clear for either. This commit shouldn't really change the binary firmware at all. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>
This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>
Simplifies the pattern of an optional arg which can be a list of at least a certain length, otherwise one is lazily initialised. Modify pyb.CAN and ESP-NOW APIs to use the helper. Note this changes the return type of pyb.CAN.recv() from tuple to list. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>
Otherwise, a very deeply nested regular expression like
re.compile("(" * 65536)
can exhaust the host stack during the compile phase. This turns
that into a `RuntimeError: maximum recursion depth exceeded`
instead.
This crash was found via fuzzing.
Signed-off-by: Jeff Epler <jepler@unpythonic.net>
This commit shortens the amount of space taken by the DMA control fields table, and explicitly marks it as `const`. The DMA fields info table used a full-size QSTR index value, and 9 bits of numeric information. Given that the QSTR index could be converted into a `qstr_short_t`, there is no fields spill outside a machine word boundary - albeit with having 7 unused bits but there isn't much that can be done for that. The effective structure size for each entry is halved, from 8 bytes down to 4. Also, the structure is only read from, yet it was not marked as `const`. Marking the structure as constant did not help reduce the final size but at least correctly signals the compiler that no write accesses are possible. This shrinks the RPI_PICO/RPI_PICO_W build by 56 bytes, with a similar size reduction for RPI_PICO2/RPI_PICO2_W. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit refactors shared ROMFS definitions appearing in some Arm boards configuration, moving them into the QEMU port's Makefile. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit reserves a memory area to mount ROMFS partitions into, and lets the port Makefile know the memory area details. A 4 MiB segment is allocated at 0x8061_0000, which is inside the emulated machine's DRAM segment. The virt board requires the image loaded in its DRAM segment to be contiguous, so the ROMFS segment is placed right after the stack area. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit reserves a memory area to mount ROMFS partitions into, and lets the port Makefile know the memory area details. A 4 MiB segment is allocated at 0x8062_0000, which is inside the emulated machine's DRAM segment. The virt board requires the image loaded in its DRAM segment to be contiguous, so the ROMFS segment is placed right after the stack area. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit lets the MPS2_AN385 machine definition load a ROMFS partition right at the beginning of the 16MiB PSRAM area. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit lets the SABRELITE machine definition load a ROMFS partition at the very end of the RAM space. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit renames the "tests/frozen" directory into "tests/assets" to make it more explicit that it does contain files that are needed for other tests to function. Right now there's only a single pre-compiled module being used for miscellaneous tests, but it will soon hold ROMFS test data as well. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit adds a new test to make sure ROMFS files are mounted and read correctly, to be run as part of the CI process. The changes also include the source binary files that have been used to create the pre-baked ROMFS partition image used in the test, along with a Makefile to allow recreating said file. The CI test ROMFS image is mounted only if no other ROMFS partition is mounted in slot 0. The specific test is executed only if there actually is a ROMFS partition mounted and if the partition is identified as the one used to run tests on. This allows for user images to be mounted and for a successful test run if that is the case. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
The esp32 port has the machine Counter and Encoder classes implemented in Python, requiring a `machine.py` that extends the built-in machine module. That previously used `__getattr__()` to delegate lookups to the built-in, but that means any failed lookup raises an `AttributeError` instead of an `ImportError`. This means (among other things) that certain tests like CAN and I2CTarget would fail because they couldn't skip the test correctly. This commit improves the situation by using `from machine import *` instead of `__getattr__()`, which puts all the built-in functions/classes/constants directly in the `machine.py` global namespace. That means an `ImportError` is now correctly raised for attributes that don't exist. Although this takes up a bit more RAM, it's now a lot faster to import from the machine module: what used to take around 100us to lookup a name now takes only 5us. Signed-off-by: Damien George <damien@micropython.org>
Enable machine.CAN if CAN1 or CAN2 or both are enabled. Signed-off-by: iabdalkader <i.abdalkader@gmail.com>
usbd_conf.c unconditionally includes shared/tinyusb/mp_usbd.h which pulls in tusb.h when MICROPY_HW_ENABLE_USBDEV is set. The BUILDING_MBOOT guard that disables TinyUSB comes after the include, so mboot builds fail with missing tusb.h. Guard the include with !BUILDING_MBOOT. Also add forward declaration of mp_usbd_ll_init() next to the MICROPY_HW_TINYUSB_LL_INIT macro in mpconfigboard_common.h, since the function is used in an inline function in mp_usbd.h but only declared in the port-specific usbd_conf.h. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Convert the mimxrt port from the old `MICROPY_EVENT_POLL_HOOK` macro to use the new `mp_event_wait_xxx()` functions in conjunction with `MICROPY_INTERNAL_WFE`. This change should be functionally equivalent to the existing behaivour because `mp_event_wait_ms()` and `mp_event_wait_indefinite()` are equal to `mp_handle_pending(MP_HANDLE_PENDING_CALLBACKS_AND_EXCEPTIONS); __WFE()`, which is what `MICROPY_EVENT_POLL_HOOK` was. Signed-off-by: Damien George <damien@micropython.org>
Convert the samd port from the old `MICROPY_EVENT_POLL_HOOK` macro to use the new `mp_event_wait_xxx()` functions in conjunction with `MICROPY_INTERNAL_WFE`. This change should be functionally equivalent to the existing behaivour because `mp_event_wait_ms()` is equal to `mp_handle_pending(MP_HANDLE_PENDING_CALLBACKS_AND_EXCEPTIONS); __WFE()`, which is what `MICROPY_EVENT_POLL_HOOK` was. Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Switch L2CAP CoC from recv+alloc_buf to the seg_recv API, which gives the application per-PDU callbacks with manual credit control. The old path issued one credit per SDU, forcing the peer to wait for the full SDU to be delivered before sending the next one. With seg_recv, credits are issued one per non-last PDU (allowing the peer to pipeline all K-frames of a single SDU) and one credit per SDU from recvinto() (for the first PDU of the next SDU), keeping at most one assembled SDU buffered. Work around a Zephyr bug in l2cap_chan_seg_recv_rx_init() which leaves rx.mps at zero for seg_recv channels (unlike l2cap_chan_rx_init for the normal path), causing immediate channel disconnect on the first received PDU. Set rx.mps = BT_L2CAP_RX_MTU in l2cap_create_channel() and use bt_l2cap_chan_give_credits() in accept/connect paths, matching the pattern from Zephyr's credits_seg_recv test. Also enable Data Length Extension (DLE) so the controller can negotiate 251-byte PDU payloads, reducing per-PDU overhead. TX pipeline: allow up to L2CAP_SDU_BUF_COUNT-1 SDUs in flight concurrently (tracked via tx_in_flight counter) rather than stalling after every send. On nRF52840 dongle (PCA10059) with PYBD (NimBLE) as central: perf_l2cap.py before: ~2,184 B/s perf_l2cap.py after: ~11,518 B/s (5.3x improvement) All 11 BLE multitests pass. Signed-off-by: Andrew Leech <andrew@alelec.net>
Replace single-SDU L2CAP accumulation buffer with a FIFO that holds multiple SDUs. Deep initial credit window (fills rx_buf) allows the peer to pipeline SDUs without per-SDU credit round-trips, which is critical for Z2Z throughput where each credit round-trip costs 2+ connection intervals. Add deferred L2CAP recv notification (rx_notify_pending) to avoid re-entrancy between seg_recv_cb and Python IRQ handlers. Each port's port_run_task must call flush_recv_notify() after work_process completes. Disable DLE auto-negotiation (CONFIG_BT_AUTO_DATA_LEN_UPDATE 0) for CYW43 compatibility — CYW43 disconnects with "Instant Passed" (0x16) when DLE is negotiated. Add l2cap_status_cb TX kick via bt_tx_irq_raise() to unblock queued SDUs when credits arrive. Run codeformat.py on extmod files. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Add ZEPHYR_BLE_POLL_INTERVAL_MS define (default 128ms) matching NimBLE convention. IRQ-driven ports use poll_now() for immediate processing; this is a fallback for timer housekeeping. Change CONFIG_BT_AUTO_DATA_LEN_UPDATE to #ifndef guard so ports with capable controllers can override via CFLAGS. Move random data generation out of the timed window in perf_l2cap.py and use getrandbits(8) for faster generation. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Cleanup unused functions, macros and debug helpers that were never called or only conditionally compiled behind disabled feature flags. Removes dead registry system, PSA crypto stubs, LIFO operations, and various unused helper functions and inlines from kernel/device/config headers. Also deletes gatt_pragma.h which is unreferenced. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Simplify HAL shim code: - Remove competing weak mp_bluetooth_zephyr_hci_uart_wfi from sem.c (poll.c has the real one that calls port_run_task). - Protect k_sem_take/give fast paths with MICROPY_PY_BLUETOOTH_ENTER/EXIT to prevent torn read-modify-write if HCI RX calls k_sem_give concurrently. - Delegate k_lifo_get to k_queue_get (dequeue from head is identical for both FIFO and LIFO; only insertion differs). - Replace runtime debug_enabled volatile in fifo.c with compile-time ZEPHYR_BLE_DEBUG guard matching sem.c and work.c patterns. - Extract ensure_sys_work_q_init() helper for the lazy init check that was duplicated in k_work_submit, k_work_schedule, k_work_reschedule. - Remove conflicting MICROPY_PY_BLUETOOTH_ENTER/EXIT no-op fallback from zephyr_ble_atomic.h (modbluetooth.h provides the correct default). Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Fix conn_handle sentinel and L2CAP credit flow: - Fix 4 call sites comparing conn_handle against 0xFFFF instead of 0xFF. mp_bt_zephyr_conn_to_handle() returns uint8_t 0xFF which widens to 0x00FF, so the 0xFFFF guards were dead code. - Remove leftover L2CAP debug macro redefinition. - Fire _IRQ_L2CAP_SEND_READY from status_cb when TX credits arrive. Previously only sent_cb fired SEND_READY, so Python never got unblocked after stalling on zero TX credits (e.g. NimBLE gives only 2 initial). - Check tx.credits in stall detection, not just buffer pool count. Without this, Python queues SDUs faster than they can transmit when the peer gives few initial credits. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Address review findings: - Rename global malloc/free to zephyr_ble_gatt_malloc/free with preprocessor redirect in gatt_wrapper.c. Add weak global aliases for -nostdlib builds where no libc malloc/free exists. - Add __attribute__((weak)) to strtoul so libc wins when linked. - Remove dead branch in k_current_get() (both paths returned same). - Use named errno constants in crypto stubs (-EINVAL, -ENOSYS, -EIO). - Rename ZEPHYR_BLE_POLL_MAX_TIMEOUT_MS to distinct per-file names. - Delete stale README.md (CLAUDE.local.md is the living design doc). - Zero ECC private key in bt_pub_key_hci_disrupted(). - Make ble_state volatile for callback safety. - Guard le_dbg with ZEPHYR_BLE_DEBUG. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
Add k_sem_reset to cooperative semaphore path (was only in FreeRTOS path). Guard byte-order macros with #ifndef to avoid -Werror redefinition conflicts with real zephyr/sys/byteorder.h on embedded ports. Mark L2CAP callback variables as __attribute__((unused)) for non-debug builds. Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
c197acc to
795da9c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This adds the Zephyr BLE host as a third backend for
modbluetooth, integrated as a port-agnostic extmod. The Zephyr host code is compiled against a HAL shim layer (extmod/zephyr_ble/hal/) that replaces Zephyr's kernel primitives (semaphores, work queues, timers, FIFOs, memory slabs) with MicroPython-compatible implementations. This means the Zephyr BLE host runs cooperatively on the main MicroPython task without requiring Zephyr RTOS itself.This was motivated by limitations in the existing stacks — NimBLE doesn't have an active BLE pre-qualification and is missing some newer BLE features, while BTstack's MicroPython integration lacks pairing/bonding and L2CAP channel support. The Zephyr BLE host stack has active qualification, full feature coverage, and is under active development by multiple silicon vendors.
The RP2 port is the first integration, providing two build variants for Pico W and Pico 2 W:
zephyr_ble— cooperative polling from the main task (preferred)zephyr_ble_freertos— HCI processing on a dedicated FreeRTOS taskAlso included: a
gap_unpair()API addition across all BLE backends, micropython-lib updates for aioble robustness, and bond key persistence via Python secret store callbacks.flowchart TD subgraph "extmod/zephyr_ble (port-agnostic)" MOD[modbluetooth_zephyr.c] --> HAL[HAL shim layer] HAL --> WORK[work queues] HAL --> SEM[semaphores] HAL --> TIMER[timers] HAL --> FIFO[FIFOs] HAL --> H4[HCI H4 transport] end subgraph "Port integration (e.g. RP2)" HCI_DRV[mpzephyrport_rp2.c<br/>CYW43 HCI driver] --> H4 POLL[main loop polling] --> WORK end subgraph "Zephyr BLE host (lib/zephyr)" HOST[hci_core / gatt / att / smp / l2cap] end HOST --> MOD HCI_DRV --> HOSTTesting
All 12 BLE multitests passing on Pico W and Pico 2 W (both
zephyr_blevariant) with PYBD (NimBLE) as central:ble_gap_advertise,ble_gap_connect,ble_characteristic,ble_gap_pair,ble_gap_pair_bond,ble_subscribe,ble_irq_calls,ble_gattc_discover_services,ble_l2cap,perf_gatt_notify,perf_l2cap,ble_gap_unpair.Performance: ~24ms/notification (GATT), ~2184 B/s (L2CAP) on Pico W; ~25ms/notification, ~7956 B/s on Pico 2 W.
Not tested:
zephyr_ble_freertosvariant on Pico 2 W.Trade-offs and Alternatives
The HAL shim layer is substantial (~3K lines) because it reimplements Zephyr kernel primitives. The alternative would be running actual Zephyr RTOS, but that would limit this to the Zephyr port only. The shim approach allows any MicroPython port with an HCI transport to use the Zephyr BLE host.
The
lib/zephyrsubmodule adds the full Zephyr BLE host source. Only the host stack files are compiled — no kernel, no drivers, no board support. The submodule is pinned to a specific commit with two small patches (wrapper files forgatt.candconn.cto expose static internals needed for clean teardown).MTU is compile-time only (
CONFIG_BT_L2CAP_TX_MTU=512); runtimeble.config(mtu=X)is not supported.ble_mtu.pytest is skipped.Generative AI
I used generative AI tools when creating this PR, but a human has checked the code and is responsible for the description above.