[SIGABRT] FFI callback crash when using multiprocessing (PROCESS mode) on Linux

[18Savage-maker]

Description

When running LiveKit agents on Linux with the default JobExecutorType.PROCESS mode, the agent process crashes with SIGABRT (exit code -6) shortly after connecting to a room and receiving audio tracks.

Environment

• Platform: Linux (Docker container, python:3.10-slim)
• livekit version: 1.1.5
• livekit-agents version: 1.5.2
• rtc-version: 1.1.5

Crash Details

The crash occurs in _ffi_client.py FFI callback:

Fatal Python error: Aborted

Thread 0x... (most recent call first):
File "livekit/rtc/_ffi_client.py", line 198 in to_python_level
File "livekit/rtc/_ffi_client.py", line 164 in ffi_event_callback

Thread 0x... (most recent call first):
File "livekit/rtc/_ffi_client.py", line 104 in put
File "livekit/rtc/_ffi_client.py", line 190 in ffi_event_callback

Multiple threads calling ffi_event_callback → loop.call_soon_threadsafe → _write_to_self concurrently, causing SIGABRT.

Root Cause

The ffi_event_callback is a ctypes CFUNCTYPE callback invoked from Rust FFI multiple threads. When running in PROCESS mode with forkserver multiprocessing context:

1. Rust FFI calls the callback from multiple threads
2. Python's ctypes callback handling is not thread-safe for concurrent calls
3. loop.call_soon_threadsafe triggers race condition → SIGABRT

Workaround

Setting job_executor_type=JobExecutorType.THREAD reduces crash probability, but doesn't fully resolve the issue.

Suggested Fix

Add thread synchronization (lock) to ffi_event_callback in _ffi_client.py, or use a single-threaded event queue pattern for FFI callbacks.

Reproduction

from livekit.agents import Agent, AgentSession, JobContext, WorkerOptions, cli

async def entrypoint(ctx: JobContext):
    session = AgentSession()
    await session.start(agent=Agent(), room=ctx.room)

cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))

Run in Docker on Linux with shm_size: '2gb' and init: true. Join a room with 2 participants publishing audio tracks → crash within 2 seconds.

[18Savage-maker]

Additional Crash Stack Trace

The crash specifically occurs in protobuf's enum_type_wrapper:

File "google/protobuf/internal/enum_type_wrapper.py", line 97 in getattr
File "livekit/rtc/_ffi_client.py", line 201 in to_python_level
File "livekit/rtc/_ffi_client.py", line 165 in ffi_event_callback

The to_python_level function accesses proto_ffi.LogLevel enum from multiple threads, causing race condition in protobuf's lazy enum loading (__getattr__).

Proposed Fix

Add thread lock to protect the entire ffi_event_callback:

import threading
_ffi_callback_lock = threading.Lock()

@ctypes.CFUNCTYPE(...)
def ffi_event_callback(data_ptr, data_len):
    with _ffi_callback_lock:
        event_data = ctypes.string_at(data_ptr, int(data_len))
        event = proto_ffi.FfiEvent()
        event.ParseFromString(event_data)
        ...