That made an otherwise simple logging problem surprisingly difficult. A line from Linux could be interrupted halfway through by a line from RISC-V. Boot messages were sometimes missing, shell escape sequences leaked into the output, and the result depended on which core reached the UART first.
The solution was not a more complicated UART lock. I gave Linux exclusive ownership of the UART and turned the small core’s output into messages:
RISC-V printf -> ring buffer -> RPMsg -> Linux driver -> printk -> UART
This article describes the complete path, including the two startup races that only appeared during cold boot.
The initial design effectively looked like this:
Linux printk ---------------------> UART
RISC-V printf --------------------> UART
The two cores do not share a scheduler, and an ordinary mutex on one core cannot serialize code running on the other. A hardware spinlock or a custom shared-memory lock could protect individual writes, but it would introduce several new questions:
The cleaner rule is:
Linux owns the debug UART. RISC-V produces log records, not UART transactions.
This moves serialization to one place and keeps all board-level UART configuration under Linux control.
I reused the board’s existing RPMsg transport. Endpoint 1 is reserved for RISC-V logs, while other endpoints remain available for their original console or application traffic.
RISC-V / FreeRTOS Linux
printf / prom_printk
|
v
_write()
|
v
rtos_console_write()
|
v
8 KiB circular buffer
|
v
log forwarding task
|
v
riscv_log_send()
|
+------ RPMsg endpoint 1 ------> receive callback
|
v
bounded skb queue
|
v
worker
|
v
printk("[RISCV] ...")
|
v
UART
There are two important boundaries in this diagram. The circular buffer separates log producers from RPMsg availability on the small core. The Linux queue and worker separate the RPMsg receive callback from text formatting and console output.
The firmware already had a newlib-style _write() hook used by printf(). Instead of sending bytes to a UART, _write() now passes them to rtos_console_write(), which appends them to an 8 KiB circular buffer.
The same path also handles low-level firmware printing. I added relay implementations of prom_printk(), prom_putchar(), and prom_putstr() so code below the C library does not bypass the new ownership rule.
Conceptually, every producer now ends at the same function:
int rtos_console_write(char *data, int len)
{
int written = circ_buffer_write(&log_buffer, data, len);
if (log_task_ready)
wake_log_task();
return written;
}
The real implementation also chooses the interrupt-safe semaphore operation when called from interrupt context.
A dedicated FreeRTOS task drains the buffer in RPMsg-sized chunks. Application code therefore continues to use normal calls such as:
printf("sensor task started\n");
There is no need to call riscv_log_send() from each feature. Keeping that transport detail below printf() makes the mechanism easy to adopt and difficult to bypass accidentally.
The RISC-V firmware build also excludes its direct serial implementation. Redirection is only reliable if no second code path can silently reclaim the UART.
Buffering solves a scheduling problem, but it does not by itself solve startup ordering.
The small core may start before Linux has probed the RPMsg device and installed its receive callback. If the forwarding task drains immediately, RPMsg can accept or discard data before a Linux consumer exists. The most valuable boot messages then disappear.
I added a tiny ready protocol:
RISC-V creates endpoint 1 and buffers logs
|
v
Linux probes endpoint 1 and installs callback
|
v
Linux sends "RISC-V-LOG-READY"
|
v
RISC-V marks host_ready = true
|
v
RISC-V drains all buffered startup logs
Until this exact message arrives, riscv_log_send() refuses to transmit and the log task leaves the circular buffer intact. Once the handshake is received, the task wakes and flushes both early and current output in order.
The marker is deliberately a protocol message instead of a fixed delay. A 100 ms delay may work on one image and fail on another; “the receiver has installed its callback” is the state the sender actually needs to know.
The first handshake implementation still failed intermittently during a cold boot. Linux reported that the RPMsg host was online, but it never created the named channels. Endpoint 1 did not appear, so Linux never had a device on which to send the ready message.
The problem was in endpoint announcement. The helper attempted rpmsg_ns_announce() repeatedly while it returned RL_NOT_READY. It performed 65,535 tight retries, which sounds generous. In practice, all those iterations could finish before Linux brought up the remote virtio/RPMsg link.
This is a useful embedded-systems lesson:
many immediate retries != waiting for a state transition
The fix was to wait on the condition represented by the transport itself:
rpmsg_lite_wait_for_link_up(rpmsg_instance);
ret = rpmsg_ns_announce(rpmsg_instance,
rpmsg_endpoint,
endpoint_name,
0);
After the link-up wait returns, one announcement is enough. Linux then creates the ingenic_rpmsg channels for endpoints 1, 2, and 3 consistently.
This race was easy to misdiagnose because the remote processor itself was already shown as running. That state only means the firmware is executing. It does not prove that the RPMsg link is up, name service completed, a Linux driver bound, or the log handshake succeeded.
On Linux, I extended the existing Ingenic RPMsg driver so that one configured endpoint is consumed by the kernel log relay. Other endpoints retain the original character-device behavior.
The receive callback executes in a context where blocking console work is undesirable. It therefore performs only bounded operations:
sk_buff with GFP_ATOMIC.The worker drains the queue later and emits the text with pr_info().
I also bounded both memory and output behavior:
. instead of reaching the terminal.Every relayed line receives an explicit prefix:
[RISCV] riscv start
[RISCV] RISC-V printer log relay ready
Linux’s own messages keep their normal format. The prefix makes the origin visible even though both cores now share the same final UART.
Endpoint 1 becomes a single-consumer channel once the kernel relay is enabled, so userspace opening that character device receives -EBUSY. This prevents a userspace reader and the kernel worker from competing for the same log records.
On this board, U-Boot may start an RISC-V image before Linux boots. The Linux remoteproc state can therefore already say running, and the firmware name can already be console.elf.
That still does not guarantee that the running bytes match the firmware installed in the current root filesystem. An older image with the same filename may lack the ready-handshake implementation and wait forever or lose early logs.
The init script consequently reloads console.elf once per Linux boot:
first start in this Linux boot
-> stop an already-running image if necessary
-> select console.elf
-> start remoteproc
-> create a volatile marker under /run
later duplicate start
-> firmware is running, endpoint exists, marker exists
-> do nothing
The marker prevents unnecessary resets during the same boot, while its location under /run makes it disappear on reboot. It records that the Linux init script performed its reload; it is not a substitute for the RPMsg ready handshake.
This small distinction is important. A process marker, remoteproc state, link state, channel state, and application readiness are five different facts.
I tested the feature as a chain of observable states rather than relying on one “running” flag.
First, confirm that the intended firmware is running:
cat /sys/class/remoteproc/remoteproc0/firmware
cat /sys/class/remoteproc/remoteproc0/state
Expected values are console.elf and running.
Next, confirm that name service produced the endpoints:
ls -l /sys/class/ingenic_rpmsg/
The result should include rpmsg-1, rpmsg-2, and rpmsg-3. Endpoint 1 proves that Linux saw the log-channel announcement.
Finally, inspect the kernel ring buffer:
dmesg | grep -E 'forwarding RISC-V|ready handshake|\[RISCV\]'
A successful cold boot contains all three stages:
forwarding RISC-V endpoint 1 to kernel console
RISC-V log ready handshake sent
[RISCV] riscv start
[RISCV] RISC-V printer log relay ready
The BusyBox version of dmesg on this target does not support dmesg -w, so repeated snapshots or the serial console itself are more portable ways to watch new output.
For an active test, any RISC-V task can call printf("printer relay test\n"). Seeing [RISCV] printer relay test in the Linux log confirms the full path from the C library, through the FreeRTOS buffer and RPMsg, to the Linux worker and UART.
The final code is relatively small, but the design carries several reusable lessons.
Prefer ownership over distributed locking. When two processors want one peripheral, choose one owner and communicate with it. The resulting failure modes are much easier to reason about.
Readiness is protocol state. Processor running, link up, channel announced, callback installed, and application ready are not synonyms. If a sender depends on one of them, represent it explicitly.
Retry based on time or state, not iteration count. A huge tight loop may take less time than the device on the other side needs for one scheduling interval.
Keep transport callbacks bounded. Queue the data in the callback and do formatting, console output, and other potentially slow work in task or process context.
Buffer at asynchronous boundaries. The small-core ring buffer protects early logs and decouples producers from link timing. The Linux queue decouples RPMsg delivery from the console.
Make observability part of the design. The [RISCV] prefix, ready-handshake messages, endpoint sysfs entries, and drop warnings turned a multi-stage boot sequence into something that can be inspected one invariant at a time.
The most important outcome is not merely that two cores can print. It is that there is now one deterministic logging path, one UART owner, and a startup protocol that explains exactly when logs are safe to send.
]]>Show live waveforms on a phone while recording, download the complete recording at the same time, and deliver whatever remains after the user presses Stop.
The device already supported the two endpoints independently. It could stream samples for a live chart, and it could generate a binary file and transfer that file later. Combining them changed the problem.
The file was no longer static while it was being downloaded. The producer could append data faster than BLE could transmit it, and stopping acquisition did not mean that the consumer had received the end of the file. A correct design therefore had to coordinate three timelines:
sensor acquisition: produces new samples
file writer: appends encoded records
BLE synchronizer: sends and acknowledges file bytes
This article explains the mental model and protocol that made the system reliable.
It is tempting to treat every byte sent over BLE as one stream, but a live waveform and a complete recording have different contracts.
A live preview is optimized for immediacy:
A recording transfer is optimized for completeness:
These paths may share a BLE connection, but they should not share semantics.
┌─> live preview queue ─> waveform notifications
sensor nodes ─> acquire ─┤
└─> append-only file ──> file synchronization
The live path answers, “What is happening now?” The file path answers, “Can I reconstruct the entire session exactly?”
Downloading a completed file is straightforward:
A growing file breaks the first assumption. Reaching the current end of the file only means that the reader has caught up for now. More data may appear on the next acquisition cycle.
The transfer loop therefore needs to distinguish two cases:
reader offset == currently available bytes
+ writer still active -> wait for growth and continue
+ writer finalized -> this is the real end of the file
I modeled the file as an append-only log. The writer owns the production offset, while the synchronizer owns the acknowledged offset:
0 produced_offset
|================ acknowledged ================|---- pending ----|
^
acknowledged_offset
The sender never needs to load the entire recording into memory. It reads a bounded chunk from the acknowledged or next-send offset, transmits it, and retains only enough state to retry unacknowledged data.
If buffered standard I/O is used, the writer must flush application buffers before expecting another file descriptor to observe newly appended bytes. fsync() is a separate durability decision: it protects against storage or power failure, but it should not be confused with protocol-level delivery.
Packets can be duplicated, delayed, or lost. A transfer protocol becomes much easier to reason about when progress is represented by a byte offset rather than by a count of packets sent.
A data message can contain:
session_id | file_offset | payload_length | payload
The phone responds with a cumulative acknowledgment:
ACK next_expected_offset
If the phone acknowledges offset 65536, it is confirming that every byte before 65536 is present. The device can discard any buffered chunks fully below that offset.
Cumulative acknowledgments provide useful properties:
The key invariant is:
0 <= acknowledged_offset <= sent_offset <= produced_offset
Violating this invariant indicates a state-machine or bookkeeping bug, not a radio-quality problem.
Suppose the recording grows at rate R_p and BLE delivers file data at net rate R_t.
If:
R_t >= R_p
the synchronizer can eventually catch up while acquisition is still active.
If:
R_t < R_p
the unsent backlog must grow. No retry strategy or thread optimization can change that conservation law.
After an acquisition lasting T, the approximate backlog is:
B ~= max(0, R_p - R_t) * T
and the minimum drain time after Stop is:
T_drain ~= B / R_t
Protocol overhead, retransmissions, connection intervals, and flash-read contention make the real time longer.
This observation changed the user-interface contract. “Stop acquisition” can happen immediately, but “file synchronization complete” may happen later. Those are separate events and should be displayed separately.
It also suggests several engineering choices:
The first failing design treated Stop as the end of the whole session:
Stop pressed
-> stop sensors
-> close the file
-> cancel transfer callbacks
-> release BLE/session state
That sequence is attractive because it looks like simple cleanup. It is also wrong when file delivery is still active.
The acquisition producer should stop first, but the transfer consumer must remain alive until it has drained the finalized tail. I separated the lifecycle into explicit states:
IDLE
|
v
COLLECTING_AND_SYNCING
|
| Stop requested
v
STOPPING_PRODUCER
|
| writer flushed and final size captured
v
DRAINING_TAIL
|
| acknowledged_offset == final_size
v
VERIFYING
|
| size and CRC32 match
v
COMPLETE
The important boundary is between STOPPING_PRODUCER and DRAINING_TAIL. Only after the writer has finished appending can the system capture a stable final_size and final CRC32.
In simplified pseudocode:
void on_stop_requested(struct session *s)
{
s->acquisition_state = ACQUISITION_STOPPING;
stop_sensor_requests_async(s);
}
void on_writer_finalized(struct session *s,
uint64_t final_size,
uint32_t final_crc32)
{
s->final_size = final_size;
s->final_crc32 = final_crc32;
s->transfer_state = TRANSFER_DRAINING;
pump_transfer(s);
}
void on_cumulative_ack(struct session *s, uint64_t next_offset)
{
advance_acknowledged_offset(s, next_offset);
if (s->writer_finalized &&
s->acknowledged_offset == s->final_size) {
send_completion_metadata(s);
s->transfer_state = TRANSFER_VERIFYING;
return;
}
pump_transfer(s);
}
Nothing in the Stop handler waits synchronously for BLE. The event loop remains free to process acknowledgments, retransmission timers, disconnects, and finalization callbacks.
The most difficult failure appeared only after Stop. Normal streaming could run for a long time, yet teardown occasionally stalled or disconnected.
The underlying lesson was broader than BLE: canceling an asynchronous operation does not necessarily erase every callback that has already been queued. A callback may still hold a pointer to session state after cleanup has started.
A dangerous sequence looks like this:
callback becomes ready
|
Stop begins teardown
|
session resources are released
|
previously ready callback executes
|
stale state is accessed
The solution is to make ownership and callback lifetime explicit:
An event-driven program often needs two forms of completion:
logical completion: the operation should no longer produce new work
physical completion: no callback can reference its resources again
Confusing them is a common source of stop-path bugs.
Reaching final_size on the sender is not enough. The phone must confirm that its local file has the same length and content.
The completion exchange can include:
session_id | final_size | final_crc32
The phone then:
CRC32 is not a cryptographic integrity mechanism, but it is appropriate for detecting accidental corruption in a transport and storage pipeline. The important point is that verification covers the completed artifact, not only individual BLE messages.
This produces a clear definition of success:
phone_size == device_final_size
&&
phone_crc32 == device_final_crc32
Only then should the UI display “synchronized.”
Timing-dependent failures are hard to diagnose from messages such as “BLE disconnected” or “timeout.” Those are consequences, not causes.
I found it more useful to log a compact snapshot whenever the lifecycle changes:
session=17
acquisition=stopped
writer=finalized
transfer=draining
produced=1824768
sent=491520
acked=475136
pending_chunks=4
connected=true
The most valuable fields were:
With those values, “the download froze” becomes a narrower question:
The happy path is not enough for a synchronization protocol. I used transition-focused tests:
Randomizing the Stop time was especially effective because it exercised different relationships between file writes, BLE notifications, timers, and callbacks.
For each test, the same invariants should hold:
acknowledged_offset never moves backward
acknowledged_offset never exceeds sent_offset
sent_offset never exceeds available file data
final_size becomes immutable after writer finalization
session memory outlives every callback that can reference it
COMPLETE is reached only after end-to-end verification
The final architecture was not one clever BLE optimization. It was a separation of responsibilities:
acquisition controls when data production stops
the writer defines when the file becomes final
the transfer state machine controls reliable delivery
the phone verifies the completed artifact
the event loop coordinates transitions without blocking
The most important lessons were:
What looked at first like a Bluetooth transfer feature became a systems problem spanning acquisition timing, storage visibility, protocol design, event-driven control flow, and resource lifetime. That is exactly why I found it valuable: reliable embedded systems are built at the boundaries between components, where individually reasonable behaviors can combine into an incorrect whole.
That model becomes more interesting when a library already owns another thread and another main loop.
I encountered exactly this problem while integrating a Bluetooth GATT server into an embedded fNIRS acquisition program. The application used libevent for CAN, UART, timers, and sampling state machines, while the GATT server used the BlueZ mainloop in a separate thread.
Both sides needed to access the same device state:
GATT thread libevent main thread
----------- --------------------
receives a phone command owns CAN communication
parses the GATT payload owns sampling state
must return a GATT response owns most fNIRS business state
Calling the business functions directly from the GATT thread would have reintroduced the races that the event-driven refactor was intended to remove. The solution was a small cross-thread dispatch layer built from three pieces:
eventfd that wakes the libevent loop,socketpair per request for returning the result.This article explains the design, the mistakes made along the way, and the boundary between safe cross-thread dispatch and a truly asynchronous protocol.
The most important design decision was not an API. It was an ownership rule:
CAN state, sampling state, and fNIRS business state are modified on the libevent main thread.
The GATT thread may receive and decode a command, but it does not execute the command against shared device state. Instead, it describes the requested work and sends that description to the owner thread.
For example, a request can be represented as a job:
struct ble_job {
uint8_t command;
uint16_t payload_length;
uint8_t payload[512];
int response_fd;
struct ble_job *next;
};
Copying the payload into the job is deliberate. A GATT callback’s input buffer may no longer be valid when the main thread eventually processes the request. The queued job therefore owns all request data needed after the callback returns.
The complete request path looks like this:
phone writes a GATT characteristic
|
v
BlueZ invokes a callback on the GATT thread
|
v
create job + create socketpair
|
v
lock queue -> enqueue job -> unlock queue
|
v
write to eventfd
|
v
libevent wakes and runs the queue callback
|
v
main thread executes the fNIRS command
|
v
write response through socketpair
|
v
GATT thread returns the result to the phone
Only the short queue operation is protected by a mutex. The business command itself runs after the job has been removed from the queue, so a slow command does not hold the queue lock.
A worker thread can safely append an item to a queue, but the event loop does not automatically know that the queue changed.
The main loop might currently be sleeping inside epoll_wait(), waiting for CAN, UART, or timer events. It needs a file descriptor that becomes readable when another thread submits work.
Linux eventfd is a good fit:
int notify_fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
struct event *notify_event = event_new(
base,
notify_fd,
EV_READ | EV_PERSIST,
run_queued_jobs,
NULL
);
event_add(notify_event, NULL);
The GATT thread wakes the loop by writing an unsigned 64-bit value:
uint64_t one = 1;
write(notify_fd, &one, sizeof(one));
The descriptor becomes readable, so libevent schedules run_queued_jobs() on the main thread. The callback drains the counter and then consumes the queue:
static void run_queued_jobs(evutil_socket_t fd, short events, void *arg)
{
uint64_t value;
while (read(fd, &value, sizeof(value)) == sizeof(value))
;
for (;;) {
struct ble_job *job = dequeue_job();
if (job == NULL)
break;
process_ble_job(job);
free(job);
}
}
eventfd stores a counter rather than a stream of individual messages. That is fine because the queue contains the actual jobs. The notification only means:
The queue may contain work; check it.
This separation is useful. The queue carries data, while eventfd carries readiness.
Waking the main loop solves only half of the problem. A GATT request often needs a status code and a response payload before the Bluetooth callback can finish.
For each request, the dispatcher creates a local Unix-domain socket pair:
int pair[2];
if (socketpair(AF_UNIX, SOCK_STREAM, 0, pair) != 0)
return -1;
The two descriptors are connected:
pair[0] <==============================> pair[1]
job/main-thread side GATT-thread side
pair[0] is stored in the job. The GATT thread keeps pair[1], enqueues the job, wakes the main loop, and then waits for a bounded amount of time:
struct pollfd pfd = {
.fd = pair[1],
.events = POLLIN,
};
if (poll(&pfd, 1, timeout_ms) <= 0) {
/* cancel or mark the operation as failed */
close(pair[1]);
return -1;
}
read_response(pair[1], &status, response, &response_length);
The main thread executes the command and writes a framed response to pair[0]:
+---------+---------+---------+--------+------------+---------+
| magic 0 | magic 1 | command | status | length (2) | payload |
+---------+---------+---------+--------+------------+---------+
Framing matters because SOCK_STREAM preserves byte order but not application message boundaries. The receiver first reads and validates the fixed-size header, obtains the payload length, and then reads that many bytes.
The response descriptor also gives each request its own completion channel. There is no need to match a result from one shared response queue to the original GATT request.
At first, using two IPC mechanisms inside one process can look unnecessary. They serve different directions and different lifetimes:
| Mechanism | Direction | Lifetime | Purpose |
|---|---|---|---|
| protected queue | GATT → main | process lifetime | stores jobs |
eventfd | GATT → main | process lifetime | wakes libevent |
socketpair | main → GATT | one request | returns status and payload |
The persistent eventfd avoids creating a new libevent registration for every request. The per-request socket pair makes response ownership and cleanup explicit.
An eventfd could also be used for completion if the response data lived in shared memory. A condition variable could wake the waiting thread. A future or promise abstraction could wrap the same idea. I used socketpair because the response was already naturally expressed as bytes and could be handled with normal poll(), read(), and write() calls.
The dispatch layer must be ready before the GATT server can accept a request.
The startup order is therefore:
create libevent base
|
initialize job queue, mutex, and eventfd
|
register eventfd with libevent
|
start the GATT thread
|
enter event_base_dispatch()
Starting GATT first creates a race: a phone may write a characteristic while the event base or notification descriptor is still uninitialized.
Shutdown needs the reverse discipline:
stop accepting GATT requests
|
stop and join the GATT thread
|
remove the eventfd event
|
close descriptors and reject pending jobs
|
destroy the mutex and event base
Pending jobs need an explicit policy. Silently freeing a job while another thread waits on its response descriptor turns shutdown into a timeout. Closing or completing the response side lets the waiting thread fail promptly.
The queue callback is intended to run only on the libevent thread. I recorded the main thread ID during initialization and checked it before processing jobs:
if (!pthread_equal(pthread_self(), main_thread_id)) {
log_error("job callback ran outside the main thread");
return;
}
This check does not create thread safety by itself. It makes a broken assumption visible.
During development, it is easy to accidentally call an event-loop function from a foreign thread, or to process a job directly as a shortcut. A thread-ID assertion turns an intermittent race into a useful log message near the source of the problem.
In the final implementation, simple commands may run directly when the dispatcher is already on the main thread. Commands whose completion depends on future timer callbacks cannot use that optimization, because waiting for their result on the same thread would deadlock the loop.
The GATT side still waits synchronously for the main-thread result. It must never wait forever.
Different commands have different expected durations:
ordinary control command -> short timeout
stream startup -> longer timeout
node scan -> longest timeout
When a timeout occurs, the implementation records the command and descriptor in a trace file, cancels any active asynchronous operation, closes its response side, and reports failure to GATT.
Useful trace messages include:
dispatch cmd=0x01
run_jobs_cb on main loop
process job cmd=0x01 fd=17
dispatch cmd=0x01 ok
or, when something goes wrong:
poll timeout fd=18 waited=12000ms
Cross-thread systems are much easier to debug when a request has a visible path through enqueue, wake-up, execution, completion, and timeout.
One detail deserves special care: after the waiting side times out and closes its descriptor, a late response must not terminate the process with SIGPIPE. Production code should define the late-completion behavior explicitly, for example by ignoring SIGPIPE, using a no-signal send option where available, and treating EPIPE as cancellation.
The hardest bug was not in the mutex, eventfd, or socket pair.
A Bluetooth SCAN command reached the main thread correctly, and the GATT thread received a valid response. However, the result reported zero nodes even though all 12 CAN nodes were connected.
The scan operation contained a loop that advanced the fNIRS discovery state. Because that loop ran inside a job callback, normal control had not yet returned to event_base_dispatch(). The CAN file descriptor could become readable, but its regular libevent callback did not get its normal opportunity to drain the frames.
The first attempted fix pumped only the higher-level hub state:
advance scan state
process already-decoded hub messages
That was insufficient. The CAN frames were still waiting in the kernel receive queue.
The working pump had to perform the complete path:
drain readable CAN frames
|
decode frames into hub messages
|
advance the scan state
|
run pending non-blocking events
After the CAN drain was included, discovery returned all 12 nodes.
This produced an important lesson:
Calling business logic is not equivalent to servicing its I/O source.
If a callback contains a nested wait or a long state transition, every dependency that would normally be serviced by the outer event loop must still make progress. The cleaner long-term solution is usually to split the operation into timer- and I/O-driven phases so that the callback returns. A nested pump can preserve legacy timing, but it increases reentrancy risk and should remain a carefully bounded exception.
It is tempting to call the entire design asynchronous because the request crosses threads through a queue. That would hide an important limitation.
The main thread receives jobs asynchronously, but the GATT thread waits in poll() until a result arrives:
GATT thread: enqueue -------------------------- wait
main thread: execute ------- respond
Therefore the current design is:
This is a reasonable compromise when the GATT API expects an immediate response and only one Bluetooth control request is normally in flight. It is not ideal if the GATT thread must remain responsive to many concurrent operations.
A fully asynchronous protocol would acknowledge the command quickly, assign a request ID, and later deliver completion through a notification:
phone sends command
|
immediate "accepted" response + request ID
|
main loop performs the operation
|
GATT notification reports completion
That design changes the phone protocol and introduces request tracking, cancellation, and reconnect behavior. The extra complexity is worthwhile only when the product requirements need it.
The implementation is specific to an embedded Bluetooth application, but the design rules apply more broadly:
Connecting two event systems is less about finding one magical API and more about preserving ownership.
In this design, BlueZ owns the GATT thread, libevent owns the fNIRS business state, a protected queue transfers commands, eventfd wakes the owner, and socketpair carries each result back. The components are ordinary Linux primitives, but together they create a clear boundary between two concurrency domains.
The most valuable outcome was not simply reducing data races. The request path became observable and explainable:
receive -> enqueue -> wake -> execute -> respond
When concurrency has a visible path and every piece of state has an owner, embedded software becomes much easier to reason about.
]]>This article records the answers that finally made the model clear to me.
On Linux, a file descriptor, usually abbreviated as fd, is a small integer that identifies an open resource inside a process.
fd 0 -> standard input
fd 1 -> standard output
fd 2 -> standard error
fd 4 -> a CAN socket
fd 5 -> a UART device
fd 6 -> a listening socket
The descriptor is not the resource itself. It is a handle that the process uses when asking the kernel to operate on that resource:
read(fd, buffer, size);
write(fd, data, length);
close(fd);
This common interface is important because an event library can monitor many different resources in the same way. A TCP socket, a CAN socket, a pipe, a timerfd, and some device files can all participate in one event loop.
Consider a normal blocking read:
read(can_fd, buffer, sizeof(buffer));
If no CAN frame is available, the calling thread waits. While it is waiting, that thread cannot process a UART message, a Bluetooth request, or an expired timer.
A naive program might try to check every source repeatedly:
while (running) {
check_can();
check_uart();
check_bluetooth();
check_timers();
}
This is polling. It can waste CPU time when nothing is happening, and it becomes awkward as the number of event sources grows.
Linux provides a better mechanism through facilities such as epoll: the application tells the kernel which descriptors it is interested in, and the kernel reports which ones are ready.
CAN fd -----\
UART fd ------> kernel waits for readiness ---> event loop wakes up
GATT fd -----/
timerfd -----/
The application does not wait on one device at a time. The kernel waits on all registered sources together.
At a high level, an event loop behaves like this:
while the application is running:
wait until one or more events are ready
mark those events as active
call the callback associated with each active event
With libevent, a read event may be registered like this:
struct event *can_event = event_new(
base,
can_fd,
EV_READ | EV_PERSIST,
can_read_callback,
context
);
event_add(can_event, NULL);
This says:
Monitor
can_fd. Whenever it becomes readable, callcan_read_callback.
libuv expresses the same general architecture with different types and APIs. Both libraries connect operating-system event notification to application callbacks.
Suppose the application has registered three callbacks:
CAN fd ready -> can_callback()
UART fd ready -> uart_callback()
GATT fd ready -> gatt_callback()
If all three descriptors become ready, the event loop activates all three events. In a single-threaded loop, the callbacks are normally executed one after another:
can_callback()
uart_callback()
gatt_callback()
The order may vary unless explicit priorities are configured, so application correctness should not depend on an accidental ordering between events of equal priority.
Sequential execution is not the same as blocking. A callback that processes already-available data and returns quickly is simply taking its turn. Blocking occurs when the callback waits for something that has not happened yet.
For example, this is dangerous:
static void can_callback(int fd, short events, void *arg)
{
read(fd, buffer, sizeof(buffer)); /* current frame */
read(fd, buffer, sizeof(buffer)); /* may wait for a future frame */
}
A common non-blocking pattern is to drain the data that is already available and stop when the kernel reports EAGAIN:
static void can_callback(int fd, short events, void *arg)
{
for (;;) {
ssize_t n = read(fd, buffer, sizeof(buffer));
if (n > 0) {
process_frame(buffer, n);
continue;
}
if (n < 0 && (errno == EAGAIN || errno == EWOULDBLOCK))
break;
break;
}
}
The callback then returns control to the event loop so that other ready events can run.
These two terms are often used together:
For example:
register interest in a socket
|
v
continue handling other work
|
v
kernel reports that the socket is readable
|
v
event loop invokes the read callback
The application expresses interest at one point in time and receives the result later through a callback. No thread has to remain blocked on that socket.
This model became much clearer when I applied it to an embedded acquisition system containing CAN, UART, Bluetooth GATT, and a sampling state machine.
The design has two important event sources:
CAN fd readable event -> receive and parse CAN frames
1 ms timer event -> advance the sampling state machine
Sampling is not implemented as one long blocking function. Instead, it is split into small phases:
configure an LED over CAN
|
wait until a deadline, without sleeping
|
send a sampling command
|
request data from a node
|
return to the event loop
|
CAN response arrives and activates the CAN callback
|
store the response and mark the node as ready
|
a later timer tick advances the state machine
The key point is that CAN reception and sampling are not two large tasks where one must finish completely before the other begins. They cooperate:
sampling tick sends a CAN request
CAN callback receives the response
next sampling tick observes the response
sampling state machine continues
Waiting is represented as state plus a deadline, not as sleep():
state->deadline_ms = now_ms + 3;
state->phase = WAIT_FOR_LED;
return;
On later timer ticks:
if (now_ms < state->deadline_ms)
return;
state->phase = START_ADC;
During those three milliseconds, the main thread remains free to process CAN, UART, GATT, and other timers.
At first, a single-threaded event loop can look inferior to multiple threads. Multiple threads can run on several CPU cores, while callbacks on one loop execute sequentially.
The missing detail is that I/O-driven programs usually spend much more time waiting than computing. The event loop lets the kernel wait for many I/O sources simultaneously, then performs only the short amount of work required for each ready event.
A single-threaded event loop provides several useful properties:
In an embedded system, a Bluetooth command, a CAN response, and a sampling timer may all touch the same state. Running their small handlers on one thread often makes the system more deterministic than allowing three threads to modify that state concurrently.
This does not mean that threads are bad. They are valuable for:
The practical design is often hybrid:
main event-loop thread
├── CAN
├── UART
├── GATT
├── timers
└── state machines
worker threads
├── expensive calculations
└── unavoidable blocking operations
The rule is simple: keep event-loop callbacks short, and move genuinely expensive or blocking work elsewhere.
A worker thread does not automatically become part of the main event loop. It needs a thread-safe notification mechanism.
libuv provides uv_async_send() for this purpose:
worker thread
|
| uv_async_send()
v
main event loop wakes up
|
v
async callback runs on the loop thread
With libevent on Linux, a similar mechanism can be built using an eventfd, pipe, or socket pair:
worker thread
|
| enqueue result
| write(eventfd)
v
eventfd becomes readable
|
v
libevent invokes its callback on the main thread
|
v
main thread consumes the queued result
The notification transfers control back to the event-loop thread. Shared data still needs a mutex, atomics, or a thread-safe queue.
libevent and libuv differ in API design and scope, but the central model is similar:
| Concept | libevent | libuv |
|---|---|---|
| Event loop | event_base | uv_loop_t |
| I/O readiness | event_new() and callbacks | stream, poll, and handle callbacks |
| Timers | timer events | uv_timer_t |
| Cross-thread notification | commonly eventfd, pipe, or event_active() with configured threading | uv_async_t and uv_async_send() |
| Worker execution | application threads or external thread pools | uv_queue_work() and the libuv thread pool |
libuv offers a broader cross-platform abstraction for networking, files, processes, DNS, and threads. libevent focuses strongly on event notification, timers, and network-oriented building blocks. Choosing between them depends on the application, but understanding either one teaches the same fundamental lesson:
Do not block while waiting for I/O. Register interest, return control to the loop, and react when the event becomes ready.
My final mental model is:
Once I stopped imagining the event loop as “one thread waiting on one device,” the architecture made sense. It is one thread coordinating many sources of readiness, while the operating system performs the waiting efficiently.
]]>