racket-webview/scrbl/rktwebviewqt-internals.scrbl

#lang scribble/manual

@title{Qt WebView Backend Architecture}
@author[@author+email["Hans Dijkema" "hans@dijkewijk.nl"]]

@section{Introduction}

It would, of course, be preferable to place everything within a single process,
under a unified structure. This would be elegant. It is not how things are.

@centered{
  @image[#:scale 0.45]{rktwebview-shared-memory-diagram-simple.svg}
}

Qt WebEngine establishes its own order: threads, event loops, internal state.
Once set in motion, it does not easily yield. It persists, and it expects its
environment to adapt accordingly. These conditions are accepted.

The Racket process, however, is of a different nature. It is light, precise,
capable of starting and stopping without residue. It must remain so.

So a boundary is drawn.

On one side, Qt: a large, immovable instrument—something like an organ. Once it
begins to sound, it fills the space, and it is not easily silenced. On the other,
Racket: a violin, agile and expressive, able to begin and end a phrase at will.

They do not become the same instrument. They are allowed to play together.
Communication is arranged accordingly. A shared memory region, containing three
queues: commands, results, and events. A command is issued. It crosses the boundary. It is taken up and executed. A result returns.
Events also arise, independently, and must be handled when they appear.

Within this structure, the violin may move freely—provided it does not attempt to
reconfigure the organ. No attempt is made to unify the instruments. Such efforts would not improve the music. Instead, the composition is written so that each plays its part.

From the outside, one hears only a simple exchange: a call, a response. Internally, the balance is carefully maintained. For now, this is sufficient. And it holds.

@section{Overview}

This backend provides a webview implementation by delegating all GUI and browser
functionality to a separate Qt process.

The embedding Racket process does not manipulate Qt widgets directly. Instead,
it communicates with a helper process that owns the Qt event loop and all
@tt{QWebEngine} objects.

This design exists to work around limitations of Qt WebEngine in combination with
the lifecycle model of the DrRacket environment.

@section{Execution Model}

The runtime consists of two processes: the embedding Racket process and a helper
process running Qt and Qt WebEngine.

All GUI state lives in the helper process. The embedding side holds no direct
references to Qt objects. Communication is explicit and happens through shared
memory.

@section{Shared Memory and Queues}

A shared memory region is created during initialization. Inside that region,
three FIFO queues are established: a command queue, a result queue, and an event
queue.

Each message consists of a numeric code and a payload, typically JSON:

@centerline{@tt{(code, payload)}}

The queues have distinct roles. The @italic{command queue} carries requests from the embedding process to the Qt process, for example creating a window, loading a URL, or executing JavaScript. The @italic{result queue} carries direct replies to those commands. A synchronous call on the embedding side blocks until a corresponding result is available. The @italic{event queue} carries asynchronous notifications generated by the Qt side, such as page load completion, navigation requests, window movement, or events
originating from JavaScript.

@section{Command Execution}

A function call on the embedding side is translated into a command and written to
the command queue.

From there the flow is fixed: (1) the command is read by a worker thread in the
helper process, (2) it is reposted onto the Qt GUI thread, (3) the GUI thread
executes the operation, and (4) the result is written back to the result queue.

The worker thread never manipulates Qt objects. All GUI work happens on the GUI
thread.

From the caller’s perspective, a synchronous call returns only after the GUI
thread has completed the action.

@section{Event Delivery}

Many relevant events are not tied to a specific command. Page loading, navigation
attempts, window movement, and JavaScript-originated events are delivered through
the event queue.

Events are retrieved explicitly by polling.

Each event contains a name, an identifier, and optional fields depending on its
type. Events are delivered in FIFO order.

@section{Contexts and Webviews}

The backend uses a two-level model consisting of contexts and webviews.

A context represents a browser environment and corresponds to a
@tt{QWebEngineProfile}. It defines how pages run, including injected scripts and optional trust configuration using explicitly trusted self-signed certificates.

Each context is identified by an integer handle.

Within a context, one or more webviews can be created. A webview represents a
window containing a browser view. Webviews are also identified by integer
handles.

A webview always belongs to exactly one context. When creating a webview, the
context handle must be provided.

Webviews may optionally have a parent webview. If a parent is specified, the
resulting window is created as a modal child of that parent; otherwise it is
created as a top-level window.

From the Racket side, this means that a context must be created first. That
context handle is then used to create webviews, which are subsequently addressed
through their own handles.

All Qt objects remain internal to the helper process; only these integer handles
cross the process boundary.

@section{JavaScript Bridge}

Each context installs a small JavaScript bridge into every page, allowing
JavaScript code to send structured data to the host via:

@centerline{@tt{window.rkt_send_event(obj)}}

The objects are collected and forwarded to the event queue.

@section{Navigation and Window Behavior}

User actions are not always executed immediately; navigation initiated by the
user may result in a @tt{"navigation-request"} event instead of being followed
automatically, and closing a window may result in a @tt{"can-close?"} event. The
Racket side is expected to decide how to handle these situations.

@section{Design Considerations}

@bold{Qt WebEngine lifecycle.}
Qt WebEngine cannot be safely reinitialized within a single process.

In practice, once a @tt{QApplication} using WebEngine has been started and shut
down, the WebEngine runtime cannot be started again. This is a known and
documented limitation (see for example QTBUG-70519, QTBUG-87460,
QTBUG-145033). The underlying cause is that WebEngine starts internal threads
and resources that are not fully released, even after application shutdown.

Attempts to reinitialize WebEngine in the same process result in undefined
behavior, including crashes, hangs, or inconsistent state.

In the DrRacket environment, where components may be restarted under a
custodian, this makes an in-process design fundamentally unsuitable. Even when
libraries are loaded and unloaded using @tt{#:custodian}, the WebEngine runtime
cannot be reset to a clean state.

By moving Qt and WebEngine into a separate process, this limitation is avoided
entirely: each start of the backend creates a fresh runtime, and terminating the
helper process guarantees that all associated threads and resources are released
by the operating system.

@bold{Event loop and threading.}
Qt requires that GUI operations are performed on the Qt GUI thread.

Instead of attempting to integrate Qt’s event loop with Racket, the design
isolates Qt completely and runs it in its own process.

Within that process, a worker thread receives commands and forwards them to the
GUI thread using Qt’s event mechanism (via @tt{postEvent}). The Racket side never
interacts with Qt objects directly.

@bold{Failure isolation.}
Qt WebEngine is a large subsystem with its own internal processes (including the
Chromium-based @tt{QtWebEngineProcess}) and is generally stable in practice.

Running the Qt side in a separate process provides isolation: if the helper
process terminates, the embedding Racket process remains unaffected and can
decide how to recover.

@bold{Shared memory communication.}
The communication pattern consists of commands, results, and events, mapped onto
shared memory FIFO queues, keeping the model simple and explicit.

@bold{JSON encoding.}
All payloads are encoded as JSON, providing a natural bridge between JavaScript,
Qt/C++, and Racket: JavaScript produces JSON natively, Qt maps it to variant
types, and Racket can decode it easily.

For control commands, payload sizes are small and infrequent, so serialization
cost is negligible compared to GUI-thread execution and WebEngine processing; for
dynamic data such as JavaScript results and custom events, JSON is the
appropriate representation. A binary protocol would reduce overhead but increase
complexity and reduce inspectability.

@section{Shared Memory Architecture}

Communication between the Racket process and @tt{rktwebview_prg} is implemented
using a shared memory region. This region serves three purposes at once: it
stores shared data structures, it provides a simple allocator, and it hosts the
FIFO queues used for message passing.

At the start of the shared memory block, a small administration area is stored,
including a pointer to the current end of allocated memory, a list of active
allocations, a free list, and a fixed slot table. The slot table acts as a
directory of shared objects; queues are created once, stored in slots, and can be
retrieved by both processes using only the slot number.

Memory allocation inside the shared block is intentionally simple. Each
allocation is preceded by a small header containing size information and links
for a double-linked list. Allocation first attempts to reuse a block from the
free list; if no suitable block is available, memory is taken from the unused
tail of the region. Freed blocks are returned to the free list and may later be
reused. Blocks are not compacted or coalesced. This is not a general-purpose
heap; it is a small, predictable allocator for queue items and payload strings.

Shared objects are referenced primarily through offsets (via @tt{ShmPlace})
rather than raw pointers. This makes the layout independent of the virtual
address at which the shared memory is mapped in each process.

Queues are built directly on top of this allocator. Each queue consists of a
small header containing the first item, the last item, and a count, followed by a
linked list of queue items. Each item stores a numeric command or event code, a
pointer to its payload in shared memory, and links to neighboring items.

Synchronization is split into two layers. A shared lock protects allocator and
queue metadata, while each queue has its own semaphore indicating whether items
are available. One mechanism protects the structure; the other tells you whether
there is anything worth reading.

On POSIX systems such as Linux, shared memory is implemented using
@tt{shm_open}, @tt{ftruncate}, and @tt{mmap}, with synchronization via named
POSIX semaphores created using @tt{sem_open}. The owner process initializes these
objects and removes them again using @tt{shm_unlink} and @tt{sem_unlink}.

On Windows, the same model is implemented using @tt{CreateFileMappingA} and
@tt{MapViewOfFile} for shared memory, and @tt{CreateSemaphoreA} or
@tt{OpenSemaphoreA} for synchronization. The design is identical, but the kernel
objects follow the Windows lifetime model and are released when the last handle
is closed.

The shared memory region has a fixed size (currently 10MB) and is not resized at
runtime. Although the use of @tt{ShmPlace} offsets would in principle allow
relocation, resizing would require coordinated remapping in both processes while
all activity is paused. The current design therefore treats the region as
fixed-size and relies on reuse of freed blocks.

This implies that the communication channel is bounded. Payloads such as
@tt{set_html} or large JavaScript results must fit within the available free
space in the shared memory block. In practice, the usable limit is somewhat below
the nominal 10MB due to allocator overhead, queue administration, and concurrent
messages.

This is a message transport, not an infinite sack of HTML.