Skip to main content

Log Ingestor: one writer, many readers

· 5 min read
Alejandro Revilla
Alejandro Revilla
jPOS project founder
AR Agent
AR Agent
AI assistant

Operational logs are part of the control surface. In a single-node demo it is tempting to let the same process receive logs, write the search index, and serve the viewer. That works until the Control Plane itself needs to scale, restart independently, or run without holding the only Lucene write lock.

jPOS now splits that responsibility into a dedicated log ingestion path: workloads emit structured jPOS JSONL logs to standard output, Fluent Bit transports labelled pod logs, a log-ingestor service owns the write side, and the jPOS Control Plane reads the shared index read-only.

What the demo shows​

The video starts with the architecture and then switches to a live Log Viewer tour:

  • workload pods and Control Plane pods emit structured JSONL log events
  • pods opt into collection with the controlplane.jpos.org/log-collect=true label
  • Fluent Bit tails container stdout and forwards matching events over HTTP
  • the log-ingestor receives batches on /admin/logs/ingest
  • the ingestor is the only component that writes the Lucene index and BinLog payload store
  • the shared volume holds both /data/log-index and /data/log-payload
  • Control Plane replicas mount the same volume read-only and serve the Log Viewer from it
  • the live viewer filters by realm, searches the structured payload, opens event details, and shows events from several hosts converging into one index

The important part is not just that logs show up in a UI. The important part is the ownership model.

One writer​

Lucene indexes have a clear rule: there should be one writer. The new log-ingestor service makes that rule explicit.

In Kubernetes, the ingestor runs as a single replica with a Recreate deployment strategy. It owns the index writer and the BinLog payload writer. Fluent Bit posts events to the ingestor's HTTP endpoint, and the ingestor appends them to the shared storage.

That keeps write locking simple. There is no race between Control Plane replicas, no accidental second index writer, and no need for every UI pod to perform ingestion work.

Many readers​

The Control Plane now has a separate read-only backend for the Log Viewer. LogReaderService opens the shared Lucene index and BinLog payload store without taking the writer role, refreshes its reader periodically, and lets the existing Log Viewer handlers resolve that read backend.

That means Control Plane pods can be scaled horizontally. Each replica can serve the same operational view, but none of them owns the write path. If one Control Plane pod restarts, ingestion continues. If the Control Plane scales from one replica to three, the log index still has one writer.

This is the same shape we want elsewhere in regulated infrastructure: separate the mutation path from the observation path, make the owner of each side obvious, and keep the operational contract easy to reason about.

Fluent Bit as transport​

Fluent Bit is deliberately used as transport, not as the source of truth.

The DaemonSet tails /var/log/containers, applies Kubernetes metadata, filters for the jPOS log-collection label, and sends matching records to the ingestor. It does not parse business meaning into a separate observability schema, and it does not own the index.

The structure still comes from jPOS log events. That is why the viewer can facet by kind, realm, host, and trace identifier, and why opening an event can show the original structured payload instead of a guessed text parse.

Shared storage​

The ingestor writes the Lucene index and the BinLog payload store to a persistent volume. The Control Plane mounts the same claim read-only.

For production, the chart is designed around a real ReadWriteMany storage class such as NFS, CephFS, EFS, or Longhorn RWX. For local single-node development, the chart can run on local-path style storage with pod affinity so the writer and readers land where the shared path is available.

The storage contract is intentionally boring: one component writes; many components read.

Ready for tenant isolation​

The chart already carries the next shape: one ingestor and one isolated storage volume per tenant.

Today, an empty tenant list renders the system-wide ingestor. When tenant-specific deployments are enabled, the same template structure can render a separate ingestor, service, and persistent claim per tenant. That keeps the scaling model aligned with the data-isolation model instead of turning logs into a shared cross-tenant bucket.

Why this matters​

The Log Viewer started as a useful way to search structured jPOS events. The log ingestor turns it into a deployable architecture for clustered systems.

It gives jPOS a clean pipeline:

  • applications emit structured events
  • Fluent Bit transports selected pod stdout
  • one ingestor writes the durable index and payload store
  • Control Plane replicas read that data read-only
  • operators get the same faceted, searchable view without coupling UI scale to ingestion

That is the point of the new architecture: not more moving parts for their own sake, but a smaller responsibility for each moving part.