Ingext Architecture
Ingext is a data fabric designed to treat all telemetry—regardless of origin, format, or transport—within a single architectural flow.
Ingext is a data fabric designed to treat all telemetry—regardless of origin, format, or transport—within a single architectural flow. The objective is to provide a consistent ingestion, processing, and delivery model that separates concerns cleanly: sources define how data enters, pipelines define how data is transformed, routing determines where it goes, and destinations encapsulate how and where it is written.
A flow follows a pattern of:
- data source
- transformation and routing
- data store
We begin with data sources. In Ingext, a data source is a complete ingestion definition: it specifies how the platform connects to an external system, how it authenticates, how it retrieves or receives data, and which transport protocol is used. This might be Syslog, HEC, S3, a webhook, an API poller, Prometheus, or a Kinesis consumer. Critically, authentication credentials and connection parameters live entirely within the source object. Downstream stages—transformation, routing, and destinations—never receive or depend on credentials or the source’s structural details. They operate only on the raw events the source provides.
Within this category, we also have Defined Applications. A Defined Application is not a different kind of source; it is a template that bundles three things together: a predefined source configuration, a predefined transformation pipeline, and a default routing configuration. It is essentially a “pre-assembled integration.” Installing a Defined Application—for example, SentinelOne, CloudTrail, Proofpoint, Okta, CrowdStrike, or any of the many S3, Syslog, or webhook-based integrations—automatically creates the source, attaches the correct parsing and transformation logic, and establishes a default route that stores processed records in Lava Lake using that application’s default schema. Users may modify the pipeline or routing, but the template ensures correctness from the start.
Once events enter the system, they move into the transformation pipeline. This stage performs parsing, transformation and routing logic. Parsing converts the raw input format into a structured event. Transformation then applies normalization rules, enrichment, error checking, type corrections, schema alignment, redaction, and any additional record shaping—these are all components of transformation and are not separate subsystems. The result of transformation is a fully structured, enriched, schema-aligned event. Importantly, transformation does not have access to source credentials or internal source configuration; it operates purely on the event payloads and enrichment context. Transformation has access to cache data, allowing it to store and retrieve historical data creating the ability to maintain state when implementing logic.
After a record has been transformed, it enters the routing layer, which can explicitly be defined in the transformation logic. Routing evaluates each event against declarative rules based on attributes such as application type, event class, tenant, enrichment outputs, severity, and more. Routing has two responsibilities: determining which destinations should receive the record, and determining whether any events should be suppressed. A single event may be routed to multiple destinations without reprocessing. If routing rules indicate that an event should be dropped—for example, due to noise suppression or compliance filtering—the event is discarded at this stage.
Next, the event arrives at the data store layer. In Ingext, a destination is any configured output that receives transformed records: Lava Lake, Elastic, Loki, S3, HEC, Kinesis, Lambda, EventWatch, or a webhook. Each destination encapsulates its own authentication requirements, indexing preferences, schemas, storage formats, or protocol details. Just as sources encapsulate credential and transport details from upstream stages, destinations encapsulate schema and write semantics from the processing logic. The transformation and routing stages hand off a schema-aligned event; the destination takes responsibility for writing it according to its own contract.
The default destination is Lava Lake, the Parquet-based structured storage layer that Ingext Search can access. Every Defined Application includes a default route into Lava Lake using the application’s default schema. This creates a consistent, strongly typed, analytical-grade dataset. However, Lava Lake is not the only option. Data may also be sent to Elastic for operational search workloads, to Loki for log-centric observability, to S3 or Kinesis for cloud processing pipelines, or to HEC for delivery into Splunk or other SIEMs. Because connectors can act as both sources and destinations, HEC, S3, Kinesis, and others may be used bidirectionally without modifying the architecture.
Finally, EventWatch, a Fluency Service, appears in this architecture purely as a destination. When routed to EventWatch, events are fed into Fluency’s correlation, behavioral analysis, and risk scoring engines. Those engines produce outputs that Fluency exposes through its own service layer; Ingext remains focused on ingestion and delivery, maintaining a clear separation between transformation and analytics.
Overall, Ingext’s architecture is defined by a consistent high-availability one-way flow:
data source → transformation & routing → destination.
Every component has a narrow responsibility. Sources encapsulate inbound authentication and protocol details. Transformation produces structured, enriched events independent of both source and destination definitions. Routing decides delivery paths. Destinations encapsulate outbound authentication, schema selection, and write semantics. This separation of concerns is what allows Ingext to support dozens of ingestion methods and dozens of destination types while preserving a stable internal architecture.
Want to experience Ingext. Try it out here.
