Reducing Telemetry Volumes With Cribl

In 2022, the engineering team at Getty Images faced a classic scaling challenge. Their main logging system received several terabytes of telemetry daily, and the projected annual increase in volume was 30%. Maintaining such dynamics threatened the company with additional expenses measured in millions of dollars.

The situation was complicated by disparate configurations, non-transparent filtering mechanisms, and the inability to preview masses before uploading. Every new source connection turned into a guessing game, and retaining all information volume “just in case” became an extremely inefficient business practice.

To solve this infrastructure challenge, specialists changed the very paradigm of data handling by using the Cribl platform. The new solution allowed processing telemetry directly in transit — the system automatically cleanses, filters, restructures, and redirects streams of information before the indexing stage.

Thanks to this approach, architects gained unprecedented visibility and flexibility. The platform became the tool that unlocked the potential for true data ownership. Now information arrives in the required format exactly where it is needed, allowing security teams to work with specific streams without overloading the main SIEM system.

The profound technical transformation required the introduction of clear engineering structures. Engineers implemented strict naming rules for sources and pipelines, and began adding metadata directly to the payload. For instance, a special source origin field became critically important for further error detection. Meanwhile, the Cribl Stream solution allowed isolating various workloads and specific use cases.

This infrastructural independence ensures that a configuration mistake in one department or a local activity spike does not affect the stability of the entire corporate network. For safe architecture testing without the risk to the production environment, routing to devnull and anonymized sample generation tools are actively used. An additional advantage became the integration of artificial intelligence: Cribl Copilot enables quick pipeline building and KQL query generation.

The most evident achievement of the optimization was halving the volume of incoming logs. Importantly, the company manages to maintain this figure at a stable level for several years despite constant expansive business growth. The main savings were achieved by reducing the requirements on the indexer cluster. However, the fundamental changes were process-related, including the introduction of a mandatory telemetry collection request form. It forces information owners to answer a number of critical questions before integration: what type of data is being uploaded, expected volumes, necessary retention period, and access policies. This shifts the responsibility for logging necessity onto the process initiator, preventing uncontrolled data hoarding.

Changing the habits of developers and related units turned out to be a much more challenging task than directly setting up the software. Experts emphasize that the successful implementation of such large-scale projects should be built around addressing the pressing issues of internal partners, rather than simply showcasing new functional capabilities. Monitoring specialists must develop the network based on a gradual transition principle, offering other teams solutions to urgent problems like licensing overage or event visibility in the infrastructure. In this paradigm, the operations division transforms into a reliable solution provider whose work remains stable in routine mode.

In summary, effective data pipeline management requires a combination of proactive routing and strict logging discipline. Storing petabytes of unfiltered information is no longer a justified step: companies need tools to structure telemetry before its indexing. This allows financial resources to be freed up and for security analysts’ work to be focused exclusively on relevant threats.