InfluxDB’s new model for time series workloads

In 2017, I was developing software focused on answering one question: “What happens when this airplane gets struck by lightning?” Not if, but when. On average, every airplane is struck by lightning once per year. And when that moment arrives, milliseconds matter. The entire system’s status must be evaluated, backup sensors activated if needed, and data across all these points need to be analyzed in true real-time. At 30,000 feet, there’s simply no time for latency.

When we began developing InfluxDB 3, solving problems like this (fast, complex, and in real time) was top of mind. Most time series databases are built to store data fast and serve it back even faster. But storing and accessing data isn’t the hard part anymore. Making intelligent decisions with it in real time is.

Real-time systems are no longer the edge case, they’re the default. Yet developers are stuck cobbling together stream processors, schedulers, and brittle glue code just to transform data, trigger alerts, and keep things running. That’s where InfluxDB 3 takes a different approach. Instead of treating the database as a passive store, it brings computation to the storage layer, so you can act on data as it streams in, not after the fact.

At the heart of this shift is our embedded Python Processing Engine. It runs directly inside the database, allowing developers to write custom Python that executes next to the data. The benefit? No extra infrastructure or external VMs. No network latency. No waiting. What was once a storage layer becomes an active, real-time engine for data transformation, automation, and alerting.

We’ve released two products built on this new architecture: InfluxDB 3 Core, our open source engine licensed under MIT and Apache 2, and InfluxDB 3 Enterprise, the commercial edition built for production. InfluxDB Core is lightweight, single-node, and designed for real-time workloads like edge transformation, streaming analytics, and embedded alerting. InfluxDB Enterprise builds on that foundation with historical query support, read replicas, high availability, workload isolation, and multi-node scalability. Same engine, two different jobs.

Computation where the data lives

The Processing Engine is a lightweight Python VM embedded directly in InfluxDB. It brings intelligence closer to where the data lives, so you can do more, faster, without shipping it between services. Instead of bolting on separate infrastructure for analysis, alerting, and automation, you run logic as the data flows in.

Included in the Processing Engine is a cache for local storage as well, making it easy to share data across executions. Need to make an API call for data enrichment? You don’t have to make it each time from an ephemeral execution state. You can store that response and use it across multiple calls to the Processing Engine, reducing costs and complexity.

This is what bringing computation to data looks like: a truly active layer that can transform, store, recall, and analyze data.

We built the Processing Engine because developers kept telling us the same thing: the tools around time series data were too fragmented. If you want to trigger an alert, you need a separate alerting service, and that requires monitoring and overhead. Cleaning or enriching your data requires a new ETL pipeline, and even a basic transformation often means spinning up a job outside the database, wiring it all together, and hoping nothing breaks along the way.

With an active storage layer, all of that happens inside InfluxDB.

We’ve already seen developers use the Processing Engine to enrich IoT sensor data with real-time weather feeds (triggering Slack notifications when temperatures cross dynamic thresholds) and run predictive models to catch metrics anomalies before they escalate. Others have built automated reporting workflows that compile dashboards with a scheduled plugin, upload them to cloud storage, and notify their team, all from inside the database. One user, dealing with “alert fatigue” from constant monitoring spikes, used the in-memory cache to suppress duplicates and apply cool-down logic that adapts in real-time.

These workflows once required full pipelines stitched together with multiple tools and services, often built on entirely different nodes with separate failure points. Now, they’re small, purpose-built Python scripts that deploy right where the data lives. It’s not about architecting around the database, it’s about building with it.

Why Python, and why now

We selected Python after reviewing and prototyping several different options. It’s familiar, expressive, and already the language many developers and data engineers use when building logic around time series data. Plus, as large language models improve at generating working Python code, the Processing Engine becomes more powerful. Even with one-shot training, these models are becoming increasingly effective at building excellent plugins. Need to automate anomaly detection or build a trend-line analysis for your dashboard? Just describe it, get the code, and run it—all without leaving the database.

InfluxDB 3 isn’t just a faster database, it’s the start of a new model for building with time series data. We’ve made it open, programmable, and flexible enough to support everything from hobbyist projects to critical infrastructure.

The Processing Engine is central to this shift. By embedding programmable logic directly into the database, we’re turning InfluxDB into an active intelligence layer, one that can transform, enrich, analyze, and act on data the moment it arrives. It’s no longer about collecting data first and figuring out what to do with it later. Now, the system itself can respond in real time.

In the months ahead, we’ll continue expanding the Processing Engine with new triggers and plugins. If you’re building systems that need to react as fast as the data moves, InfluxDB 3 gives you the infrastructure to do it with active intelligence built in.

Pete Barnett is lead product manager at InfluxData.

—

New Tech Forum provides a venue for technology leaders—including vendors and other outside contributors—to explore and discuss emerging enterprise technology in unprecedented depth and breadth. The selection is subjective, based on our pick of the technologies we believe to be important and of greatest interest to InfoWorld readers. InfoWorld does not accept marketing collateral for publication and reserves the right to edit all contributed content. Send all inquiries to doug_dineley@foundryco.com.