# Tool for InfluxDB batch ETL

**URL:** <https://community.influxdata.com/t/tool-for-influxdb-batch-etl/56955>\
**Category:** InfluxDB 2\
**Tags:** influxdb\
**Created:** [April 8, 2025, 4:42pm UTC](https://community.influxdata.com/t/tool-for-influxdb-batch-etl/56955 "2025-04-08T16:42:12Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![ArthurKretzer](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/arthurkretzer/32/19789_2.png) [@ArthurKretzer](https://community.influxdata.com/u/ArthurKretzer)\
**Post date:** [April 8, 2025, 4:42pm UTC](https://community.influxdata.com/t/tool-for-influxdb-batch-etl/56955/1 "2025-04-08T16:42:12Z")

</div>

Hello everyone,

I’m currently facing challenges with replicating InfluxDB data to a data lake.

I’m using InfluxDB v2 OSS on-premises and have developed a fault-tolerant Python script to replicate its raw data. While functional, the script has become a headache due to the large volume of data—it takes too long to process.

My goal is to achieve full data replication to Parquet files stored in a MinIO instance. However, I haven’t found a tool that fits my needs. The Airflow operator isn’t sufficient, Telegraf and Kapacitor don’t seem to solve the problem, and Quix Streams would require me to set up Kafka, which I’d prefer to avoid since I’m not dealing with streaming right now.

Is there any tool that can handle this, similar to how Airbyte manages batch replication for Postgres and other databases?

Thanks in advance!

---

<div class="post-metadata">

**Author:** ![Anaisdg](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/anaisdg/32/6401_2.png) [@Anaisdg](https://community.influxdata.com/u/Anaisdg)\
**Post date:** [April 15, 2025, 6:24pm UTC](https://community.influxdata.com/t/tool-for-influxdb-batch-etl/56955/2 "2025-04-15T18:24:33Z")

</div>

Hello @ArthurKretzer,  
Might I interest you in InfluxDB v3? You can use the python processing engine that’s embedded in it and someone in the org created a parquet exporter so you can export parquet in influxdb v3 to iceberg so you can read iceberg tables from datalakes like duckdb or snowflake. Unfortunately its not on a public repo but I’m working to move it there asap. You might be interested in learning about the python processing engine though for influxdb 3 core and enterprise.

> **[Processing engine and Python plugins | InfluxDB 3 Enterprise Documentation](https://docs.influxdata.com/influxdb3/core/plugins/)**
>
> Use the InfluxDB 3 Processing engine with Python to trigger and execute custom
> code on different events in an InfluxDB 3 Enterprise instance.

> **[GitHub - influxdata/influxdb3\_plugins: Python plugins for InfluxDB 3](https://github.com/influxdata/influxdb3_plugins)**
>
> Python plugins for InfluxDB 3

oooooh also fyi Quix doesn’t require you to setup kafka, it handles all the kafka under the hood for you so it might be easier than you think. It’s whole selling point is trying to relieve you of the pain you just mentioned.