Hello Community, I have 2 big measurements, one with GPS data and the other with particulate matter (pm10 and pm25) values. I want to join the longitude and latitude fields to the measurement with the sensor data to show them later on a map. Following Flux Query is working when I just use data of o…

Hello @knalla66 , You might want to try experimental.join() function | Flux 0.x Documentation

[image] knalla66: But with more than 100 different measurements this would be too much work overload and I hoped that I can just join 2 big files as well. Im still learning , but I think the way your data is structured currently would require the whole thing in memory for the pivot and join o…

thx for the suggestion - looked good and easy but also with experimental it does not work. could it be that experimental is only working on influxdb 2.0? I still have influxdb 1.8.1 in us.

I used the pivot function to get two fields instead of one (latitude and longitude for example) at the same time in the query. Didn’t find any other way - mostly there is only 1 field selected in the filter when I looked up possibilities. I am also a new user and maybe there are other ways. I am us…

@knalla66 Flux in InfluxDB 1.x is read-only, so you aren’t able to write data back to InfluxDB with Flux (in 1.x). @FixTestRepeat is correct. Flux has to pull the entire queried dataset into memory to perform the pivot and the join, both of which are very expensive operations. With the nature of you…

ok, thank you for your help. I will bear in mind for future time series to make tasks and pre-transform the data before storing them. For the available data, is there a possibility to transfer the data from influxDB 1.8.1 to the new InfluxDB 2.0 and than transform it?

Nathaniel on the Flux team here. I wanted to chime in with some of my thoughts. First off the advice you have received so far is spot on. Joins in Flux currently require that the whole data fit in memory. That said the experimental.join function that is available in InfluxDB 2.0 is significantly mor…

Thanks for the information and your advice nathaniel. Does the “drop” function reduce the data for the join? I guess not because the data has to be first in memory before the drop function applies, right? Are there any documentations for 1.8.1 migration to 2.0? Thanks

We haven’t focused on this yet. But we are working on these as part of of the InfluxDB 2.0 OSS GA… Our guidance at this point is to dual-write data to both InfluxDB 1.x and 2.x. However, we mentioned that there is a big change coming as of InfluxDB 2.0 Beta 17 – which will re-align the storage en…

[image] InfluxDB OSS 2.0 General Availability Roadmap InfluxDB 2 Introduction This is a message for our community of InfluxDB Open Source users and contributors, from InfluxData engineering. InfluxData is the company behind the popular InfluxDB Time Series Data…

Join big measurements together - Error Bad Gateway

Fluxlang

tim.hall August 20, 2020, 10:31pm 11

Topic		Replies	Views
Poor performance for join(): cpu and memory grows quadratically with row count Fluxlang influxdb , query , flux , join	6	1720	June 30, 2023
Join Data quickly -- InfluxQL 1.8 vs Flux 2.0 performance InfluxDB 2 influxdb , chronograf , flux , join , union	2	1507	January 28, 2022
Flux join() performance InfluxDB 2 influxdb , query , flux	8	985	May 9, 2023
Join function query generates error in Flux Fluxlang influxdb , grafana	4	3057	July 18, 2018
Joining two Measurements Shows No Results Fluxlang influxdb	1	781	January 27, 2021

Join big measurements together - Error Bad Gateway

Related topics