Export data to HTTP post

[boris377]

I am looking to export data to a vendor as it comes into our system. I was writing a kapacitor tickscript - but I cannot edit the structure of the json (slightly annoying) and am having difficulty getting it to post correctly.

Our DB is a big flat table (maybe not the best, but not my idea). We have 3 different sensors coming in (12 tags each sensor, total 36 tags). We need to constantly send the data to a vendor and kapacitor came up because it has a built in httppost function.

Problem:
First when i attempt to query and return the data, it only returns the first sensor (12 ish tags), but then returns blank rows for every other tag type. So i know its attempting, but i feel like there is a column limit in kapacitor or it cannot handle something I want it to do. Example (Fig1) below:

I can tell the lines are different because sn is common across 2 of them and is updating to the correct value. Then i started thinking "is this my lack of understanding of TSDB’s that’s causing this?

So I thought, split this up into each sensor and do 3 separate kapacitor jobs… super annoying and not what i’m supposed to send, but maybe that works… However then it starts sending bad/duplicate subsets of data. It will randomly post with columns 1 and 2, then send the same data with every column. I want it to only post if it sends every column.

The downside of the second approach is that it’s now 3 separate jsons because the post doesn’t have a join method on it for me to merge this all together. Can anyone lend some insight? I’m clearly in over my head with kapacitor and am slowly starting to explore options outside of influx altogether.
edit I feel like telegraf might even be a better choice here?

Ideally I want to make my own json structure in here, but that’s proving… ridiculous. Otherwise if I can get all the data into one json, that would be ideal.

"columns": ["time", "SweepCount", "Uptime", "Uptime_1", "boardTemp", "measurement", "rh", "s0TempPost", "s0TempPre", "s0mag", "s0phase", "s1TempPost", "s1TempPre", "s1mag", "s1phase", "s2TempPost", "s2TempPre", "s2mag", "s2phase", "s3TempPre", "s3mag", "s3phase", "s4TempPost", "s4TempPre", "s4mag", "s4phase", "sn"],
	"values": [
		["2020-07-27T20:59:35.493Z", 118, 10271, 10271, 8681, "EQUIP1", 15.73, 88.1395, 90.3608, 16369700, -5.98295, 87.9217, 88.0523, 15466700, -3.69109, 87.4885, 87.6181, 13641500, -27.205, 87.4022, 3110180, -76.1478, 87.144, 87.5317, 34950.5, -81.1914, 1627],
		["2020-07-27T20:59:35.533Z", null, null, null, null, "EQUIP1", null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null],
		["2020-07-27T20:59:35.563Z", null, 10365, 10365, 100, "EQUIP1", null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, 6237],

[Anaisdg]

Hello @boris377,
I’m sorry you’re having trouble. Yes, Kapacitor is tricky. Have you considered using Flux, Tasks, joins() and http.post?
I think you might have more success using flux than using kapacitor.

[boris377]

I have not, not even sure where to start though. The biggest thing is it needs to constantly stream data which is why Mike and I chose Kapacitor since it can run batch/streaming jobs. However it has been terrible and every time i go a step forward, i find a problem.

Also of note is that i am on the enterprise version, so i do not have flux 2.0, so what I can do MIGHT be limited in flux. I don’t know if the join method is included? When I go to add flux to the

That said if I started writing a tickscript that would just query the 3 rows, join together into one file, but join wasn’t supported in httppost for tickscript that I could find.

Any further ideas? Funny though, we pay for enterprise and can actually do less it seems

P.S. In looking at your solution, “Tasks” is what I am using, these are kapacitor run jobs… so i’m not sure what is different here other than maybe using flux which i don’t think I have the option to?

[rickspencer3]

Hi @boris377,

As I understand it:

Enterprise has Kapacitor support, but not Tasks support. You use Tick Script with Kapacitor.
2.0 has Tasks support, but not Kapacitor support. You use Flux with Tasks.

Essentially, Tasks are 2.0’s answer to Kapacitor.

I think, therefore, that as an Enterprise customer, @Anaisdg suggestion won’t work for you.

[Anaisdg]

Hello @boris377,
Can you please share your TICK script?

[boris377]

var data = batch
|query(‘SELECT “measurement”,“Sn” as “sn”,“Density” as “density”,“Uptime”,“SweepCount”,“S0TempPre” as “s0TempPre”,“S0TempPost” as “s0TempPost”,“S0mag” as “s0mag”,“S0phase” as “s0phase”, “S1TempPre” as “s1TempPre”,“S1TempPost” as “s1TempPost”,“S1mag” as “s1mag”,“S1phase” as “s1phase”,“S2TempPre” as “s2TempPre”
,“S2TempPost” as “s2TempPost”,“S2mag” as “s2mag”,“S2phase” as “s2phase”,“S3TempPre” as “s3TempPre”,“S3TempPost” as “s2TempPost”
,“S3mag” as “s3mag”,“S3phase” as “s3phase”,“S4TempPre” as “s4TempPre”,“S4TempPost” as “s4TempPost”,“S4mag” as “s4mag”
,“S4phase” as “s4phase”,“BoardTemp” as “boardTemp”,“RH” as “rh”,“Density” as “density”,“Oil_temp” as “oiltemp”,“below_short” as “status_TF_below_short”
,“above_damaged” as “status_TF_above_damaged”,“ASIC_over_temp” as “status_TF_ASIC_over_temp”,“fit_failed” as “status_fit_failed_to_converge”
,“RTD_res_below_shorted” as “status_RTD_res_below_shorted”,“RTD_res_above_damaged” as “status_RTD_res_above_damaged”
,“RTD_over_temp” as “status_RTD_over_temp”,“system_error” as “status_system_error”,“Ferrous < 100” as “feLt100”,“Ferrous 100-200” as “fe100_200”,“Ferrous 200-300” as “fe200_300”,“Ferrous 300-400” as “fe300_400”
,“Ferrous 400-500” as “fe400_500”,“Ferrous 500-600” as “fe500_600”,“Ferrous 600-700” as “fe600_700”,“Ferrous >700” as “feGt700”
,“Non-Ferrous < 200” as “nfLt200”,“Non-Ferrous 200-300” as “nf200_300”,“Non-Ferrous 300-400” as “nf300_400”,“Non-Ferrous 400-500” as “nf400_500”
,“Non-Ferrous 500-600” as “nf500_600”,“Non-Ferrous 600-700” as “nf600_700”,“Non-Ferrous 700-800” as “nf700_800”,“Non-Ferrous > 800” as “nfGt800”,“FlowRate” as “flowRate”
FROM “MEM_Sensor”.autogen.equipment’)
// “-e”,
.every(30s)
.period(3000s)
.align()
|httpPost()
.endpoint(‘Poseidon’)
.captureResponse()

So there are 3 sensors in here.the S#'s are the first sensor, the status group second, and the Ferrous/Non-Ferrous third. I expect 3 rows because the timestamps for each sensor are a few hundredths of a second different, but what kapacitor does, is the first example I gave… it throws out the first groups columns for all 3 timestamps.It seems to have a column limit or doesn’t know how to do multiple data columns? Join isn’t an option because HTTP in tickscript doesn’t support.

When I split it into 3 separate queries, it swaps between giving me all the items per sensor and only 2-4 columns from the query depending on when i capture the post.