Use output plugin config variables in my custom json serializer

I currently am writing my custom json serializer in combination with my custom file output plugin. (For usecase explanation see bottom)

Now i want to use the config variables from the output plugin inside my json serializer.

E.g. here are some of the config variables:

Files               []string          `toml:"files"`
RotationInterval    internal.Duration `toml:"rotation_interval"`
RotationMaxSize     internal.Size     `toml:"rotation_max_size"`
RotationMaxArchives int               `toml:"rotation_max_archives"`
myCustomVariable    int               `toml:"my_custom_variable"`
...

Now i’d like to use myCustomVariable inside the json.go Serializer.
What would be a good way to achieve this?

Example Solution:
Would i create a Serializer attribute with a new set-function in my json serializer like this?

func (s *Serializer) setMyVar(int myVar) () {
    s.myVar = myVar
}

type Serializer struct {
    TimestampUnits time.Duration
    int myVar
}

And call this in the output plugin in the SetSerializer-function?

func (f *File) SetSerializer(serializer serializers.Serializer) {
    f.serializer = serializer   
    f.serializer.setMyVar(f.myCustomVariable)
}

I thought what i outlined above would be a good way, but then i am messing with the serializer signature: functions/attributes and this might entail doing some further configurations in other files to register these changes (and i do not know where i need to do this)

Disclaimer: i was able to create my own json-serializer and my own file-output plugin, but ONLY when they use the same function/attribute signatures.

Is there some way better how i can access these config-variables from inside the serializer?

Is is even okay that i do this? If not i’d be very happy to get some suggestions how to do it the right/intended way.

Usecase explanation: (indirectly relevant for question)

My usecase is, i want that the metrics i receive to be NOT appended to a output file, but

that the metrics will replace old metrics. Since i only care about the last metrics for each.

E.g. my output file metrics.json contains the following:

...
"metric1": 18.9,
"metric2": 20.9,
"metric3": 1.0,
...

if i now receive a new value: “metric1”: 8.5

i want that the my metrics.json afterwards looks like this (replaced):

...
"metric1": 8.5,
"metric2": 20.9,
"metric3": 1.0,
...

and NOT like this (appended, as it is the case with the default json serializer and file output):

...
"metric1": 18.9,
"metric2": 20.9,
"metric3": 1.0,
"metric1": 8.5,
...

Thanks for reading any suggestions appreciated.

I don’t have a real solution for this, as Telegraf is used to capture series of data and store/send them, the “ovverride the latest” is a use case I’ve never seen so far, and to me, telegraf is not the tool for this. it’s something managed by whoever access/query the data, and not by the gatherer itelf.

here are some plugins that may help you:

The first 2 might help you reducing “continuous append”, but you can also play with the gathering frequency itself by specifying the interval parameter at plugin level (that’s up to the case).
Telegraf itself is used.

with the Exec and Execd you can do more or less whatever you want as you run an executable (which can do whatever you want)

1 Like

Plugins are meant to be independent, so there’s not a good way to access variables of one plugin from another. You shouldn’t need to share something like myCustomVariable across plugins. If you do it likely means you have something architecturally wrong in your design.

It sounds like your new output will need to keep a copy of the latest of each metric it receives, then on the flush interval, write them all to the output file. The valuecounter aggregator does a similar job. It keeps track of metrics that pass by and outputs aggregated metrics on a separate interval. Maybe it would be helpful to check out how it works.

1 Like

I basically solved my usecase by storing my metrics in memory inside the json.go serializer.
I also use the batch-write mechanism of the file.go output plugin.

So my json-serializer receives a new batch of metrics from file.go → checks the current data in memory if metrics have to be replaced / or not → returns the data from memory.

Thanks for all suggestions, they were helpful.