I am trying to understand how the Telegraf plug-in works. If multiple applications are sending Span events for a given trace, at what point does Telegraf consider the trace complete? Does a span need to identify itself as the final span in a trace? Or is there a time factor (i.e., no spans after a given time elapses means the trace is complete)?
Am I fundamentally missing something? I found no indication on how a trace is ended on any web searches.
Are you using the zipkin plugin? A trace doesn’t have a concept of being complete, it’s just a collection of spans within a time range.
Thanks Daniel. I’m not using anything yet (just influxdb). I thought an end span to the trace is needed for the trace to be summarized, but I guess the trace is just stored as a collection of spans with a unifying trace id (or rolling up to the same parent span). If that is the case, I am wondering if running analytics on the trace can be performant - such as: show me the 95 percentile latency of a particular trace throughout the day, when the trace occurs 0.5 - 1 million times a day.
Hello! Daniel is right. There is no concept of “end” inside a trace. You can keep adding things to a trace event after a year if it is still stored.
You can create an ‘end’ span if you think that tracking it can be useful.
I am wondering if running analytics on the trace can be performant
You need to remember that a trace is just a set of points with a different correlation, usually trace_id to correlate spans between each other and span_id to identify a single span — nothing more than a point with the concept of duration. Performing aggregation and calculation on them can be very useful. we do things like "how much time service-a spends speaking with service-b.
This is the use case I am trying to solve:
Let’s say a set of 10 services creates 20-30 unique traces. By unique I mean a combination of spans and presence of a tag (or tags) within a span. Let’s say there are one million such traces per day - each one classifiable as one of these unique traces. How feasible is it to expect a front end to graph the latency of a particular (unique) trace (end to end) or to set an SLO (real time alert) on the trace?
While the information in the span is important, my first level problem is to understand the end-to-end trace as a composite event / KPI and set SLOs on them.
I appreciate the response. Thank you, Alan
I think there is not a lot more to say other than trying. We use https://github.com/influxdata/jaeger-store to store traces in InfluxDB, our number are different and Jaeger offers sampling as well if you would like to exclude a % of traces.
When you will have a set of points in influxdb you can use the normal features provided by the stack such as continuous queries to do the SLO part.
Thanks! We certainly will try…
