Newbie - data ingestion

1

I have sensors producing burst of data (32 samples per burst) in every 2ms. I have 32 of them. 1024 samples per 2ms. I have 1024 as the batch size. However, each batch insertion take about 15-20ms. I must insert within 2ms. I am currently using influx 2.0.4. I have no constrain on memory and processor (at this moment, may be this could be issue). I really appreciate any help. What is the best practice for real time scenarios?

I have 2 tags, 2 float value and timestamp. I have retention policy of 1 hr.

2

I’m not quite clear what your actual question is.
What exactly do you mean by “real time”? From where to where?
Do you use Telegraf? If so, which plugins?
Or a client library?

3

Where are you trying to store this data - to a hard disk, SSD, RAMdisk?

Also, how big is a “sample”? You have 32 x 32 samples every 2ms, but how much
data is this, in bytes?

It might also help to give us an idea of what structure this data has, so we
know what Influx needs to do with it.

Regards,

Antony.

4

For me, the real problem is not yet clearly outlined.

  • Is it the timestamp? Is it recorded too inaccurately?
  • Is it the sheer mass of data per time? If so, how much are we talking about?
  • Does the data really need to be in InfluxDB after 2ms? If so, why? What is then attached to it at the back for further processing?
5

Thank you for the response.

Data is stored in SSD. Below is the data structure. I am using C# infux client. I am running Influx as Docker container on the IoT Edge gateway. I didnt see any resource issues (memory, disk, and processor)


POCO

Measurement{
        [Column(IsTimestamp = true)] 
        public DateTime Time { get; set; }

        [Column("AssetId", IsTag = true)]
        public string AssetId { get; set; }

        [Column("ChannelId", IsTag = true)]
        public string ChannelId { get; set; }

        [Column("X")]
        public float X { get; set; }

         [Column("Y")]
        public float Y { get; set; }

        [Column("Z")]
        public float Z { get; set; }
}
.....
var measurements = List<Measurements>()
....

await writeClientAsync.WriteMeasurementsAsync(dbName,dublinOrg.Name, WritePrecision.Ms,measurements); 
// this line take 15ms to 25ms for ~3000 list items. When I batch 10 samples together - 35000 items, it takes over 600ms.

//this is how i create a client in service constructor

  const string host = "http://localhost:8086";
                const string bucket = "abc123";
                var options = new InfluxDBClientOptions.Builder()
                .Url(host)
                .AuthenticateToken( InfluxToken.ToCharArray())
                .Org(Org.Name)
                .Bucket(bucket)
                .Build();
                influxDBClient = InfluxDBClientFactory.Create(options);
               
                writeClientAsync = influxDBClient.GetWriteApiAsync();

I also tried WriteRecordsAsync. I followed c# client examples provided with c# client source code in the github. Meantime, I am going to use HttpClient instead of influx client and test performance.

list contains about 35,000 items.

Use case: I have x sensors (currently 32 sensors like temp, pressure, etc) my use case need a sampling in every 2ms and read values from sensors. I need to store this data before next sample arrives, that is 2ms.

What is the fastest way for insertion? I never used telegraph. Is it what I should use in my use case? Many thanks.

6

Unfortunately the sourcecode is not formatted properly, so I can’t see that much.
Please post code snippets in markdown format here in the forum:

```csharp
put the C# code here
```

But apart from that, 600ms is way too long.
Maybe you are not using the C# client library correctly.
But someone who knows the C# library better would have to help.

7

I updated my response based on your feedback.

8

I am not familiar with C#.
But could it be that you create a new instance of the client each time you write? That would be very inconvenient.
But there should better help someone who knows the C# client.

9

Client, WriteApi instance is only created once at the constructor.

10

@Antony, Thanks. I posted more details and code in a following post.