Raspberry Pi beginner

Telegraf
, ,
1

Hello,

I’m a beginner at all this, and i’ve been trying for weeks to get MQTT data out of Python, into Telegraf, on to Influxdb and out on to a grafana graphics environment.

I have all of the components installed in place. I am successfully transmitting MQTT data out of Python. This is where I start falling down though. The instruction and help I find all over the internet is not proving sufficient for me, so I think i’m needing a more personalised approach to support. I’m hoping that I can find this here as nowhere else is able to help me. I think im getting data into Telegraf, but there is always parcing issues, or formatting issues in my live logging, or virtually no message at all. Sometimes I’ve had “metrics delivered” or words to that effect, to influxdb, but I really don’t know how to check this in Influxdb, and I certainly cannot get anything to grafana.
what is the best approach for me to get much more of a one on one support line on this software where I can show my config files and evidence of my MQTT messages from Python to check for compatibility etc please.

Best Regards,

Chris

2

When you go to http://your-computer-ip-address:8086 and login, then go to the Influx Data Explorer, do you see anything that resembles what you are piping out via Telegraf?

3

Hi Grant, thank you for your reply.

I think you’re directing me to the influxdb webpage, but unfortunately that doesn’t work on a Raspberry Pi running the still advised 32-bit operating system, so I can’t use that for any level of transparency.

I’ll probably have to post up here or some place what I have - config file wise so that people can point me out the error of my ways.

Until I’m in a position to do this, what I can do right now, is explain what I’m trying to do…

I’ve written a Python 3 script which takes in a bunch of variables independently of one another. They are all floats.
I’m of the understanding that they need to be converted into JSON for transmitting to Telegraf, so I do this. When i look at the “type” of the variable after conversion using this command within Python:
convertedfloatvariable = json.dumps(floatvariablename) - they display as <class ‘str’>. The first thing I have a question about is this. Is this correct? Is a JSON variable a string?

Then I’m MQTT publishing the converted JSON variable with an attached TOPIC name out of Python and on to my home network.
Here is an example of one of a complete command for this (I’ve chosen my outside temperature variable to demonstrate here):
topic chosen is ‘HMI/Outside/Temperature’ (This is a string format)
Outside Temperature variable name is ‘OS_Temp’ (This is in JSON format - shows up as “str”)

command written in Python as an example based on the above insertions:

client.publish(topic, OS_Temp, 0)

I know that this is going out because I can subscribe to this topic on my mobile phone using the MQTTool app.

This is where it gets tricky for me because depending on the changes I make to my Telegraf config file (found here on my Raspberry Pi: /etc/telegraf/telegraf.conf) I get different results.

I’ve had the following results:

(1)
[inputs.mqtt_consumer] Error in plugin: metric parse error: expected tag at 1:4: “9.6” (where 9.6 was actually the value I was sending to Telegraf on MQTT)
[outputs.influxdb] Buffer fullness: 0 / 10000 metrics

(2)
[outputs.influxdb] Wrote batch of 1 metrics in 17.743206ms
[outputs.influxdb] Buffer fullness: 0 / 10000 metrics

(3)
I dont have a photograph of this one, but I’ve also had a result where only 1 line is given, which I think was the same as the second line from (1) & (2), but not a first line. It was just silent.

I’ve done that much faffing over weeks and weeks that I’ve had some resemblance of “success” when i look at Influxdb because when I query it with the commands I get some results which look familiar sometimes, but I don’t understand them really:

use homevars (this is the name of my database)

show series
key

mqtt_consumer,HMI=HMI/OS_Temp,host=raspberrypi,influxdb_database=homevars,password=password,username=username
mqtt_consumer,HMI/Outside/Temperature=HMI/Outside/Temperature,host=chrispi
mqtt_consumer,host=chrispi,topic=HMI/Outside/Temperature

etc etc

show measurements
name: measurements
name

mqtt_consumer
tagkey

HMI
HMI/Outside/Temperature
host
influxdb_database
password
topic
username

show field keys
name: mqtt_consumer
fieldKey fieldType

value float

To me, all of this is a bit of jumble, and I’m not sure exactly at what point each of these entities (is entities an acceptable descriptor when you don’t know what you’re talking about?) was created and stored within influxdb.

The only other things I can tell right now, are some of the commands I have set in my Telegraf config file (from some photos on my phone here, so not a complete or exhaustive list by any means)

urls = [“http://127.0.0.1:8086”] (I believe this is the description of my Influxdb instance)
database = “homevars” (this is what I have called my database, which I first created within Influxdb - rightly or wrongly)

I’ve been using these here as these have been providing me the most positive results in the live telegraf logs when i watch them. I’d have expected to have used “json” for data_type but it doesn’t like that at all!?
data_type = “float”
data_format = “value”

precision = “”

debug = true
quiet = false

[inputs.mqtt_consumer]]

servers = [“tcp://192.168.0.51:1883”] (This is my MQTT broker address)

topics = [“HMI/Outside/Temperature”]

I’ve configured nothing in influxdb config file, and the only thing I ever see within Grafana is a reference to “mqtt_consumer” and “value”.

I can of course copy and paste my complete Telegraf config file when I get home from work to show you it complete.

I REALLY need help with this. I don’t have long enough to live to continue to try and fumble about in the dark trying to figure this out by myself. I’ve proven that fact to myself over the past month or so.

Can you help?

4

On an old (2006 era) PC, I installed Raspberry PI OS 32-bit, and installed the latest InfluxDB on it, and it works fine, so I believe you can run it on a Raspberry Pi.

You mentioned that you already installed InfluxDB. What version did you install?

5

I was told the opposite by the internet some place about 32-bit versus 64-bit operating system on a RP for these programs (might not have been explicitly been Influxdb?). I don’t know where now, but there was definitely a recommendation to keep with the 32-bit operating system when first building your Pi too.

As for the version of influx (and Telegraf?) I’ve installed, I don’t know. I’ve got a note of the commands used to install them though, and i’m sure it tells you when you first run them up some place, so i can check that tonight.

6

“On an old (2006 era) PC, I installed Raspberry PI OS 32-bit”

I’m intrigued - how do you install Raspberry O/S on a PC?

Antony.

7

@Pooh I know! I surprised even myself! I did this about two months ago and just installed this over the Windows Vista OS. Raspberry Pi Desktop for PC and Mac – Raspberry Pi

It’s clearly 32-bit.

After that, I went here and did this:

# Ubuntu/Debian
wget https://dl.influxdata.com/influxdb/releases/influxdb2-2.4.0-xxx.deb
sudo dpkg -i influxdb2-2.4.0-xxx.deb

and it’s been running perfectly since then. I only use it for test data and such, but have never received an error message or such.

8

Ah, so it’s actually x86 Debian, with the Raspberry Desktop GUI added.

I initially thought you meant you had found a way of installing the standard
Raspberry (for ARM) O/S image onto an x86 or AMD64 CPU.

So, once you’ve installed “Debian with Raspberry Pi Desktop”, and added
InfluxDB, you’re effectly simply running Influx on Debian on an x86 32-bit
system.

Thanks for the clarification,

Antony.

9

@Rissy

Recently, we’ve released a Cloud Native MQTT connector that allows users to connect their MQTT data to InfluxDB without using telegraf, assuming you’re a paying cloud user, this capability is available to you. Can you try out the non-telegraf method and let us know how it goes?

Here is some documentation that can help you get started. https://docs.influxdata.com/influxdb/cloud/write-data/no-code/native-subscriptions/?t=JSON

10

Hi Amy,

Thanks for this information. I’ve actually had some success in the last couple of days. I was yet to post this up. I was keen to replicate as close as possible, a similar setup we have at work. So on this basis, as I know they have telegraf involved, I would still have pursued the avenue i’ve gone down. Once i’m fully familiar with the way i have it now, then i may venture down alternative paths as an experiment.

I’ve really not found the whole mqtt-telegraf-influxdb-grafana solution an easy one to figure out. It’s definitely not a thing for the average lamen to pick up and get running. I’m still not even sure what changes i could make to my telegraf config to streamline it without breaking it. It’s not intuitive. The fault is with myself too, or mostly I guess. I don’t know enough about the different mechanisms required to make this stuff work. As it turns out, one of the things which was stopping my setup from talking, was not including “{” and “}” in my message packet over MQTT. It may seem obvious to someone more educated or experienced than I, but for me, something like this was such a simple thing which caused me so much fruastration and headache and puzzlement on what was going wrong and why I was continually suffering parcing errors (I even had to google “parcing” to understand what it was this word was trying to tell me!) Such a simple thing could be SO EASILY overcome if someone, somewhere would put a message-throughlife working example of such a thing on the internet, or youtube, for people like myself to see it all hanging together and working but seeing each stage fully explained, and if individuals are missing certain aspects of the example in their knowlege, then at least they’d be drawn towards understanding what it was they still had to learn or understand. I simply didn’t know what I didn’t know.

I may be temped to put something together myself for the wider community. There must be lots of people out there who’d want to do something like I’ve done, but the learning curve is just too steep. I’ve only come through (only just) because i’m an extremely tenacious person. I don’t like it when something doesn’t work, and i don’t know why.

11

Hello people,

As advised above, I’ve had some success with this. Thanks to grant1 for his time spent with me over the weekend on this.

It highlighted some additional questions i have if anyone can field these for me please?

  1. Is it possible to delete unused “measurements” from influx, since I’ve done that much messing around trying to get this working over the past month or so, that I really need to do a bit of a clear out of unwanted content. (not sure why this item is titled as this because it’s not a measurement, it’s more of a data subset title from what I can make out)

  2. Is there a way of determining how large I’m happy for the database to be (period of stored history) so that I don’t risk running out of space on my microSD card?

  3. Can you make the X and Y axis text larger on screen within Grafana? - right now the units are tiny on-screen.

I’ll leave it there for now. Thanks!