I’m new to Linux, InfluxDB, and telegraf so please bear with me. I am running Ubuntu 22.04, InfluxDB local, and Telegraf 1.30.0. I’m using a telegraf plugin for Influx called inputs.neptune_apex.
I’m sure this isn’t the technical terms, but what it’s doing is pulling xml data from sensors on a local machine and storing it. All is fine for the first 10 minutes and then all of a sudden telegraf starts sending "Error in plugin: http GET failed: Get “http://reef.local/cgi-bin/status.xml”: context deadline exceeded (Client.Timeout exceeded while awaiting headers).
Now, here’s the deal. On the machine running Influx and Telegraf, it takes forever to access http://reef.local/cgi-bin/status.xml after 10 minutes. In the first 10 minutes, it’s immediate. After 10 minutes, it either doesn’t load, comes back with "error on line 69 at coumbus 18: expected ‘>’ " or connection refused. But on other computers on the network, there is no issues accessing the data.
I’m assuming there is some sort of issue with the network settings on the Ubuntu machine but I’ve searched for days and I can’t figure it out. I’m assuming that’s because I don’t have any idea what I’m doing and I’m not even searching the right thing. reef.local has a static ip address which I’ve added to /etc/hosts. That did speed up access during the first 10 minutes, but didn’t change the 10 minute timeout deal.
traceroute to reef.local even when I’m getting the client.timeout is around 7.3 ms, if that means anything to anyone.
Additionally, I have a homeassistant vm that runs on the same machine and will pull the same data. I turned if off thinking maybe the localhost was being overwhelmed, but it will still pull data from the source during this time. Sometimes, a few of the sensors are missing the normal data on the pull, but it’s accessible. Not sure if that helps or not.
Even if you can’t tell me exactly what to do, if you could point me in the direction of where to look I’d greatly appreciate it! Thanks.
plugin for Influx called inputs.neptune_apex
I assume this is a plugin you are including as an external plugin or something you yourself developed?
context deadline exceeded (Client.Timeout exceeded while awaiting headers).
comes back with "error on line 69 at coumbus 18: expected ‘>’ " or connection refused
Because I don’t know what this plugin in, it is hard to say what might be going on. However, the context deadline exceeded is a generic error in Go when it comes to networking. It can mean a whole host of networking issues like a firewall or proxy got in the way, DNS is down, your connection is blocked, etc.
This could also be due to the fact that you are getting data back and your are unable to parse the message. Hence the expected comment. It could also have something to do with how the plugin is managing the connections with respects to some timeout.
The fact that it happens after 10mins does seem to indicate some timeout is getting thrown somewhere along the way. Whether that is a firewall, the client’s connection, or the server is unknown.
My apologies, I said plugin but it’s a telegraf configuration. If you go to load data and then create configuration, then search for apex it’s Neptune Apex there.
The data that’s coming back should be exactly the same every time.
Ha I don’t think I’ve ever looked at this plugin! So looking at the plugin it is expecting XML, which would explain this message:
After 10 minutes, it either doesn’t load, comes back with "error on line 69 at coumbus 18: expected ‘>’
That would seem to indicate that it got invalid XML. For this one, what I might suggest you use curl/wget against the endpoint see what you are getting back. The plugin is basically hitting http://apex.local/cgi-bin/status.xml where apex.local is your server name. This might also get us insight into the connection refused messages.
$ curl http://reef.local/cgi-bin/status.xml
<?xml version="1.0"?>
<status software="5.11L_9F23" hardware="1.0">
<hostname>Reef</hostname>
<serial>AC5:21506</serial>
<timezone>-4.00</timezone>
<date>03/18/2024 15:14:27</date>
<power><failed>03/15/2024 13:20:25</failed>
<restored>03/15/2024 13:20:50</restored></power>
<probes>
<probe>
<name>SumpT</name> <value>78.9 </value>
<type>Temp</type></probe><probe>
<name>pH</name> <value>8.11 </value>
<type>pH</type></probe><probe>
<name>SkimF</name> <value> 0 </value>
</probe><probe>
<name>SumpLV</name> <value> 0.0 </value>
</probe><probe>
<name>MP40-backA</name> <value>0.2 </value>
</probe><probe>
<name>MP-40LeftA</name> <value>0.1 </value>
</probe><probe>
<name>MP-40RA</name> <value>0.4 </value>
</probe><probe>
<name>LightsA</name> <value>2.3 </value>
</probe><probe>
<name>HydrosWaveA</name> <value>0.0 </value>
</probe><probe>
<name>DSolenoidA</name> <value>0.0 </value>
</probe><probe>
<name>HandpumpA</name> <value>0.0 </value>
</probe><probe>
<name>DHeaterA</name> <value>5.8 </value>
</probe><probe>
<name>MP40-backW</name> <value> 1 </value>
</probe><probe>
<name>MP-40LeftW</name> <value> 7 </value>
</probe><probe>
<name>MP-40RW</name> <value> 29 </value>
</probe><probe>
<name>LightsW</name> <value> 259 </value>
</probe><probe>
<name>HydrosWaveW</name> <value> 1 </value>
</probe><probe>
<name>DSolenoidW</name> <value> 1 </value>
</probe><probe>
<name>HandpumpW</name> <value> 1 </value>
</probe><probe>
<name>DHeaterW</name> <value> 664 </value>
</probe><probe>
<name>Volt_3</name> <value>115 </value>
</probe><probe>
<name>MainT</name> <value>78.3 </value>
<type>Temp</type></probe><probe>
<name>Salini</name> <value> 0.2 </value>
<type>Cond</type></probe><probe>
<name>AirTmp</name> <value>80.2 </value>
<type>Temp</type></probe><probe>
<name>pHCaRx</name> <value>6.10 </value>
<type>pH</type></probe><probe>
<name>ReturnPumpA</name> <value>0.9 </value>
</probe><probe>
<name>UVlightA</name> <value>0.0 </value>
</probe><probe>
<name>SkimmerBubA</name> <value>0.7 </value>
</probe><probe>
<name>SkimmerFeedA</name> <value>0.3 </value>
</probe><probe>
<name>ATOA</name
The above was from the Ubuntu device after the “timeout”.
This is from a windows device, also during the “timeout.” There’s a lot more here. There must be something going on with Ubuntu but I’m lost on where to go from here!
curl 192.168.4.114/cgi-bin/status.xml
<?xml version="1.0"?>
<status software="5.11L_9F23" hardware="1.0">
<hostname>Reef</hostname>
<serial>AC5:21506</serial>
<timezone>-4.00</timezone>
<date>03/18/2024 15:46:13</date>
<power><failed>03/15/2024 13:20:25</failed>
<restored>03/15/2024 13:20:50</restored></power>
<probes>
<probe>
<name>SumpT</name> <value>79.5 </value>
<type>Temp</type></probe><probe>
<name>pH</name> <value>8.12 </value>
<type>pH</type></probe><probe>
<name>SkimF</name> <value> 0 </value>
</probe><probe>
<name>SumpLV</name> <value> 0.0 </value>
</probe><probe>
<name>MP40-backA</name> <value>0.2 </value>
</probe><probe>
<name>MP-40LeftA</name> <value>0.0 </value>
</probe><probe>
<name>MP-40RA</name> <value>0.6 </value>
</probe><probe>
<name>LightsA</name> <value>2.3 </value>
</probe><probe>
<name>HydrosWaveA</name> <value>0.0 </value>
</probe><probe>
<name>DSolenoidA</name> <value>0.0 </value>
</probe><probe>
<name>HandpumpA</name> <value>0.0 </value>
</probe><probe>
<name>DHeaterA</name> <value>5.8 </value>
</probe><probe>
<name>MP40-backW</name> <value> 2 </value>
</probe><probe>
<name>MP-40LeftW</name> <value> 3 </value>
</probe><probe>
<name>MP-40RW</name> <value> 37 </value>
</probe><probe>
<name>LightsW</name> <value> 259 </value>
</probe><probe>
<name>HydrosWaveW</name> <value> 1 </value>
</probe><probe>
<name>DSolenoidW</name> <value> 1 </value>
</probe><probe>
<name>HandpumpW</name> <value> 1 </value>
</probe><probe>
<name>DHeaterW</name> <value> 670 </value>
</probe><probe>
<name>Volt_3</name> <value>116 </value>
</probe><probe>
<name>MainT</name> <value>79.0 </value>
<type>Temp</type></probe><probe>
<name>Salini</name> <value> 0.2 </value>
<type>Cond</type></probe><probe>
<name>AirTmp</name> <value>80.8 </value>
<type>Temp</type></probe><probe>
<name>pHCaRx</name> <value>6.10 </value>
<type>pH</type></probe><probe>
<name>ReturnPumpA</name> <value>0.9 </value>
</probe><probe>
<name>UVlightA</name> <value>0.0 </value>
</probe><probe>
<name>SkimmerBubA</name> <value>0.7 </value>
</probe><probe>
<name>SkimmerFeedA</name> <value>0.3 </value>
</probe><probe>
<name>ATOA</name> <value>0.0 </value>
</probe><probe>
<name>CaRxPowerA</name> <value>0.2 </value>
</probe><probe>
<name>AlgaeTSA</name> <value>0.4 </value>
</probe><probe>
<name>FragReturnA</name> <value>0.0 </value>
</probe><probe>
<name>ReturnPumpW</name> <value> 101 </value>
</probe><probe>
<name>UVlightW</name> <value> 4 </value>
</probe><probe>
<name>SkimmerBubW</name> <value> 80 </value>
</probe><probe>
<name>SkimmerFeedW</name> <value> 21 </value>
</probe><probe>
<name>ATOW</name> <value> 2 </value>
</probe><probe>
<name>CaRxPowerW</name> <value> 21 </value>
</probe><probe>
<name>AlgaeTSW</name> <value> 42 </value>
</probe><probe>
<name>FragReturnW</name> <value> 19 </value>
</probe><probe>
<name>Volt_12</name> <value>116 </value>
</probe><probe>
<name>AlkTri</name> <value>10.07</value>
<type>alk</type></probe><probe>
<name>CalTri</name> <value> 520 </value>
<type>ca</type></probe><probe>
<name>MgTri</name> <value>1347 </value>
<type>mg</type></probe><probe>
<name>FreeA</name> <value>0.0 </value>
</probe><probe>
<name>FreeA</name> <value>0.0 </value>
</probe><probe>
<name>FreeA</name> <value>0.0 </value>
</probe><probe>
<name>CaRxSolenoidA</name> <value>0.0 </value>
</probe><probe>
<name>Free_5A</name> <value>0.0 </value>
</probe><probe>
<name>Pump_14_6A</name> <value>0.0 </value>
</probe><probe>
<name>FreeA</name> <value>0.0 </value>
</probe><probe>
<name>Fan_14_8A</name> <value>0.0 </value>
</probe><probe>
<name>FreeW</name> <value> 0 </value>
</probe><probe>
<name>FreeW</name> <value> 0 </value>
</probe><probe>
<name>FreeW</name> <value> 0 </value>
</probe><probe>
<name>CaRxSolenoidW</name> <value> 5 </value>
</probe><probe>
<name>Free_5W</name> <value> 0 </value>
</probe><probe>
<name>Pump_14_6W</name> <value> 0 </value>
</probe><probe>
<name>FreeW</name> <value> 0 </value>
</probe><probe>
<name>Fan_14_8W</name> <value> 0 </value>
</probe><probe>
<name>Volt_14</name> <value>124 </value>
</probe></probes>
<outlets>
<outlet>
<name>SndAlm_I6</name>
<outputID>0</outputID>
<state>AOF</state>
<deviceID>base_Alarm</deviceID>
</outlet>
<outlet>
<name>SndWrn_I7</name>
<outputID>1</outputID>
<state>AOF</state>
<deviceID>base_Warn</deviceID>
</outlet>
<outlet>
<name>EmailAlarm</name>
<outputID>2</outputID>
<state>AOF</state>
<deviceID>base_email</deviceID>
</outlet>
<outlet>
<name>Email2Alm_I9</name>
<outputID>3</outputID>
<state>AOF</state>
<deviceID>base_email2</deviceID>
</outlet>
<outlet>
<name>Alarm_2_2</name>
<outputID>4</outputID>
<state>AOF</state>
<deviceID>2_2</deviceID>
</outlet>
<outlet>
<name>MP40-back</name>
<outputID>5</outputID>
<state>OFF</state>
<deviceID>3_1</deviceID>
</outlet>
<outlet>
<name>MP-40Left</name>
<outputID>6</outputID>
<state>AON</state>
<deviceID>3_2</deviceID>
</outlet>
<outlet>
<name>MP-40R</name>
<outputID>7</outputID>
<state>AON</state>
<deviceID>3_3</deviceID>
</outlet>
<outlet>
<name>Lights</name>
<outputID>8</outputID>
<state>ON</state>
<deviceID>3_4</deviceID>
</outlet>
<outlet>
<name>HydrosWave</name>
<outputID>9</outputID>
<state>OFF</state>
<deviceID>3_5</deviceID>
</outlet>
<outlet>
<name>DSolenoid</name>
<outputID>10</outputID>
<state>AOF</state>
<deviceID>3_6</deviceID>
</outlet>
<outlet>
<name>Handpump</name>
<outputID>11</outputID>
<state>OFF</state>
<deviceID>3_7</deviceID>
</outlet>
<outlet>
<name>DHeater</name>
<outputID>12</outputID>
<state>AON</state>
<deviceID>3_8</deviceID>
</outlet>
<outlet>
<name>LinkA_3_9</name>
<outputID>13</outputID>
<state>AOF</state>
<deviceID>3_9</deviceID>
</outlet>
<outlet>
<name>SumpHigh</name>
<outputID>14</outputID>
<state>AOF</state>
<deviceID>Cntl_A1</deviceID>
</outlet>
<outlet>
<name>Fill-2-Part</name>
<outputID>15</outputID>
<state>OFF</state>
<deviceID>Cntl_A2</deviceID>
</outlet>
<outlet>
<name>Extra</name>
<outputID>16</outputID>
<state>OFF</state>
<deviceID>2_1</deviceID>
</outlet>
<outlet>
<name>ATO-Empty</name>
<outputID>17</outputID>
<state>OFF</state>
<deviceID>Cntl_A3</deviceID>
</outlet>
<outlet>
<name>ReturnSpeed</name>
<outputID>18</outputID>
<state>1ReturnRamp</state>
<deviceID>7_1</deviceID>
</outlet>
<outlet>
<name>VarSpd2_7_2</name>
<outputID>19</outputID>
<state>OFF</state>
<deviceID>7_2</deviceID>
</outlet>
<outlet>
<name>VarSpd3_7_3</name>
<outputID>20</outputID>
<state>OFF</state>
<deviceID>7_3</deviceID>
</outlet>
<outlet>
<name>SkimmerSpeed</name>
<outputID>21</outputID>
<state>OFF</state>
<deviceID>7_4</deviceID>
</outlet>
<outlet>
<name>Skimmer0-10</name>
<outputID>22</outputID>
<state>OFF</state>
<deviceID>7_5</deviceID>
</outlet>
<outlet>
<name>WhtLED_7_6</name>
<outputID>23</outputID>
<state>OFF</state>
<deviceID>7_6</deviceID>
</outlet>
<outlet>
<name>AWC-New</name>
<outputID>24</outputID>
<state>TBL</state>
<deviceID>8_1</deviceID>
</outlet>
<outlet>
<name>AWC-Waste</name>
<outputID>25</outputID>
<state>TBL</state>
<deviceID>8_2</deviceID>
</outlet>
<outlet>
<name>FMM2-24v</name>
<outputID>26</outputID>
<state>OFF</state>
<deviceID>4_1</deviceID>
</outlet>
<outlet>
<name>DOS-Alk</name>
<outputID>27</outputID>
<state>OFF</state>
<deviceID>9_1</deviceID>
</outlet>
<outlet>
<name>DOS-Calc</name>
<outputID>28</outputID>
<state>OFF</state>
<deviceID>9_2</deviceID>
</outlet>
<outlet>
<name>LeftAuxTk</name>
<outputID>29</outputID>
<state>TBL</state>
<deviceID>10_1</deviceID>
</outlet>
<outlet>
<name>RightAuxTk</name>
<outputID>30</outputID>
<state>TBL</state>
<deviceID>11_1</deviceID>
</outlet>
<outlet>
<name>ReturnPump</name>
<outputID>31</outputID>
<state>AON</state>
<deviceID>12_1</deviceID>
</outlet>
<outlet>
<name>UVlight</name>
<outputID>32</outputID>
<state>ON</state>
<deviceID>12_2</deviceID>
</outlet>
<outlet>
<name>SkimmerBub</name>
<outputID>33</outputID>
<state>AON</state>
<deviceID>12_3</deviceID>
</outlet>
<outlet>
<name>SkimmerFeed</name>
<outputID>34</outputID>
<state>AON</state>
<deviceID>12_4</deviceID>
</outlet>
<outlet>
<name>ATO</name>
<outputID>35</outputID>
<state>AON</state>
<deviceID>12_5</deviceID>
</outlet>
<outlet>
<name>CaRxPower</name>
<outputID>36</outputID>
<state>AON</state>
<deviceID>12_6</deviceID>
</outlet>
<outlet>
<name>AlgaeTS</name>
<outputID>37</outputID>
<state>AON</state>
<deviceID>12_7</deviceID>
</outlet>
<outlet>
<name>FragReturn</name>
<outputID>38</outputID>
<state>AON</state>
<deviceID>12_8</deviceID>
</outlet>
<outlet>
<name>Macrolights</name>
<outputID>39</outputID>
<state>OFF</state>
<deviceID>12_9</deviceID>
</outlet>
<outlet>
<name>Cheatogrow</name>
<outputID>40</outputID>
<state>ON</state>
<deviceID>12_10</deviceID>
</outlet>
<outlet>
<name>LinkB_3_10</name>
<outputID>41</outputID>
<state>AOF</state>
<deviceID>3_10</deviceID>
</outlet>
<outlet>
<name>Trident_13_3</name>
<outputID>42</outputID>
<state>AOF</state>
<deviceID>13_3</deviceID>
</outlet>
<outlet>
<name>Alk_13_4</name>
<outputID>43</outputID>
<state>AOF</state>
<deviceID>13_4</deviceID>
</outlet>
<outlet>
<name>Alarm_4_2</name>
<outputID>44</outputID>
<state>AOF</state>
<deviceID>4_2</deviceID>
</outlet>
<outlet>
<name>Cheato-Over</name>
<outputID>45</outputID>
<state>AOF</state>
<deviceID>Cntl_A4</deviceID>
</outlet>
<outlet>
<name>Trident</name>
<outputID>46</outputID>
<state>OFF</state>
<deviceID>Cntl_A5</deviceID>
</outlet>
<outlet>
<name>PowerOutage</name>
<outputID>47</outputID>
<state>AOF</state>
<deviceID>Cntl_A6</deviceID>
</outlet>
<outlet>
<name>FeedDisplay</name>
<outputID>48</outputID>
<state>AOF</state>
<deviceID>Cntl_A7</deviceID>
</outlet>
<outlet>
<name>Free</name>
<outputID>49</outputID>
<state>OFF</state>
<deviceID>14_1</deviceID>
</outlet>
<outlet>
<name>Free</name>
<outputID>50</outputID>
<state>OFF</state>
<deviceID>14_2</deviceID>
</outlet>
<outlet>
<name>Free</name>
<outputID>51</outputID>
<state>OFF</state>
<deviceID>14_3</deviceID>
</outlet>
<outlet>
<name>CaRxSolenoid</name>
<outputID>52</outputID>
<state>AON</state>
<deviceID>14_4</deviceID>
</outlet>
<outlet>
<name>Free_5</name>
<outputID>53</outputID>
<state>OFF</state>
<deviceID>14_5</deviceID>
</outlet>
<outlet>
<name>Pump_14_6</name>
<outputID>54</outputID>
<state>OFF</state>
<deviceID>14_6</deviceID>
</outlet>
<outlet>
<name>Free</name>
<outputID>55</outputID>
<state>OFF</state>
<deviceID>14_7</deviceID>
</outlet>
<outlet>
<name>Fan_14_8</name>
<outputID>56</outputID>
<state>OFF</state>
<deviceID>14_8</deviceID>
</outlet>
<outlet>
<name>LinkA_14_9</name>
<outputID>57</outputID>
<state>AOF</state>
<deviceID>14_9</deviceID>
</outlet>
<outlet>
<name>LinkB_14_10</name>
<outputID>58</outputID>
<state>AOF</state>
<deviceID>14_10</deviceID>
</outlet>
<outlet>
<name>Feeder</name>
<outputID>59</outputID>
<state>AOF</state>
<deviceID>15_1</deviceID>
</outlet>
</outlets></status>
Hmm this would be a question for the apex software as to why you are getting different responses depending on the system. The Ubuntu output is invalid XML as well, which would confirm my susplicions about this error message:
"error on line 69 at coumbus 18: expected ‘>’
In fact the Ubuntu output ends on line 69 and does not close the name tag, nor the probes tag.
Do you have any idea what direction to go within Ubuntu that it would work properly for 10 minutes after a reset but then stop working? I imagine it is some sort of Ubuntu networking/network manager configuration rather than an Apex issue. Android, Windows and Mac all get good results so I’m guessing there’s something wrong with Ubuntu that isn’t pulling it. It’s available just as a webpage as well and HomeAssistant VM generally pull the data as well.
My next step would be to get a network trace on both systems and compare them.
Ok, now my noob is going to show through rather dramatically. I punted for now and closed the console, assuming this would stop data being transmitted.
My previous process was to start device. Telegraf and InfluxDB (2.75) would start as a service. I have a telegraf configuration. I would then open a console and pass export INFLUX_TOKEN and then start telegraf --config http://localhost:8086/api/vs/telegrafs/xxxxxxxxxxx. This is where for 10 minutes I’d see it correctly passing data in the logs to Influx and then it would start timing out.
Now for the confusion. I went into Influx to view the data, and apparently the entire time it was actually receiving data regularly every 10 minutes. Occasionally when I’d restart telegraf I’d have brief intervals of every 15 seconds.
Photo of InfluxDB data graph for some values
Does this change anything? Now I’m wondering if I was even doing the right process to begin with. Thanks for any assistance!
If telegraf is run as a service and then you are running it by hand, that would cause two to run at the same time, no?
In case anyone happens to stumble across this thread and are looking for answers, I’m using a laptop. I always left the laptop open but wanted to close the lid and stick it on a shelf unless I needed it. Apparently, something was sleeping when the screen would turn off because accidentally solved this problem by doing this:
To disable entering the sleep mode I had to edit the /etc/systemd/logind.conf file and modify the line:
#HandleLidSwitch=suspend
to
HandleLidSwitch=ignore
Additionally, ensure that the file also has this line:
LidSwitchIgnoreInhibited=no
Then restart the OS via:
sudo service systemd-logind restart
1 Like
