Writing into influxdb cloud instance gives read timeout

Client SDKs
, ,
1

Hello,

I’m using requests library for python to save points into influxdb and we are suffering timeouts.
We implemented tenacity with 5 retries and we use requests with 10 second timeout, but still we are suffering from timeouts.

What could be going on?

The host is marty-300d7f91.influxcloud.net

Thanks in advance

File "/env/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/env/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "/opt/python3.7/lib/python3.7/http/client.py", line 1369, in getresponse
    response.begin()
  File "/opt/python3.7/lib/python3.7/http/client.py", line 310, in begin
    version, status, reason = self._read_status()
  File "/opt/python3.7/lib/python3.7/http/client.py", line 271, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/opt/python3.7/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
  File "/opt/python3.7/lib/python3.7/ssl.py", line 1071, in recv_into
    return self.read(nbytes, buffer)
  File "/opt/python3.7/lib/python3.7/ssl.py", line 929, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/env/local/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/env/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 725, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/env/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 403, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/env/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 735, in reraise
    raise value
  File "/env/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/env/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 428, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/env/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 336, in _raise_timeout
    self, url, "Read timed out. (read timeout=%s)" % timeout_value
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='marty-300d7f91.influxcloud.net', port=8086): Read timed out. (read timeout=10)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker_v2.py", line 449, in run_background_function
    _function_handler.invoke_user_function(event_object)
  File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker_v2.py", line 268, in invoke_user_function
    return call_user_function(request_or_event)
  File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker_v2.py", line 265, in call_user_function
    event_context.Context(**request_or_event.context))
  File "/user_code/main.py", line 10, in function_entry_point
    return collector_agent(event, context)
  File "/user_code/data_ingestion/app.py", line 89, in collector_agent
    [datapoint],
  File "/user_code/data_ingestion/app.py", line 181, in write_datapoint_to_influx_db
    stop=(stop_after_delay(10) | stop_after_attempt(5)), reraise=True
  File "/env/local/lib/python3.7/site-packages/tenacity/__init__.py", line 382, in __iter__
    do = self.iter(retry_state=retry_state)
  File "/env/local/lib/python3.7/site-packages/tenacity/__init__.py", line 360, in iter
    raise retry_exc.reraise()
  File "/env/local/lib/python3.7/site-packages/tenacity/__init__.py", line 193, in reraise
    raise self.last_attempt.result()
  File "/opt/python3.7/lib/python3.7/concurrent/futures/_base.py", line 428, in result
    return self.__get_result()
  File "/opt/python3.7/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/user_code/data_ingestion/app.py", line 189, in write_datapoint_to_influx_db
    timeout=10,
  File "/env/local/lib/python3.7/site-packages/requests/sessions.py", line 578, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/env/local/lib/python3.7/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/env/local/lib/python3.7/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/env/local/lib/python3.7/site-packages/requests/adapters.py", line 529, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='marty-300d7f91.influxcloud.net', port=8086): Read timed out. (read timeout=10)
2

Hello @mazzi,
Welcome!
Has it worked before? Are you suffering timeouts sporadically? If you’ve never been able to write points, can you share your python script?
Thanks

3

Sorry for the late reply @Anaisdg
We increased the timeout on our python influx db client setting the timeout in the constructor (before we were using timeouts in tenacity) and looks that is somehow… fixed?
We increased from 10 to 15 secs and now 20…
Thanks and sorry again for the late reply.

4

Hi @Anaisdg,

Following with this topic, having timeout=20 still gives us timeout.
Any suggestion here? Can we change the host? There’s another thing that can be done?
Frequency of timeouts (histogram last 2 days)

Screenshot 2022-02-09 at 09.12.38
Screenshot 2022-02-09 at 09.12.382712×308 73.7 KB

and following, python code:

class InfluxClient:
    """A tiny wrapper around InfluxDBClient"""

    __slots__ = ("_initialized", "_client", "_write_client", "token")

    def __init__(self, token: str = None) -> None:
        self._initialized = False
        self.token = token
        self._client: InfluxDBClient
        self._write_client: WriteApi

    @property
    def client(self) -> InfluxDBClient:
        self._initialize()
        return self._client

    @property
    def write_client(self) -> InfluxDBClient:
        self._initialize()
        return self._write_client

    def _initialize(self) -> None:
        if self._initialized is False:
            token = self.token if self.token else settings.INFLUXDB_TOKEN
            configuration = {
                "url": settings.INFLUXDB_URL,
                "token": token,
                "org": settings.INFLUXDB_ORG
                # TODO: available in the next release
                # "verify_ssl": settings.INFLUXDB_VERIFY_SSL,
            }
            self._client = InfluxDBClient(
                timeout=settings.INFLUX_CLIENT_TIMEOUT, **configuration
            )
            self._write_client = self._client.write_api(write_options=SYNCHRONOUS)
            self._initialized = True

    # Delay increased to 20 because of timeout errors
    @retry(stop=(stop_after_delay(20) | stop_after_attempt(5)), reraise=True)
    def write_points(self, points, bucket: str = None) -> None:
        bucket = bucket if bucket else settings.INFLUXDB_BUCKET_NAME
        self.write_client.write(bucket, None, points, write_precision=WritePrecision.MS)

Thanks,

5

The host with timeout is eu-central-1-1.aws.cloud2.influxdata.com