# Write to InfluxDB cloud fails with ApiException: (503) Reason: Service Unavailable; upstream connect error or disconnect/reset before headers. reset reason: connection failure

**URL:** <https://community.influxdata.com/t/write-to-influxdb-cloud-fails-with-apiexception-503-reason-service-unavailable-upstream-connect-error-or-disconnect-reset-before-headers-reset-reason-connection-failure/21501>\
**Category:** InfluxDB 2\
**Tags:** client-libraries, performance, influxdb-cloud-2-0\
**Created:** [September 1, 2021, 3:36pm UTC](https://community.influxdata.com/t/write-to-influxdb-cloud-fails-with-apiexception-503-reason-service-unavailable-upstream-connect-error-or-disconnect-reset-before-headers-reset-reason-connection-failure/21501 "2021-09-01T15:36:41Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![dabeeeenster](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/dabeeeenster/32/4882_2.png) [@dabeeeenster](https://community.influxdata.com/u/dabeeeenster)\
**Post date:** [September 7, 2021, 9:51am UTC](https://community.influxdata.com/t/write-to-influxdb-cloud-fails-with-apiexception-503-reason-service-unavailable-upstream-connect-error-or-disconnect-reset-before-headers-reset-reason-connection-failure/21501/21 "2021-09-07T09:51:07Z")

</div>

Hi,

I’ve run into this issue. We are sending 100’s of millions of requests to the InfluxDB2 API and it has been running perfectly up until a couple of weeks ago.

![image](https://us1.discourse-cdn.com/flex023/uploads/influxdata/original/2X/d/d0527682f3a8325c5655e97da3f91e6424045c3e.png)

This is from our Sentry. You can see quite clearly that the API was running perfectly and then started returning errors on the 17th of August.

I don’t really think that making a retry is a reasonable solution here. Returning a 500 is an error, and should be corrected. It was also clearly running absolutely perfectly up until the middle of August. Clearly something changed within the InfluxDB infra at this point!

I don’t think just catching the 500 and retrying is acceptable for a paid service! The Influx API was returning so many errors that we blew out our Sentry quota in August!

---

<div class="post-metadata">

**Author:** ![asmith](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/asmith/32/8251_2.png) [@asmith](https://community.influxdata.com/u/asmith)\
**Post date:** [September 7, 2021, 1:28pm UTC](https://community.influxdata.com/t/write-to-influxdb-cloud-fails-with-apiexception-503-reason-service-unavailable-upstream-connect-error-or-disconnect-reset-before-headers-reset-reason-connection-failure/21501/22 "2021-09-07T13:28:10Z")

</div>

That’s really really interesting.  
I just checked our AWS logs and we see the same pattern.  
I hadn’t realised this!  
Here is a chart from the last 2 months that counts every instance of the error (503)  
The errors start on 17th August

 ![image](https://us1.discourse-cdn.com/flex023/uploads/influxdata/original/2X/1/1d1af9a428859c15586951548d8f16ee8f29e814.png)

In addition I notice that it happened for a very short period on 28th July and 6th August.  
But nothing outside of this

---

<div class="post-metadata">

**Author:** ![asmith](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/asmith/32/8251_2.png) [@asmith](https://community.influxdata.com/u/asmith)\
**Post date:** [September 7, 2021, 1:44pm UTC](https://community.influxdata.com/t/write-to-influxdb-cloud-fails-with-apiexception-503-reason-service-unavailable-upstream-connect-error-or-disconnect-reset-before-headers-reset-reason-connection-failure/21501/23 "2021-09-07T13:44:59Z")

</div>

Hi @Anaisdg ,  
I think that maybe I need to do as you said and submit an issue in the client library repo.  
Do you mean here [GitHub - influxdata/influxdb-client-python: InfluxDB 2.0 python client](https://github.com/influxdata/influxdb-client-python) ?

---

<div class="post-metadata">

**Author:** ![dabeeeenster](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/dabeeeenster/32/4882_2.png) [@dabeeeenster](https://community.influxdata.com/u/dabeeeenster)\
**Post date:** [September 7, 2021, 1:46pm UTC](https://community.influxdata.com/t/write-to-influxdb-cloud-fails-with-apiexception-503-reason-service-unavailable-upstream-connect-error-or-disconnect-reset-before-headers-reset-reason-connection-failure/21501/24 "2021-09-07T13:46:09Z")

</div>

Is this specific to the Python SDK?!

---

<div class="post-metadata">

**Author:** ![asmith](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/asmith/32/8251_2.png) [@asmith](https://community.influxdata.com/u/asmith)\
**Post date:** [September 7, 2021, 2:00pm UTC](https://community.influxdata.com/t/write-to-influxdb-cloud-fails-with-apiexception-503-reason-service-unavailable-upstream-connect-error-or-disconnect-reset-before-headers-reset-reason-connection-failure/21501/25 "2021-09-07T14:00:05Z")

</div>

No. I’m getting the same fails with a backup client I wrote to use http

---

<div class="post-metadata">

**Author:** ![asmith](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/asmith/32/8251_2.png) [@asmith](https://community.influxdata.com/u/asmith)\
**Post date:** [September 7, 2021, 2:01pm UTC](https://community.influxdata.com/t/write-to-influxdb-cloud-fails-with-apiexception-503-reason-service-unavailable-upstream-connect-error-or-disconnect-reset-before-headers-reset-reason-connection-failure/21501/26 "2021-09-07T14:01:25Z")

</div>

I’ve now raised an issue on Github.  
You might like to visit and confirm it’s affecting you to get it sorted.  
Thanks @dabeeeenster. I thought I was going mad!

> <https://github.com/influxdata/influxdb-client-python/issues/326>
>
> \_\_Steps to reproduce:\_\_
> Using the Python client make repeated calls to the writ…e API.
> Do this simultaneously from 2 or 3 different clients (not sure if this is relevant but this is the fail condition)
> Write rates are a call to the API every second or so
> I'm using the SYNCHRONOUS calls
> 
> \`
> influx\_client = InfluxDBClient(url=os.environ\['influx\_url'\], token=os.environ\['token'\],retries=retries)
> write\_api = influx\_client.write\_api(write\_options=SYNCHRONOUS)
> \`
> (repeatedly)
> \` influx\_returns = write\_api.write(InfluxBucket, my\_org, Influx\_lines,'ms') \`
> 
> 
> 
> 
> 
> \_\_Expected behavior:\_\_
> Until 17th Aug we were seeing the expected behaviour.
> All the writes succeeded.  
> Data was written reliably
> 
> \_\_Actual behavior:\_\_
> From 17th August we started getting 
> 
> \> ApiException: (503) 
> \>Reason: Service Unavailable; upstream connect error or disconnect/reset before headers. 
> \>reset reason: connection failure
> 
> The errors definitely started on 17th Aug and I know we didn't change anything because everyone was on holiday!
> I have AWS logs which show when this started
> !\[image\](https://user-images.githubusercontent.com/69196723/132357356-01a72c1b-97a4-4154-9cef-5968fe878b4e.png)
> 
> You can see the discussion \[here \](https://community.influxdata.com/t/write-to-influxdb-cloud-fails-with-apiexception-503-reason-service-unavailable-upstream-connect-error-or-disconnect-reset-before-headers-reset-reason-connection-failure/21501/5)that it's affecting multiple users now
> 
> \_\_Specifications:\_\_
> - Python 3.8:
> - InfluxDB Version: Cloud v2
> - Platform: AWS Lambda written in Python
> -

---

<div class="post-metadata">

**Author:** ![Anaisdg](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/anaisdg/32/6401_2.png) [@Anaisdg](https://community.influxdata.com/u/Anaisdg)\
**Post date:** [September 7, 2021, 3:06pm UTC](https://community.influxdata.com/t/write-to-influxdb-cloud-fails-with-apiexception-503-reason-service-unavailable-upstream-connect-error-or-disconnect-reset-before-headers-reset-reason-connection-failure/21501/27 "2021-09-07T15:06:15Z")

</div>

Hello @asmith and @dabeeeenster,  
I’ve created an issue for the storage team to take a look at your question and concerns. I’ll also bug someone if they haven’t responded in a couple days. I appreciate your detailed questions and thanks for your patience.

> <https://github.com/influxdata/influxdb/issues/22404>
>
> \- \[\] https://community.influxdata.com/t/write-to-influxdb-cloud-fails-with-apie…xception-503-reason-service-unavailable-upstream-connect-error-or-disconnect-reset-before-headers-reset-reason-connection-failure/21501/21
> 
> Thank you!

---

<div class="post-metadata">

**Author:** ![asmith](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/asmith/32/8251_2.png) [@asmith](https://community.influxdata.com/u/asmith)\
**Post date:** [September 7, 2021, 3:24pm UTC](https://community.influxdata.com/t/write-to-influxdb-cloud-fails-with-apiexception-503-reason-service-unavailable-upstream-connect-error-or-disconnect-reset-before-headers-reset-reason-connection-failure/21501/28 "2021-09-07T15:24:47Z")

</div>

I appreciate all your help over the years @Anaisdg .

By the way if you can help with some stub code to use urllib3 to write line data to InfluxDB 2.0 cloud then I could test whether the failures also happen with the urllib3 library which I think matches what Python API client uses.

I managed to get the requests library working but the following doesn’t work:

```auto
import json
import urllib3

headers={'Content-Type': 'application/vnd.flux','Authorization': 'Token xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx=='}
url = 'https://eu-central-1-1.aws.cloud2.influxdata.com/api/v2/write?orgID=xxxxxxxxxxxxxx&bucket=TEST_bucket&precision=ms'
payload = 'cpu_load_short,host=server01,region=us-west value1=99.64\ncpu_load_short,host=server01,region=us-west value2=5.64\n'

#import requests
#x = requests.post(url, data=payload, headers=headers) # this succeeds with [204]

# THIS FAILS
http = urllib3.PoolManager()
r = http.request(
    'POST',
    url,
    body=payload,
    headers=headers
    )
returnval = r.data.decode('utf-8')
print(f"Request returned:{returnval}\n")

```

---

<div class="post-metadata">

**Author:** ![asmith](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/asmith/32/8251_2.png) [@asmith](https://community.influxdata.com/u/asmith)\
**Post date:** [September 7, 2021, 3:25pm UTC](https://community.influxdata.com/t/write-to-influxdb-cloud-fails-with-apiexception-503-reason-service-unavailable-upstream-connect-error-or-disconnect-reset-before-headers-reset-reason-connection-failure/21501/29 "2021-09-07T15:25:24Z")

</div>

(uncommenting the two requests lines works)

---

<div class="post-metadata">

**Author:** ![asmith](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/asmith/32/8251_2.png) [@asmith](https://community.influxdata.com/u/asmith)\
**Post date:** [September 7, 2021, 3:27pm UTC](https://community.influxdata.com/t/write-to-influxdb-cloud-fails-with-apiexception-503-reason-service-unavailable-upstream-connect-error-or-disconnect-reset-before-headers-reset-reason-connection-failure/21501/30 "2021-09-07T15:27:19Z")

</div>

The error returned is:

> HTTPSConnectionPool(host=‘[eu-central-1-1.aws.cloud2.influxdata.com](http://eu-central-1-1.aws.cloud2.influxdata.com)’, port=443): Max retries exceeded with url: /api/v2/write?orgID=xxxxxxxxxxxx&bucket=TEST\_bucket&precision=ms (Caused by SSLError(SSLError(“bad handshake: Error([(‘SSL routines’, ‘tls\_process\_server\_certificate’, ‘certificate verify failed’)])”)))

Clearly, using the urllib3 library is not as simple as using the requests library.

---

<div class="post-metadata">

**Author:** ![Anaisdg](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/anaisdg/32/6401_2.png) [@Anaisdg](https://community.influxdata.com/u/Anaisdg)\
**Post date:** [September 7, 2021, 3:28pm UTC](https://community.influxdata.com/t/write-to-influxdb-cloud-fails-with-apiexception-503-reason-service-unavailable-upstream-connect-error-or-disconnect-reset-before-headers-reset-reason-connection-failure/21501/31 "2021-09-07T15:28:06Z")

</div>

Hello @asmith,  
Sure I’ll give it a try and get back to you end of day.

---

<div class="post-metadata">

**Author:** ![asmith](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/asmith/32/8251_2.png) [@asmith](https://community.influxdata.com/u/asmith)\
**Post date:** [September 7, 2021, 3:29pm UTC](https://community.influxdata.com/t/write-to-influxdb-cloud-fails-with-apiexception-503-reason-service-unavailable-upstream-connect-error-or-disconnect-reset-before-headers-reset-reason-connection-failure/21501/32 "2021-09-07T15:29:09Z")

</div>

hey thanks @Anaisdg  
Cheers.

---

<div class="post-metadata">

**Author:** ![asmith](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/asmith/32/8251_2.png) [@asmith](https://community.influxdata.com/u/asmith)\
**Post date:** [September 8, 2021, 9:35am UTC](https://community.influxdata.com/t/write-to-influxdb-cloud-fails-with-apiexception-503-reason-service-unavailable-upstream-connect-error-or-disconnect-reset-before-headers-reset-reason-connection-failure/21501/33 "2021-09-08T09:35:34Z")

</div>

Thanks to @bednar  
Here is code to test if the problem persists when using the urllib3 library

```auto
import urllib3
import certifi

https_url = 'https://eu-central-1-1.aws.cloud2.influxdata.com'
# DON'T FORGET THE HTTPS
org = 'my_org'
token = 'my_token'
bucket = 'my_bucket'

headers = {'Content-Type': 'application/vnd.flux', 'Authorization': ('Token %s' % token)}
url = '%s/api/v2/write?org=%s&bucket=%s&precision=ms' % (https_url, org, bucket)
payload = 'cpu_load_short,host=server01,region=us-west value1=99.64\ncpu_load_short,host=server01,region=us-west value2=5.64\n'

https = urllib3.PoolManager(ca_certs=certifi.where())
r = https.request(
    'POST',
    url,
    body=payload,
    headers=headers
)

print(f"Response status: '{r.status}', success: {r.status == 204}\n")

```

---

<div class="post-metadata">

**Author:** ![asmith](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/asmith/32/8251_2.png) [@asmith](https://community.influxdata.com/u/asmith)\
**Post date:** [September 17, 2021, 5:41pm UTC](https://community.influxdata.com/t/write-to-influxdb-cloud-fails-with-apiexception-503-reason-service-unavailable-upstream-connect-error-or-disconnect-reset-before-headers-reset-reason-connection-failure/21501/34 "2021-09-17T17:41:56Z")

</div>

@Anaisdg ,  
There have been some pretty worrying developments to this problem.  
The errors have been absolutely flooding our systems today (we got 30,000 failures of the (503) type.

> <https://github.com/influxdata/influxdb-client-python/issues/326>
>
> \_\_Steps to reproduce:\_\_
> Using the Python client make repeated calls to the writ…e API.
> Do this simultaneously from 2 or 3 different clients (not sure if this is relevant but this is the fail condition)
> Write rates are a call to the API every second or so
> I'm using the SYNCHRONOUS calls
> 
> \`
> influx\_client = InfluxDBClient(url=os.environ\['influx\_url'\], token=os.environ\['token'\],retries=retries)
> write\_api = influx\_client.write\_api(write\_options=SYNCHRONOUS)
> \`
> (repeatedly)
> \` influx\_returns = write\_api.write(InfluxBucket, my\_org, Influx\_lines,'ms') \`
> 
> 
> 
> 
> 
> \_\_Expected behavior:\_\_
> Until 17th Aug we were seeing the expected behaviour.
> All the writes succeeded.  
> Data was written reliably
> 
> \_\_Actual behavior:\_\_
> From 17th August we started getting 
> 
> \> ApiException: (503) 
> \>Reason: Service Unavailable; upstream connect error or disconnect/reset before headers. 
> \>reset reason: connection failure
> 
> The errors definitely started on 17th Aug and I know we didn't change anything because everyone was on holiday!
> I have AWS logs which show when this started
> !\[image\](https://user-images.githubusercontent.com/69196723/132357356-01a72c1b-97a4-4154-9cef-5968fe878b4e.png)
> 
> You can see the discussion \[here \](https://community.influxdata.com/t/write-to-influxdb-cloud-fails-with-apiexception-503-reason-service-unavailable-upstream-connect-error-or-disconnect-reset-before-headers-reset-reason-connection-failure/21501/5)that it's affecting multiple users now
> 
> \_\_Specifications:\_\_
> - Python 3.8:
> - InfluxDB Version: Cloud v2
> - Platform: AWS Lambda written in Python
> -

Can you give me any updates on how the storage team are getting on with finding and solving the issue?

Thanks for your help.

---

<div class="post-metadata">

**Author:** ![Anaisdg](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/anaisdg/32/6401_2.png) [@Anaisdg](https://community.influxdata.com/u/Anaisdg)\
**Post date:** [September 17, 2021, 6:25pm UTC](https://community.influxdata.com/t/write-to-influxdb-cloud-fails-with-apiexception-503-reason-service-unavailable-upstream-connect-error-or-disconnect-reset-before-headers-reset-reason-connection-failure/21501/35 "2021-09-17T18:25:40Z")

</div>

Hello @asmith,  
There was an outage. Are you still having trouble?  
In the future try checking:

> **[InfluxDB Cloud Status](https://status.influxdata.com/)**
>
> Welcome to InfluxDB Cloud's home for real-time and historical data on system performance.

As far as 503’s go, that the issue is known and Engineering is working hard to fix that as a high priority. They have already applied a mitigation.

---

<div class="post-metadata">

**Author:** ![asmith](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/asmith/32/8251_2.png) [@asmith](https://community.influxdata.com/u/asmith)\
**Post date:** [September 23, 2021, 7:54pm UTC](https://community.influxdata.com/t/write-to-influxdb-cloud-fails-with-apiexception-503-reason-service-unavailable-upstream-connect-error-or-disconnect-reset-before-headers-reset-reason-connection-failure/21501/36 "2021-09-23T19:54:48Z")

</div>

Hi @Anaisdg , thanks for pointing out the status link. Is there an API way of checking status so that I create log events in our architecture that help with diagnosing performance issues from a dashboard?

Also, I can report that the (503) errors have stopped now.  
May I ask what the storage team found and fixed?  
I’m wanting to learn more about the types of failure of the InfluxDB cloud platform that we may have to mitigate in the future. Until we observed the 503 errors, In my naivety, I had no concept that InfluxDB Cloud could be temporarily unavailable. I now need to go away and design some AWS components that can store the data temporarily when InfluxDB has issues. Of course nothing has 100% uptime eh?

Again, please can you clarify what the storage team found and fixed?

And pass on my congrats and thanks for fixing it!! 😄

---

<div class="post-metadata">

**Author:** ![Anaisdg](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/anaisdg/32/6401_2.png) [@Anaisdg](https://community.influxdata.com/u/Anaisdg)\
**Post date:** [September 28, 2021, 6:36pm UTC](https://community.influxdata.com/t/write-to-influxdb-cloud-fails-with-apiexception-503-reason-service-unavailable-upstream-connect-error-or-disconnect-reset-before-headers-reset-reason-connection-failure/21501/37 "2021-09-28T18:36:55Z")

</div>

Hello @asmith,  
I don’t think there’s a public API for checking the status, but I’ll ask.

I don’t remember what was causing the errors anymore tbh.

---

<div class="post-metadata">

**Author:** ![asmith](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/asmith/32/8251_2.png) [@asmith](https://community.influxdata.com/u/asmith)\
**Post date:** [September 28, 2021, 9:36pm UTC](https://community.influxdata.com/t/write-to-influxdb-cloud-fails-with-apiexception-503-reason-service-unavailable-upstream-connect-error-or-disconnect-reset-before-headers-reset-reason-connection-failure/21501/38 "2021-09-28T21:36:34Z")

</div>

Hi @Anaisdg , I think we got our wires crossed. I was asking about the 503 errors that have been existing for about a month. I was asking whether it was fixed and what caused it.

I think your reply was referring to the [outage on 17th Aug](https://status.influxdata.com/incidents/l7943t0xlm01) which is maybe a different issue and not the cause of the 503 errors in this topic.

So, maybe I should check again … Have the 503 errors been fixed? Because today I was getting Timeout failures on InfluxDB writes which is another symptom of Influx cloud not being available… as are the same other customers. See the [github link](https://github.com/influxdata/influxdb-client-python/issues/326)

---

<div class="post-metadata">

**Author:** ![Anaisdg](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/anaisdg/32/6401_2.png) [@Anaisdg](https://community.influxdata.com/u/Anaisdg)\
**Post date:** [September 29, 2021, 6:25pm UTC](https://community.influxdata.com/t/write-to-influxdb-cloud-fails-with-apiexception-503-reason-service-unavailable-upstream-connect-error-or-disconnect-reset-before-headers-reset-reason-connection-failure/21501/39 "2021-09-29T18:25:43Z")

</div>

Hello @asmith,  
Thanks for helping me understand I didn’t read carefully enough.  
I believe engineering has upgraded many of their 3rd party services which was the initial cause but are still investigating.

---

<div class="post-metadata">

**Author:** ![Anaisdg](https://sea1.discourse-cdn.com/flex023/user_avatar/community.influxdata.com/anaisdg/32/6401_2.png) [@Anaisdg](https://community.influxdata.com/u/Anaisdg)\
**Post date:** [September 30, 2021, 3:35pm UTC](https://community.influxdata.com/t/write-to-influxdb-cloud-fails-with-apiexception-503-reason-service-unavailable-upstream-connect-error-or-disconnect-reset-before-headers-reset-reason-connection-failure/21501/40 "2021-09-30T15:35:06Z")

</div>

Hello @asmith,  
I was wrong please take a look at:  
The subscribe button on the status page that gives various options including webhooks and RSS feeds.  
[https://status.influxdata.com/](https://status.influxdata.com/)  
There’s a limited api with details here: [InfluxDB Cloud Status - API](https://status.influxdata.com/api)

[Previous page](https://community.influxdata.com/t/write-to-influxdb-cloud-fails-with-apiexception-503-reason-service-unavailable-upstream-connect-error-or-disconnect-reset-before-headers-reset-reason-connection-failure/21501.md?page=1)

[Next page](https://community.influxdata.com/t/write-to-influxdb-cloud-fails-with-apiexception-503-reason-service-unavailable-upstream-connect-error-or-disconnect-reset-before-headers-reset-reason-connection-failure/21501.md?page=3)