RFC: Pre-release feedback on InfluxDB 2.0.4 DockerHub build

dan-moran · January 28, 2021, 9:54pm

Hello community! As part of the upcoming 2.0.4 release of InfluxDB, we’re going to resume pushing releases to DockerHub with updated documentation. If anyone has time / interest, I’m looking for pre-release feedback on a preview of the new image build & docs. I’ve pushed an RC0 build to quay.io/influxdb/influxdb-dockerhub-rcs:v2.0.4-rc0, and pasted the Markdown for the new docs below. Please let me know if you get time to take a look; even if you don’t have time to run the image, I’d appreciate feedback on the details in the docs.

New docs:

InfluxDB

InfluxDB is a time series database built from the ground up to handle high write and query loads. InfluxDB is meant to be used as a backing store for any use case involving large amounts of timestamped data, including DevOps monitoring, application metrics, IoT sensor data, and real-time analytics.

InfluxDB Documentation

`latest` updated to 2.0

The latest tag for this image now points to the latest released implementation of InfluxDB 2.0. If you are using the latest tag and
would like to stay on the 1.x line, please update your environment to reference the 1.8 tag.

Using this Image - 2.x

Running the container

The InfluxDB image exposes a shared volume under /var/lib/influxdb2. You can mount a host directory to that point to access persisted container data. A typical invocation of the container might be:

$ docker run -p 8086:8086 \
      -v $PWD:/var/lib/influxdb2 \
      quay.io/influxdb/influxdb-dockerhub-rcs:v2.0.4-rc0

Modify $PWD to the directory where you want to store data associated with the InfluxDB container.

You can also have Docker control the volume mountpoint by using a named volume.

$ docker run -p 8086:8086 \
      -v influxdb2:/var/lib/influxdb2 \
      quay.io/influxdb/influxdb-dockerhub-rcs:v2.0.4-rc0

Exposed Ports

The following ports are important and are used by InfluxDB.

8086 HTTP UI and API port

The HTTP port will be automatically exposed when using docker run -P.

Find more about API Endpoints & Ports here.

Configuration

InfluxDB can be configured using a mix of a config file, environment variables, and CLI options. To mount a configuration file and use it with the server, you can use this command to generate the default configuration file:

$ docker run --rm quay.io/influxdb/influxdb-dockerhub-rcs:v2.0.4-rc0 \
      influxd print-config > config.yml

Modify the default configuration, which will now be available under $PWD. Then start the InfluxDB container:

$ docker run -p 8086:8086 \
      -v $PWD/config.yml:/etc/influxdb2/config.yml:ro \
      quay.io/influxdb/influxdb-dockerhub-rcs:v2.0.4-rc0

Modify $PWD to be the directory where you want to store the configuration file.

Individual config settings can be overridden by environment variables. The variables must be named using the format INFLUXD_${SNAKE_CASE_NAME}. The SNAKE_CASE_NAME for an option will be the option’s name with all dashes (-) replaced by underscores (_), in all caps.

Examples:

# Config setting: bolt-path
INFLUXD_BOLT_PATH=/root/influxdb.bolt
# Config setting: no-tasks
INFLUXD_NO_TASKS=true
# Config setting: storage-wal-fsync-delay
INFLUXD_STORAGE_WAL_FSYNC_DELAY=15m

Finally, all config options can be passed as CLI options:

$ docker run -p 8086:8086 \
      quay.io/influxdb/influxdb-dockerhub-rcs:v2.0.4-rc0 --storage-wal-fsync-delay=15m

CLI options take precedence over environment variables.

Find more about configuring InfluxDB here.

Database Setup

InfluxDB 2.0 requires authentication. A special API exists to bootstrap the first super-user in the database, along with an initial organization and bucket. It’s possible to access this API manually, or to run it automatically via environment variables.

Manual Setup

If your InfluxDB container is running locally (or on a host exposed to the network), you can perform initial setup from outside the container using either the UI or the influx CLI. Find more about setting up InfluxDB using these methods here.

It’s also possible to perform manual setup from within the container using docker exec. For example, if you start the container:

$ docker run -d -p 8086:8086 \
      --name influxdb2 \
      -v $PWD:/var/lib/influxdb2 \
      quay.io/influxdb/influxdb-dockerhub-rcs:v2.0.4-rc0

You can then run the influx client in the container:

$ docker exec influxdb2 influx setup \
      --username $USERNAME \
      --password $PASSWORD \
      --org $ORGANIZATION \
      --bucket $BUCKET

Running setup from within the container will cause CLI configs to be written to /etc/influxdb2/influx-configs. You can then use the influx CLI from within the container to extract the generated admin token:

# Using table output + cut
$ docker exec influxdb2 influx auth list \
      --user $USERNAME \
      --hide-headers | cut -f 3

# Using JSON output + jq
$ docker exec influxdb2 influx auth list \
      --user $USERNAME \
      --json | jq -r '.[].token'

Alternatively, you could configure your initial InfluxDB run to mount /etc/influxdb2 as a volume:

$ docker run -d -p 8086:8086 \
      --name influxdb2 \
      -v $PWD/data:/var/lib/influxdb2 \
      -v $PWD/config:/etc/influxdb2 \
      quay.io/influxdb/influxdb-dockerhub-rcs:v2.0.4-rc0

This will make the generated CLI configs available to the host.

Automated Setup

The InfluxDB image contains some extra functionality to automatically bootstrap the system. This functionality is enabled by setting the INFLUXD_INIT_MODE environment variable to the value setup when running the container. Additional environment variables are used to configure the setup logic:

INFLUXD_INIT_USERNAME: The username to set for the system’s initial super-user (Required).
INFLUXD_INIT_PASSWORD: The password to set for the system’s inital super-user (Required).
INFLUXD_INIT_ORG: The name to set for the system’s initial organization (Required).
INFLUXD_INIT_BUCKET: The name to set for the system’s initial bucket (Required).
INFLUXD_INIT_RETENTION: The duration the system’s initial bucket should retain data. If not set, the initial bucket will retain data forever.
INFLUXD_INIT_ADMIN_TOKEN: The authentication token to associate with the system’s initial super-user. If not set, a token will be auto-generated by the system.

Automated setup will generate metadata files and CLI configurations. It’s recommended to mount volumes at both paths to avoid losing data.

For example, a minimal invocation of automated setup is:

$ docker run -d -p 8086:8086 \
      -v $PWD/data:/var/lib/influxdb2 \
      -v $PWD/config:/etc/influxdb2 \
      -e INFLUXD_INIT_MODE=setup \
      -e INFLUXD_INIT_USERNAME=my-user \
      -e INFLUXD_INIT_PASSWORD=my-password \
      -e INFLUXD_INIT_ORG=my-org \
      -e INFLUXD_INIT_BUCKET=my-bucket \
      quay.io/influxdb/influxdb-dockerhub-rcs:v2.0.4-rc0

And an example using all available options is:

$ docker run -d -p 8086:8086 \
      -v $PWD/data:/var/lib/influxdb2 \
      -v $PWD/config:/etc/influxdb2 \
      -e INFLUXD_INIT_MODE=setup \
      -e INFLUXD_INIT_USERNAME=my-user \
      -e INFLUXD_INIT_PASSWORD=my-password \
      -e INFLUXD_INIT_ORG=my-org \
      -e INFLUXD_INIT_BUCKET=my-bucket \
      -e INFLUXD_INIT_RETENTION=1w \
      -e INFLUXD_INIT_ADMIN_TOKEN=my-super-secret-auth-token \
      quay.io/influxdb/influxdb-dockerhub-rcs:v2.0.4-rc0

NOTE: Automated setup will not run if an existing boltdb file is found at the configured path. This behavior allows for the InfluxDB container to reboot post-setup without encountering “DB is already set up” errors.

Upgrading from 1.x

InfluxDB 2.0 provides a 1.x-compatible API, but expects a different storage layout on disk. To bridge this mismatch, the InfluxDB image contains extra functionality to migrate 1.x data and config into 2.x layouts automatically before booting the influxd server.

The automated upgrade process also bootstraps an initial admin user, organization, and bucket in the system, so it uses the same set of environment variables as the automated setup process described above:

INFLUXD_INIT_USERNAME: The username to set for the system’s initial super-user (Required).
INFLUXD_INIT_PASSWORD: The password to set for the system’s inital super-user (Required).
INFLUXD_INIT_ORG: The name to set for the system’s initial organization (Required).
INFLUXD_INIT_BUCKET: The name to set for the system’s initial bucket (Required).
INFLUXD_INIT_RETENTION: The duration the system’s initial bucket should retain data. If not set, the initial bucket will retain data forever.
INFLUXD_INIT_ADMIN_TOKEN: The authentication token to associate with the system’s initial super-user. If not set, a token will be auto-generated by the system.

It also requires extra volumes to be mounted into the 2.x container:

Data from the 1.x instance
Custom config from the 1.x instance (if any)

The upgrade process searches for mounted 1.x data / config in this priority order:

A config file referred to by the INFLUXDB_INIT_UPGRADE_V1_CONFIG environment variable
A data directory referred to by the INFLUXDB_INIT_UPGRADE_V1_DIR environment variable
A config file mounted at /etc/influxdb/influxdb.conf
A data directory mounted at /var/lib/influxdb

Finally, the INFLUXD_INIT_MODE environment variable must be set to upgrade.

Automated upgrade will generate both data and config files, by default under /var/lib/influxdb2 and /etc/influxdb2. It’s recommended to mount volumes at both paths to avoid losing data.

NOTE: Automated upgrade will not run if an existing boltdb file is found at the configured path. This behavior allows for the InfluxDB container to reboot post-upgrade without overwriting migrated data.

Find more about the InfluxDB upgrade process here. See below for examples of common upgrade scenarios.

Upgrade Example - Minimal

Assume you’ve been running a minimal 1.x deployment:

$ docker run -p 8086:8086 \
      -v influxdb:/var/lib/influxdb \
      quay.io/influxdb/influxdb-dockerhub-rcs:v2.0.4-rc0:1.8

To upgrade this deployment to 2.x, stop the running 1.x container, then run:

$ docker run -p 8086:8086 \
      -v influxdb:/var/lib/influxdb \
      -v influxdb2:/var/lib/influxdb2 \
      -e INFLUXD_INIT_MODE=upgrade \
      -e INFLUXD_INIT_USERNAME=my-user \
      -e INFLUXD_INIT_PASSWORD=my-password \
      -e INFLUXD_INIT_ORG=my-org \
      -e INFLUXD_INIT_BUCKET=my-bucket \
      quay.io/influxdb/influxdb-dockerhub-rcs:v2.0.4-rc0

Upgrade Example - Custom 1.x Config

Assume you’ve been running a 1.x deployment with customized config:

$ docker run -p 8086:8086 \
      -v influxdb:/var/lib/influxdb \
      -v $PWD/influxdb.conf:/etc/influxdb/influxdb.conf:ro \
      quay.io/influxdb/influxdb-dockerhub-rcs:v2.0.4-rc0:1.8

To upgrade this deployment to 2.x, stop the running container, then run:

$ docker run -p 8086:8086 \
      -v influxdb:/var/lib/influxdb \
      -v influxdb2:/var/lib/influxdb2 \
      -v $PWD/influxdb.conf:/etc/influxdb/influxdb.conf:ro \
      -e INFLUXD_INIT_MODE=upgrade \
      -e INFLUXD_INIT_USERNAME=my-user \
      -e INFLUXD_INIT_PASSWORD=my-password \
      -e INFLUXD_INIT_ORG=my-org \
      -e INFLUXD_INIT_BUCKET=my-bucket \
      quay.io/influxdb/influxdb-dockerhub-rcs:v2.0.4-rc0

Upgrade Example - Custom Paths

Assume you’ve been running a 1.x deployment with data and config mounted at custom paths:

$ docker run -p 8086:8086 \
      -v influxdb:/root/influxdb/data \
      -v $PWD/influxdb.conf:/root/influxdb/influxdb.conf:ro \
      quay.io/influxdb/influxdb-dockerhub-rcs:v2.0.4-rc0:1.8 -config /root/influxdb/influxdb.conf

To upgrade this deployment to 2.x, first decide if you’d like to keep using custom paths, or use the 2.x defaults. If you decide to use the defaults, you’d stop the running 1.x container, then run:

$ docker run -p 8086:8086 \
      -v influxdb:/root/influxdb/data \
      -v influxdb2:/var/lib/influxdb2 \
      -v $PWD/influxdb.conf:/root/influxdb/influxdb.conf:ro \
      -e INFLUXD_INIT_MODE=upgrade \
      -e INFLUXD_INIT_USERNAME=my-user \
      -e INFLUXD_INIT_PASSWORD=my-password \
      -e INFLUXD_INIT_ORG=my-org \
      -e INFLUXD_INIT_BUCKET=my-bucket \
      -e INFLUXDB_INIT_UPGRADE_V1_CONFIG=/root/influxdb/influxdb.conf \
      quay.io/influxdb/influxdb-dockerhub-rcs:v2.0.4-rc0

To retain your custom paths, you’d run:

$ docker run -p 8086:8086 \
      -v influxdb:/root/influxdb/data \
      -v influxdb2:/root/influxdb2/data \
      -v $PWD/influxdb.conf:/root/influxdb/influxdb.conf:ro \
      -e INFLUXD_INIT_MODE=upgrade \
      -e INFLUXD_INIT_USERNAME=my-user \
      -e INFLUXD_INIT_PASSWORD=my-password \
      -e INFLUXD_INIT_ORG=my-org \
      -e INFLUXD_INIT_BUCKET=my-bucket \
      -e INFLUXDB_INIT_UPGRADE_V1_CONFIG=/root/influxdb/influxdb.conf \
      -e INFLUXD_CONFIG_PATH=/root/influxdb2/config.toml \
      -e INFLUXD_BOLT_PATH=/root/influxdb2/influxdb.bolt \
      -e INFLUXD_ENGINE_PATH=/root/influxdb2/engine \
      quay.io/influxdb/influxdb-dockerhub-rcs:v2.0.4-rc0

Custom Initialization Scripts

The InfluxDB image supports running arbitrary initialization scripts after initial system setup, on both the setup and upgrade paths. Scripts must have extension .sh and be mounted inside of the /docker-entrypoint-initdb.d directory. When multiple scripts are present, they will be executed in lexical sort order by name.

As a convenience for script-writers, the image will export a number of variables into the environment before executing any scripts:

INFLUX_CONFIGS_PATH: Path to the CLI configs file written by setup/upgrade
INFLUX_HOST: URL to the influxd instance running setup logic
INFLUXDB_INIT_USER_ID: ID of the initial admin user created by setup/upgrade
INFLUXDB_INIT_ORG_ID: ID of the initial organization created by setup/upgrade
INFLUXDB_INIT_BUCKET_ID: ID of the initial bucket created by setup/upgrade

For example, if you wanted to grant write-access to a 1.x client on your initial bucket, you’d first create the file $PWD/scripts/setup-v1.sh with contents:

#!/bin/bash
set -e

influx v1 dbrp create \
  --bucket-id ${INFLUXDB_INIT_BUCKET_ID} \
  --db ${V1_DB_NAME} \
  --rp ${V1_RP_NAME} \
  --default \
  --org ${INFLUXDB_INIT_ORG}

influx v1 auth create \
  --username ${V1_AUTH_USERNAME} \
  --password ${V1_AUTH_PASSWORD} \
  --write-bucket ${INFLUXDB_INIT_BUCKET_ID} \
  --org ${INFLUXDB_INIT_ORG}

Then you’d run:

$ docker run -p 8086:8086 \
      -v $PWD/data:/var/lib/influxdb2 \
      -v $PWD/config:/etc/influxdb2 \
      -v $PWD/scripts:/docker-entrypoint-initdb.d \
      -e INFLUXD_INIT_MODE=setup \
      -e INFLUXD_INIT_USERNAME=my-user \
      -e INFLUXD_INIT_PASSWORD=my-password \
      -e INFLUXD_INIT_ORG=my-org \
      -e INFLUXD_INIT_BUCKET=my-bucket \
      -e V1_DB_NAME=v1-db \
      -e V1_RP_NAME=v1-rp \
      -e V1_AUTH_USERNAME=v1-user \
      -e V1_AUTH_PASSWORD=v1-password \
      quay.io/influxdb/influxdb-dockerhub-rcs:v2.0.4-rc0

NOTE: Custom scripts will not run if an existing boltdb file is found at the configured path (causing setup or upgrade to be skipped). This behavior allows for the InfluxDB container to reboot post-initialization without encountering errors from non-idempotent script commands.

Topic		Replies	Views
Release Announcement: InfluxDB OSS 2.1.0 InfluxDB 2	0	618	November 8, 2021
New to Influxdb 2 Welcome & Getting Started	3	789	May 17, 2021
InfluxDB 2 and Docker InfluxDB 2	0	459	September 24, 2019
Stress testing influxdb InfluxDB 2 influxdb , performance , docker	0	441	January 14, 2023
InfluxDB 2 for arm32v7 release date? InfluxDB 2 influxdb , date , docker	6	7042	November 8, 2023