Skip to content

Commit aab84b5

Browse files
richturnerrichturner
committed
Added auto upgrade for major version changes
1 parent 9a18b62 commit aab84b5

3 files changed

Lines changed: 210 additions & 34 deletions

File tree

‎Dockerfile‎

Lines changed: 21 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -1,36 +1,13 @@
1-
# -----------------------------------------------------------------------------------------------
2-
# POSTGIS and TimescaleDB (inc. toolkit for hyperfunctions) image built for aarch64 support
3-
# using timescaledev/timescaledb-ha base image with:
4-
#
5-
# - OR specific ENV variables and a healthcheck added
6-
# - PGDATA path set to match old Alpine image (for ease of DB migration)
7-
# - POSTGRES user UID and GID changed to match old Alpine image (for ease of DB migration)
8-
# - OR_DISABLE_REINDEX env variable with associated scripts to determine if a REINDEX of the entire DB should be carried
9-
# out at first startup with existing DB (checks whether or not $PGDATA/OR_REINDEX_COUNTER.$OR_REINDEX_COUNTER exists).
10-
# This is used when a collation change has occurred (glibc version change, muslc <-> glibc) which can break the indexes;
11-
# migration can either be manually handled or auto handled depending on OR_DISABLE_REINDEX env variable value.
12-
# NOTE THAT A REINDEX CAN TAKE A LONG TIME DEPENDING ON THE SIZE OF THE DB! And startup will be delayed until completed
13-
# This functionality is intended to simplify migration for basic users; advanced users with large DBs should take care of this
14-
# themselves.
15-
#
16-
#
17-
#
18-
#
19-
# timescale/timescaledb-ha image is ubuntu based and only currently supports amd64; they are
20-
# working on ARM64 support in timescaledev/timescaledb-ha see:
21-
#
22-
# https://github.com/timescale/timescaledb-docker-ha/pull/355
23-
#
24-
# See this issue for POSTGIS base image aarch64 support discussion:
25-
#
26-
# https://github.com/postgis/docker-postgis/issues/216
27-
# ------- ----------------------------------------------------------------------------------------
28-
29-
# TODO: Switch over to timescale/timescaledb-ha once arm64 supported
30-
# We get POSTGIS and timescale+toolkit from this image
1+
2+
ARG PG_MAJOR_PREVIOUS=14
3+
ARG PG_MAJOR=15
4+
315
FROM timescaledev/timescaledb-ha:pg15-multi as trimmed
326
MAINTAINER support@openremote.io
337

8+
ARG PG_MAJOR_PREVIOUS
9+
ARG PG_MAJOR
10+
3411
USER root
3512

3613
# Give postgres user the same UID and GID as the old alpine postgres image to simplify migration of existing DB
@@ -54,12 +31,22 @@ RUN chmod +x /docker-entrypoint-initdb.d/*
5431
# Below is mostly copied from https://github.com/timescale/timescaledb-docker-ha/blob/master/Dockerfile (with OR specific entrypoint,
5532
# workdir and OR env defaults)
5633

34+
# Get multi all image
35+
FROM timescaledev/timescaledb-ha:pg15-multi-all as trimmed-all
36+
37+
ARG PG_MAJOR_PREVIOUS
38+
ARG PG_MAJOR
5739

5840
## Create a smaller Docker image from the builder image
5941
FROM scratch
6042
COPY --from=trimmed / /
6143

62-
ARG PG_MAJOR=15
44+
ARG PG_MAJOR_PREVIOUS
45+
ARG PG_MAJOR
46+
47+
## Copy previous PG MAJOR version executable
48+
COPY --from=trimmed-all /usr/lib/postgresql/${PG_MAJOR_PREVIOUS} /usr/lib/postgresql/${PG_MAJOR_PREVIOUS}
49+
COPY --from=trimmed-all /usr/share/postgresql/${PG_MAJOR_PREVIOUS} /usr/share/postgresql/${PG_MAJOR_PREVIOUS}
6350

6451
# Increment this to indicate that a re-index should be carried out on first startup with existing data; REINDEX can still be overidden
6552
# with OR_DISABLE_REINDEX=true
@@ -86,9 +73,11 @@ ENV PGROOT=/var/lib/postgresql \
8673
POSTGRES_DB=${POSTGRES_DB:-openremote} \
8774
POSTGRES_USER=${POSTGRES_USER:-postgres} \
8875
POSTGRES_PASSWORD=${POSTGRES_PASSWORD:-postgres} \
76+
PG_MAJOR=$PG_MAJOR \
8977
OR_REINDEX_COUNTER=${OR_REINDEX_COUNTER} \
9078
OR_DISABLE_REINDEX=${OR_DISABLE_REINDEX:-false} \
91-
POSTGRES_MAX_CONNECTIONS=${POSTGRES_MAX_CONNECTIONS:-50}
79+
POSTGRES_MAX_CONNECTIONS=${POSTGRES_MAX_CONNECTIONS:-50} \
80+
OR_DISABLE_AUTO_UPGRADE=${OR_DISABLE_AUTO_UPGRADE:-false}
9281

9382
WORKDIR /var/lib/postgresql
9483
EXPOSE 5432 8008 8081

‎README.md‎

Lines changed: 25 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,28 @@
11
# Postgresql docker image
22
[![build multirach postgresql Docker image and push to it dockerhub](https://github.com/openremote/postgresql/actions/workflows/postgresql.yml/badge.svg)](https://github.com/openremote/postgresql/actions/workflows/postgresql.yml)
33

4-
PostgreSQL docker image based on [Timescale's](https://www.timescale.com/) [timescaledev/timescaledb-ha](https://hub.docker.com/r/timescaledev/timescaledb-ha) docker image which contains post GIS and timescaleDB extensions.
4+
POSTGIS and TimescaleDB (inc. toolkit for hyperfunctions) image built for aarch64 support using `timescaledev/timescaledb-ha` base image with:
5+
6+
- OR specific ENV variables and a healthcheck added
7+
- PGDATA path set to match old Alpine image (for ease of DB migration)
8+
- POSTGRES user UID and GID changed to match old Alpine image (for ease of DB migration)
9+
- Auto upgrade of database with PG major version changes from previous PG major version; can be disabled using
10+
OR_DISABLE_AUTO_UPGRADE=true.
11+
- OR_DISABLE_REINDEX env variable with associated scripts to determine if a REINDEX of the entire DB should be carried
12+
out at first startup with existing DB (checks whether or not $PGDATA/OR_REINDEX_COUNTER.$OR_REINDEX_COUNTER exists).
13+
This is used when a collation change has occurred (glibc version change, muslc <-> glibc) which can break the indexes;
14+
migration can either be manually handled or auto handled depending on OR_DISABLE_REINDEX env variable value.
15+
NOTE THAT A REINDEX CAN TAKE A LONG TIME DEPENDING ON THE SIZE OF THE DB! And startup will be delayed until completed
16+
This functionality is intended to simplify migration for basic users; advanced users with large DBs should take care of this
17+
themselves.
18+
19+
`timescale/timescaledb-ha` image is ubuntu based and only currently supports amd64; they are working on ARM64 support in timescaledev/timescaledb-ha see:
20+
21+
https://github.com/timescale/timescaledb-docker-ha/pull/355
22+
23+
See this issue for POSTGIS base image aarch64 support discussion:
24+
25+
https://github.com/postgis/docker-postgis/issues/216
26+
27+
TODO: Switch over to timescale/timescaledb-ha once arm64 supported
28+
We get POSTGIS and timescale+toolkit from this image

‎or-entrypoint.sh‎

Lines changed: 164 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,8 @@
11
#!/usr/bin/env bash
22

33
# THIS FILE IS FOR MIGRATION OF EXISTING DB TO TIMESCALEDB IMAGE AS TIMESCALE INIT SCRIPTS AREN'T RUN WHEN DB
4-
# ALREADY EXISTS; IT ALSO DOES AN AUTOMATIC REINDEX OF THE DB WHEN OR_REINDEX_COUNTER changes TO SIMPLIFY MIGRATIONS
4+
# ALREADY EXISTS; IT ALSO DOES AN AUTOMATIC REINDEX OF THE DB WHEN OR_REINDEX_COUNTER CHANGES TO SIMPLIFY MIGRATIONS
5+
# IT ALSO AUTOMATICALLY HANDLES UPGRADING OF DATABASE AND DURING MAJOR VERSION CHANGES
56

67
source /docker-entrypoint.sh
78
docker_setup_env
@@ -20,6 +21,168 @@ if [ -n "$DATABASE_ALREADY_EXISTS" ]; then
2021
echo "timescaledb.telemetry_level=off" >> "$PGDATA/postgresql.conf"
2122
fi
2223

24+
########################################################################################
25+
# Do upgrade checks - Adapted from https://github.com/pgautoupgrade/docker-pgautoupgrade
26+
########################################################################################
27+
28+
# Get the version of the PostgreSQL data files
29+
DB_VERSION=$PG_MAJOR
30+
if [ -s "${PGDATA}/PG_VERSION" ]; then
31+
DB_VERSION=$(cat "${PGDATA}/PG_VERSION")
32+
fi
33+
34+
# Try and upgrade if needed
35+
if [ "$DB_VERSION" != "$PG_MAJOR" ] && [ "$OR_DISABLE_AUTO_UPGRADE" != "true" ]; then
36+
37+
echo "----------------------------------------------------------------------"
38+
echo "Postgres major version is newer than the existing DB, performing auto upgrade..."
39+
echo "----------------------------------------------------------------------"
40+
41+
if [ ! -d "/usr/lib/postgresql/${DB_VERSION}" ]; then
42+
echo "Postgres executable version '$DB_VERSION' is not included in this image so cannot auto upgrade"
43+
exit 1
44+
fi
45+
46+
# Check for presence of old/new directories, indicating a failed previous autoupgrade
47+
echo "----------------------------------------------------------------------"
48+
echo "Checking for left over artifacts from a failed previous autoupgrade..."
49+
echo "----------------------------------------------------------------------"
50+
OLD="${PGDATA}/old"
51+
NEW="${PGDATA}/new"
52+
if [ -d "${OLD}" ]; then
53+
echo "*****************************************"
54+
echo "Left over OLD directory found. Aborting."
55+
echo "*****************************************"
56+
exit 10
57+
fi
58+
if [ -d "${NEW}" ]; then
59+
echo "*****************************************"
60+
echo "Left over NEW directory found. Aborting."
61+
echo "*****************************************"
62+
exit 11
63+
fi
64+
echo "-------------------------------------------------------------------------------"
65+
echo "No artifacts found from a failed previous autoupgrade. Continuing the process."
66+
echo "-------------------------------------------------------------------------------"
67+
68+
# Don't automatically abort on non-0 exit status, as that messes with these upcoming mv commands
69+
set +e
70+
71+
# Move the PostgreSQL data files into a subdirectory of the mount point
72+
echo "---------------------------------------"
73+
echo "Creating OLD temporary directory ${OLD}"
74+
echo "---------------------------------------"
75+
mkdir "${OLD}"
76+
if [ ! -d "${OLD}" ]; then
77+
echo "*********************************************************************"
78+
echo "Creation of temporary directory '${OLD}' failed. Aborting completely"
79+
echo "*********************************************************************"
80+
exit 7
81+
fi
82+
echo "--------------------------------------------"
83+
echo "Creating OLD temporary directory is complete"
84+
echo "--------------------------------------------"
85+
86+
echo "-------------------------------------------------------"
87+
echo "Moving existing data files into OLD temporary directory"
88+
echo "-------------------------------------------------------"
89+
mv -v "${PGDATA}"/* "${OLD}"
90+
echo "-------------------------------------------------------------------"
91+
echo "Moving existing data files into OLD temporary directory is complete"
92+
echo "-------------------------------------------------------------------"
93+
94+
echo "---------------------------------------"
95+
echo "Creating NEW temporary directory ${NEW}"
96+
echo "---------------------------------------"
97+
mkdir "${NEW}"
98+
if [ ! -d "${NEW}" ]; then
99+
echo "********************************************************************"
100+
echo "Creation of temporary directory '${NEW}' failed. Aborting completely"
101+
echo "********************************************************************"
102+
# With a failure at this point we should be able to move the old data back
103+
# to its original location
104+
mv -v "${OLD}"/* "${PGDATA}"
105+
exit 8
106+
fi
107+
echo "--------------------------------------------"
108+
echo "Creating NEW temporary directory is complete"
109+
echo "--------------------------------------------"
110+
111+
echo "-----------------------------------------------------"
112+
echo "Changing permissions of temporary directories to 0700"
113+
echo "-----------------------------------------------------"
114+
chmod 0700 "${OLD}" "${NEW}"
115+
echo "---------------------------------------------------------"
116+
echo "Changing permissions of temporary directories is complete"
117+
echo "---------------------------------------------------------"
118+
119+
# Return the error handling back to automatically aborting on non-0 exit status
120+
set -e
121+
122+
# Initialise the new PostgreSQL database directory
123+
echo "--------------------------------------------------------------------------------------------------------------------"
124+
echo "Initialising new database directory"
125+
echo "--------------------------------------------------------------------------------------------------------------------"
126+
initdb -D $PGDATA/new
127+
echo "------------------------------------"
128+
echo "New database directory initialisation complete"
129+
echo "------------------------------------"
130+
131+
# Change into the PostgreSQL database directory, to avoid a pg_upgrade error about write permissions
132+
cd "${PGDATA}"
133+
134+
# Run the pg_upgrade command itself
135+
echo "---------------------------------------"
136+
echo "Running pg_upgrade command, from $(pwd)"
137+
echo "---------------------------------------"
138+
pg_upgrade --link -b /usr/lib/postgresql/${DB_VERSION}/bin -B /usr/lib/postgresql/${PG_MAJOR}/bin -d $OLD -D $NEW
139+
echo "--------------------------------------"
140+
echo "Running pg_upgrade command is complete"
141+
echo "--------------------------------------"
142+
143+
# Move the new database files into place
144+
echo "-----------------------------------------------------"
145+
echo "Moving the upgraded database files to the active directory"
146+
echo "-----------------------------------------------------"
147+
mv -v "${NEW}"/* "${PGDATA}"
148+
echo "-----------------------------------------"
149+
echo "Moving the upgraded database files is complete"
150+
echo "-----------------------------------------"
151+
152+
# Re-use the pg_hba.conf and pg_ident.conf from the old data directory
153+
echo "--------------------------------------------------------------"
154+
echo "Copying the old pg_hba and pg_ident configuration files across"
155+
echo "--------------------------------------------------------------"
156+
cp -f "${OLD}/pg_hba.conf" "${OLD}/pg_ident.conf" "${PGDATA}"
157+
echo "-------------------------------------------------------------------"
158+
echo "Copying the old pg_hba and pg_ident configuration files is complete"
159+
echo "-------------------------------------------------------------------"
160+
161+
# Copy any reindex counter files
162+
echo "--------------------------------------------------------------"
163+
echo "Copying reindex files across"
164+
echo "--------------------------------------------------------------"
165+
cp -f ${OLD}/OR_REINDEX_* ${PGDATA}
166+
echo "-------------------------------------------------------------------"
167+
echo "Copying reindex files is complete"
168+
echo "-------------------------------------------------------------------"
169+
170+
# Remove the left over database files
171+
echo "---------------------------------"
172+
echo "Removing left over database files"
173+
echo "---------------------------------"
174+
set +e
175+
rm -rf "${OLD}" "${NEW}" delete_old_cluster.sh
176+
set -e
177+
echo "---------------------------------------------"
178+
echo "Removing left over database files is complete"
179+
echo "---------------------------------------------"
180+
181+
echo "**********************************************************"
182+
echo "Automatic upgrade process finished with no errors reported"
183+
echo "**********************************************************"
184+
fi
185+
23186
# Do re-indexing check
24187
if [ "$OR_DISABLE_REINDEX" == 'true' ] || [ -z "$OR_REINDEX_COUNTER" ]; then
25188
echo "REINDEX check is disabled"

0 commit comments

Comments
 (0)