This follows on from #19030.

Further testing with the customer has shown the issue to not be reproducible on a modern version of Coder, and after further discussion with Dean it was agreed the logical way to solve this problem was to offer an additional method of reading the database connection details from a file on disk. In addition to reading from the file on disk the implementation should include support for re-reading the contents of the file when initiating a new connection to the database in order to ensure new connections are always initiated using the most up to date credentials.

As testing has shown with minimal load there are still failed authentication attempts to the db when performing a rollout restart immediately after credentials are rotated this demonstrates though Coder uses db connection pools, it is not suitable to prevent a situation where old and invalid credentials are still attempted for use.

A potential implementation of this functionality could then be configured as follows:

Helm chart:

  env:
    - name: CODER_PG_CONNECTION_URL_FILE
      value: "/etc/coder/pg/pg-url.txt"

  volumes:
    - name: coder-pg-url-file
      secret:
        secretName: coder-pg-url-file

  volumeMounts:
    - name: coder-pg-url-file
      mountPath: /etc/coder/pg
      readOnly: true

Preparation:

kubectl create secret generic coder-pg-url-file \
  -n pg-coder \
  --from-file=pg-url.txt=./pg-url.txt

cat pg-url.txt     
postgres://coder_pg:coder_pg@my-local-db-host.tld:5432/coder_pg?sslmode=require