cluster.env is the only file that differs between staging (one machine) and production (several). It is KEY=value lines; ${OTHER} is expanded; # comments are allowed.
- On a server it is
/opt/lake/cluster.env, mode 640, owner = the admin login, group lake (step 19).
- Never commit it (
.gitignore blocks *.env except *.env.example). The template is infra/cluster.env.example.
- Replace every
CHANGE_ME. Generate secrets with openssl rand -hex 16 (use hex, not base64: /, + and = break connection strings).
- In a shell:
set -a; source /opt/lake/cluster.env; set +a. If $S3_ENDPOINT is empty, this was forgotten.
- On a developer machine the apps read
cluster.env, then the repo-root local.env (tunnel ports, no secrets), then real environment variables; the later one wins.
Install scripts render cluster.env into each service’s own file in /etc/lake/<service>/. No running service reads cluster.env. After you change a value, re-run the install step of every service that uses it (install-all.sh <step>).
| Setting |
Meaning |
Staging |
INGEST_HOST |
NiFi, Nessie, PostgREST (and Keycloak) |
127.0.0.1 |
PROCESS_HOST |
Spark and the worker |
127.0.0.1 |
QUERY_HOST |
Trino and PostgreSQL |
127.0.0.1 |
STORAGE_HOST |
SeaweedFS |
127.0.0.1 |
In production these become the real host names of four servers.
| Setting |
Meaning |
S3_ENDPOINT |
http://${STORAGE_HOST}:8333 |
S3_REGION |
us-east-1 (any value; S3 libraries require one) |
S3_ACCESS_KEY, S3_SECRET_KEY |
The storage identity (secret) |
RAW_BUCKET, LAKE_BUCKET |
raw, lake |
PG_HOST, PG_PORT |
PostgreSQL host and port |
PG_ADMIN_PASSWORD, NESSIE_DB_PASSWORD, POSTGREST_AUTH_PASSWORD, LAKE_WRITER_PASSWORD, INGEST_DB_PASSWORD, KEYCLOAK_DB_PASSWORD |
One password per database role (secrets) |
NESSIE_URI |
http://${INGEST_HOST}:19120 |
Each downloaded program is pinned. To upgrade change both lines; a script run with an empty checksum prints the real value and stops so you can compare it with the project’s release page.
| Version |
Checksum |
SEAWEED_VERSION |
SEAWEED_SHA256 |
POSTGREST_VERSION |
POSTGREST_SHA256 |
NESSIE_VERSION |
NESSIE_SHA256 |
JAVA25_URL |
JAVA25_SHA256 |
TRINO_VERSION |
TRINO_SHA256, TRINO_CLI_SHA256 |
SPARK_VERSION |
SPARK_SHA512 |
ICEBERG_VERSION |
(checked against Maven Central) |
NIFI_VERSION |
NIFI_SHA512 |
KEYCLOAK_VERSION |
KEYCLOAK_SHA256 |
CHP_VERSION |
(npm) |
| Setting |
Meaning |
TRINO_HEAP |
Trino’s Java heap. About 25 % of RAM on a shared box, 30-50 % when Trino has it alone |
SPARK_WORKER_CORES, SPARK_WORKER_MEMORY |
What the Spark worker may use (staging: 6 and 24g) |
NIFI_HEAP |
NiFi’s heap (4g) |
| Setting |
Meaning |
NIFI_USER, NIFI_PASSWORD |
NiFi’s single user; the password must be 12+ characters |
KEYCLOAK_ADMIN_PASSWORD |
Keycloak’s admin account; also used by the API to create users (planned: a limited service account) |
DEV_USER_PASSWORD |
Password of the dev test users; development only |
JUPYTERHUB_CLIENT_SECRET |
openssl rand -hex 32; the secret of the Keycloak client chdc-notebook |
NIFI_URL, SPARK_WORKER_UI, SEAWEED_MASTER_UI, SEAWEED_FILER_UI: default to the role host plus the usual port. Set them when admins reach services through another name (VPN host name or reverse proxy).
| Setting |
Default |
JUPYTERHUB_PUBLIC_URL |
http://localhost:18000 (production: the real https address) |
KEYCLOAK_PUBLIC_URL |
http://localhost:18180 |
NOTEBOOK_MEM_LIMIT |
4G |
NOTEBOOK_CPU_LIMIT |
2 |
NOTEBOOK_IDLE_MINUTES |
60 |
| Setting |
Default |
Meaning |
WEB_API_PORT |
4000 |
Where the API listens |
UPLOAD_MAX_MB |
500 |
Largest single upload |
SQL_MAX_ROWS / SQL_MAX_RESULT_MB / SQL_TIMEOUT_SECONDS |
1000 / 10 / 60 |
Limits of the SQL page |
SQL_MAX_PER_USER / SQL_MAX_TOTAL |
2 / 6 |
Concurrent queries |
SQL_EXPORT_MAX_ROWS / SQL_EXPORT_MAX_MB / SQL_EXPORT_TIMEOUT_SECONDS |
1,000,000 / 500 / 600 |
CSV export limits |
SQL_EXPORT_MAX_PER_USER / SQL_EXPORT_MAX_TOTAL |
1 / 2 |
Concurrent exports |
AUDIT_RETENTION_DAYS / API_LOG_RETENTION_DAYS / AUDIT_RETENTION_ENABLED |
365 / 90 / true |
How long logs are kept |
AUDIT_SEARCH_SCAN / AUDIT_QUERY_TIMEOUT_MS / AUDIT_MAX_CONCURRENT |
5000 / 5000 / 2 |
Audit search protection |
VIEW_REFRESH_ENABLED / VIEW_REFRESH_POLL_SECONDS / VIEW_RUN_TIMEOUT_SECONDS / VIEW_BRONZE_KEEP_RUNS |
true / 15 / 1800 / 3 |
View builds |
API_MAX_ROWS / API_DEFAULT_ROWS / API_RATE_PER_MIN / API_MAX_CONCURRENT / API_TIMEOUT_SECONDS / API_RESULT_MAX_MB |
1000 / 100 / 60 / 8 / 30 / 10 |
Published API limits |
PUSH_RATE_PER_MIN |
120 |
Push API calls per key per minute |
NOTEBOOK_URL |
unset |
Address of JupyterHub for the “Notebooks” menu link |
Ports for your machine through the tunnel: PG_PORT=15432, S3_ENDPOINT=http://localhost:18333, NESSIE_URI=http://localhost:29120, TRINO_HOST=localhost, TRINO_PORT=18080, KEYCLOAK_URL=http://localhost:18180, and so on. See Developer helpers.