Server apps, one by one

Each page answers the same questions in the same order: what it is for, where it lives, how to check it, how to use it, how to configure it, and what goes wrong.

The whole list

App	What for	systemd unit	Port (127.0.0.1)	Install step
SeaweedFS	S3 object storage	seaweed	8333 S3, 9333 master, 8888 filer, 8082 volume	01, 03.5, 04
PostgreSQL	Registry, catalog store, logins	postgresql	5432	02, 14
Nessie	Iceberg catalog	nessie	19120 (9000 management)	05
Trino	SQL over the lake	trino	8080	06, 06.5
Spark	Processing engine	spark-master, spark-worker	7077 master, 8081 master UI, 8083 worker UI	07, 07.5
Keycloak	Logins and roles	keycloak	8180 web, 9180 health	13
JupyterHub	Notebooks	jupyterhub	8000 (8001, 8091 internal)	18
NiFi	Pull files from vendor systems	nifi	8443 (https)	09
PostgREST	REST over PostgreSQL (older path)	postgrest	3000	03, 08
lakestatus	One-screen health report	(a command)	none	17
The worker	Turns files into tables	lake-worker	none	15 (Worker)

Everything listens on localhost. See Developer helpers for the tunnel that makes these reachable from your machine, and never open these ports to the internet.

Where things live

What	Location
Software	/opt/<service> (versioned folder plus a symlink, for example /opt/trino > /opt/trino-484)
Configuration	/etc/lake/<service>/ (mode 640, group lake)
The one settings file	/opt/lake/cluster.env (mode 640)
Install scripts and SQL	/opt/lake/scripts/, /opt/lake/sql/
Big data	/data/seaweed, /data/trino, /data/spark, /data/nifi (a separate large disk)
Notebook homes	/var/lib/nb-<name>-<hash>
Logs	journalctl -u <unit>; some also write files (Spark in /data/spark/logs, NiFi in /opt/nifi/logs)

Commands that work for every service

systemctl is-active seaweed postgresql nessie trino spark-master spark-worker keycloak jupyterhub nifi lake-worker postgrest
systemctl status trino                  # state, last log lines
sudo systemctl restart trino            # restart one service
journalctl -u trino -n 100 --no-pager   # last 100 log lines
journalctl -u trino -f                  # follow live
sudo ss -ltnp | grep -E 'java|postgres|weed|postgrest|python|node'   # who listens where
lakestatus                              # everything on one screen

Start order and dependencies

postgresql, seaweed            (no dependencies)
   nessie                      needs postgresql, seaweed
   trino                       needs nessie, seaweed (and postgresql for the serving catalog)
   spark-master > spark-worker needs nessie, seaweed
   keycloak                    needs postgresql
   lake-worker                 needs postgresql, spark-master
   jupyterhub                  needs keycloak, trino
   nifi, postgrest             need seaweed / postgresql respectively

After a reboot, services start by themselves (they are enabled). If one is failed, start its dependency first and restart it. A reboot-restart test of the whole stack has not been done on staging yet (Keycloak in particular), so check lakestatus after any restart.