Spark

What it is for

Spark does the heavy lifting of turning a file into bronze, silver and gold tables. You normally never start a job yourself: the worker launches one spark-submit per uploaded file (and one per Archive or Delete). Spark is the engine, the worker decides what and when.

Iceberg support is added with two jars; the catalog lake points at Nessie; files are read from s3a://raw/... and tables are written through Iceberg’s own S3 client.

Where it lives


Software	`/opt/spark-<version>` with symlink `/opt/spark` (Spark 4.0.x, Scala 2.13, Hadoop 3)
Extra jars in `/opt/spark/jars`	`iceberg-spark-runtime-4.0_2.13-<v>`, `iceberg-aws-bundle-<v>`, `hadoop-aws-<hadoop v>`
Config	`/etc/lake/spark/spark-env.sh`, `/etc/lake/spark/spark-defaults.conf`
Services	`spark-master` and `spark-worker` (step 07)
Ports	7077 master, 8081 master web page, 8083 worker web page
Work and logs	`/data/spark/work`, `/data/spark/logs`, `/data/spark/tmp`

Size: SPARK_WORKER_CORES and SPARK_WORKER_MEMORY in cluster.env (staging: 6 cores, 24 GB). The worker gives each job JOB_CORES (2), so a few jobs can run side by side, but the worker itself sends one file at a time.

Check it

systemctl is-active spark-master spark-worker
curl -s http://127.0.0.1:8081/json/ | head -c 400        # master status as JSON: workers, cores, applications

Web pages (through the tunnel): master http://localhost:18081, worker http://localhost:18083. The master page lists running and completed applications, and each application’s executors and logs.

How to use it

For an admin, a SQL shell as the right user (it can read the secrets):

lake-spark-sql -e "SHOW NAMESPACES IN lake"
lake-spark-sql -e "SELECT count(*) FROM lake.silver.chdc_health__cleaning__hospitals"

lake-spark-sql is a small wrapper installed by step 07: it runs spark-sql as the user lake with SPARK_CONF_DIR=/etc/lake/spark.

To see why a file failed, read the worker’s log (journalctl -u lake-worker, it shows the STAGE and ERROR lines), then the application’s stderr on the master page.

To run a job by hand (rare): sudo -u lake env SPARK_CONF_DIR=/etc/lake/spark /opt/spark/bin/spark-submit --master spark://127.0.0.1:7077 /opt/lake-worker/process_csv.py ... (see the arguments in worker.py).

Configure it

Everything is rendered by step 07 from cluster.env. The catalog block in spark-defaults.conf:

Setting	Value
`spark.sql.extensions`	Iceberg’s extensions
`spark.sql.catalog.lake`	`org.apache.iceberg.spark.SparkCatalog`, implementation `NessieCatalog`, uri `<NESSIE_URI>/api/v2`, ref `main`
`spark.sql.catalog.lake.warehouse`	`s3://lake/warehouse`
`spark.sql.catalog.lake.io-impl`	`S3FileIO`, path-style, endpoint and keys from `cluster.env`

Step 07.5 appends the S3A block (between # BEGIN s3a / # END s3a) for reading s3a:// paths. Step 07 rewrites the whole file, so always re-run 07.5 after 07.

Two environment variables in spark-env.sh keep the AWS SDK compatible with SeaweedFS: AWS_REQUEST_CHECKSUM_CALCULATION=when_required and AWS_RESPONSE_CHECKSUM_VALIDATION=when_required.

Things that go wrong

Symptom	Likely cause
Job fails with `NoClassDefFoundError ... software/amazon/awssdk`	The AWS SDK v2 bundle jar is missing: add the version named by `aws-java-sdk-v2.version` in the Hadoop POM
File stuck in processing	The job died or the worker restarted; the worker moves such files back to `uploaded` on start, or use Process again
Files wait but nothing runs	`spark-master` has no registered worker: `systemctl status spark-worker`
A local driver cannot use the cluster	Executors must connect back to the driver, which fails through tunnels. Use `master=local[*]` for local work, or run on the server

Spark’s ports are not in the default tunnel list (7077) on purpose.