What is the Data Lake?
The CHDC Data Lake is a staging data lake: a place where many vendors send files, and where analysts and other systems can then query that data safely. It runs on plain servers (systemd services, no Docker, no Kubernetes) and is built to be repeated on several bare-metal servers in a data centre. Every server step is a script, so a server can be rebuilt from nothing.
What it does
Section titled “What it does”| Need | How the lake answers it |
|---|---|
| Process most file types | Any file is accepted and stored untouched. CSV and TSV are processed today; other types are kept and marked quarantined until a processor exists. |
| Bronze / silver / gold storage | Every processed file is written to three layers of tables. See Data model. |
| Data and metadata through a secured API | Logins use Keycloak. Data leaves only through API keys, the SQL workspace, or CSV export, and every action is audited. |
| Analysts run queries and publish them | The SQL workspace runs queries on Trino. A saved query becomes a view that is built through bronze, silver and gold and can be published as a live API. |
| Vendors upload by API and web page | One upload path for both. Same registry, same audit, same status tracking. |
| Pull from vendor systems | NiFi is installed for pull flows (databases, APIs, SFTP). Flows are still to be built. |
Who uses it
Section titled “Who uses it”- Vendors send files and watch them being processed. A vendor login sees only its own vendor.
- Analysts read the data of the projects they are given, run SQL, export results, save views, publish APIs and use notebooks.
- Admins create vendors and analysts, decide who may read what, and watch activity, audit and system health.
What it deliberately does not do yet
Section titled “What it deliberately does not do yet”Honest limits, so nobody is surprised:
- Excel (
.xlsx) is stored but not yet converted for processing. - There is no approval step before an analyst publishes an API.
- Refresh of a published view happens when its source data changes or on demand, not on a time schedule.
- The platform is proven on one staging server. A full rebuild on an empty server, and a multi-server layout, are the next proof points.
How to read the rest of this site
Section titled “How to read the rest of this site”The site follows the way a new person learns the system: concepts first (Architecture), then the guides for each kind of user, then the applications we wrote, then every server app and script one by one.