StackMAX DATA AI LAKE · Technical deep dive
How the data lake is built, zone by zone.
From the sources that feed it to the zones that refine the data, the four basic functions, the technologies behind each one and the infrastructure it runs on.
One central repository
Every kind of data, in one place.
StackMAX DATA AI LAKE is a central repository for structured and unstructured data at scale. A set of open-source building blocks comes prebuilt into one framework, so teams start from a working platform instead of assembling one.
Data lake architecture
From raw data to trusted insight, in zones.
Data can come in from anywhere, in its native form. It first lands in a transient loading zone for basic quality checks, is stored in the raw zone, and is refined into trusted data for discovery and consumption. Governance runs underneath every zone.
Reports, what-if analysis and the insight behind informed decisions.
Underneath every zone sits an integration platform that manages, monitors and governs metadata, data quality, the data catalog and security, built here on Atlas, Ranger, Knox and Kerberos.
Managed ingestion
Managed ingestion controls how data is ingested, where it comes from, when it arrives and where it is stored. Data lands in a staging area first and is either admitted or discarded, so governance rules apply to everything that enters the lake.
Master data
The basic datasets, cleansed and validated. In a healthcare organization, for example, member names and addresses, plus attributes such as dates of birth.
Reference data
The single source of truth for blended datasets, built by merging tables from the master data. Anyone who needs member data reads it from here and can rely on it.
Four basic functions.
Every lake moves data through the same four stages, from many sources to final consumption, under one layer of management and governance.
Ingestion
- Scalable and extensible, capturing streaming and batch data
- Business logic, filters, validation, data quality and routing applied on the way in
Storage and retention
- Data placed in HDFS, Hive, HBase, search or in-memory, depending on the requirement
- Metadata management
- Policy-based data retention
Processing
- Batch and near-real-time use cases
- Workflows for repeatable processing
- Handling of late-arriving data
Access
- Dashboards and applications that deliver business insight
- Data available to consumers through APIs, message queues and database access
Also in earlier reference designs: Cloudera Manager · Cloudera Navigator
Access follows rights. A retailer, for example, might show customer names and contact data to sales and customer service, but expose sensitive personal or financial fields only to finance. Policy-based retention decides how long data stays in the raw repository and where it is archived, so old data stays queryable without going to tape.
Batch or streaming, by use case.
Processing happens when data is read, so different users can standardize and transform the same data in different ways. The tools depend on whether the use case is batch or streaming.
Scheduled, high-volume work
Traditional ETL workloads, for example processing billing information to produce a daily operational report.
Typical tools: Hive, Spark, MapReduce
Results that cannot wait
Reports needed in real time or near real time, for example a courier company that needs the current location of every truck.
Typical tools: Kafka, Flink, Spark streaming
Features and benefits
Less hardware, deployed where you are.
Open-source Hadoop with zero-touch deployment and operations, built to cut hardware and license cost without locking you in. Run it on the environment you already have, or on the SMX.IQ virtualization platform.
- Open-source Hadoop
- Uses less hardware
- Zero-touch deployment
- Zero-touch operations
- Infrastructure monitoring
- No vendor lock-in
- Drastic cost reduction
- Less hardware consumption
- Easy migration from existing Hadoop distributions such as Cloudera and Hortonworks
- Tailored to your needs
- No vendor lock-in
- Deploy on your existing environment, or on the SMX.IQ virtualization platform
Deploy on the environment you already have.
VMwareMicrosoftOpenStackRed HatAWSGoogle CloudIBM CloudAzure
A fraction of the nodes for the same data.
A conventional Hadoop deployment is hardware-hungry. Sizing a platform to the environment at hand needs far fewer compute nodes.
About 5x fewer compute nodes.
65% fewer compute nodes, from the OCTO Telematics case study.
Sizing figures are indicative and depend on workload, data profile and the validated design of each deployment.
Building-block catalog.
The ecosystem of open-source building blocks used across deployments. Highlighted components are part of the current validated StackMAX DATA AI LAKE stack; the rest are selected, and their support level agreed, per project.
HadoopSparkKafkaHBaseAmbariZooKeeperAtlasHiveTezRangerRanger KMSKnoxLivyOoziePhoenixZeppelinNiFiAirflow AccumuloCalciteDataFuPigSqoopStormFalconFlumeMiNiFiHueElasticsearchKibanaCassandraR ServerTalend
Advanced capabilities
Enterprise data and knowledge, presented as simple choices.
Open table formats and time travel
Hive ACID and Iceberg tables on HDFS and Ozone with schema evolution, snapshots and hidden partitioning, readable by Hive, Spark, Flink, Trino and any Iceberg-compatible engine.
Federated SQL without copying
Query platform tables, operational databases and external systems in one SQL layer, with Ranger policies applied on the way.
Streaming and change data capture
Capture changes and events with NiFi and Kafka, enrich them in flight with Flink, and keep tables, indexes and context current.
Hybrid retrieval with citations
Keyword, vector and graph results scored together, filtered by tenant, purpose and Ranger tags, re-ranked and returned with provenance.
Graph-enhanced RAG and impact analysis
Multi-hop reasoning over dependencies, timelines and rules for root-cause analysis, impact assessment and explainable answers.
Governed memory and agent harness
Agent memory that is classified, scoped, aged and deletable, and an execution harness with Ranger-governed tool permissions, human approval, rollback and a kill switch.
Advanced capabilities depend on the validated storage, processing, retrieval and architecture selected for each deployment.
Production data platform
Open data without the integration burden.
StackMAX DATA AI LAKE is designed, deployed, validated and supported as one production platform, rather than an unsupported collection of Apache projects, connectors and scripts.
Validated architecture
Storage, table formats, catalog, engines, search, vector and graph stores, context services, identity and observability are tested as a complete solution.
Controlled lifecycle
Every pinned component follows a documented release, patching and upgrade policy, with Ambari-driven rolling upgrades.
Sovereign and air-gapped ready
Data locality, customer-controlled keys, tenant isolation, local repositories and controlled update procedures inside your security boundary.
Flexible operating model
Customer-operated, co-managed, fully managed or a dedicated SRE Pod.
Where it runs.
The Hadoop cluster is happiest on servers with local disks, and smaller clusters can run virtualized on StackMAX; the newer services want containers. StackMAX DATA AI LAKE runs both on the same infrastructure, with one identity and one security configuration shared between them.
Two reference configurations
Small or virtualized clusters: HDFS runs on StackMAX virtual machines with block storage (SAN), or as KVM virtual machines on HCI. Larger clusters: HDFS, Ozone, YARN and HBase data nodes on servers with local disks or bare metal. Node labels cover GPU and memory-heavy nodes in both.
Private cloud compute
Clusters as OpenStack projects with quotas and self-service, Nova instances for cluster roles, Ironic for bare-metal data nodes, and a Ceph S3 tier for cold data and backups.
Kubernetes
Trino, Superset, OpenSearch, Milvus, JanusGraph services, the Context and Agent services and the model gateway, deployed with Helm, sharing the cluster's Kerberos realm and Ranger policies.
Air-gapped sites run every component inside the boundary with local package, chart, image and model repositories; updates arrive as signed offline bundles.
Overview
Why a data lake, use cases and a real case study.
The overview page covers the AI extensions, the business case, the comparison with a data warehouse, use cases, the OCTO Telematics case study, migration paths and the FAQ.
Build what the world depends on.
Planning a cloud for government, financial services, pharmaceuticals, healthcare, energy, transport, industry, service-provider operations or private AI? Define the architecture, security boundary and operating model your environment requires.
Or email us directly at webinquiries@smxiq.com
Apache, Hadoop, HDFS, Ozone, YARN, Hive, HBase, Phoenix, Tez, Spark, Flink, Kafka, NiFi, Oozie, Airflow, Livy, Zeppelin, Superset, Ranger, Knox, Atlas, Ambari, Iceberg, ZooKeeper, Trino, Ceph, OpenSearch, Milvus, JanusGraph, Temporal, Kerberos, Cloudera, Databricks, Snowflake, Microsoft, Azure, Amazon, AWS, Google Cloud, IBM, VMware, Red Hat, OpenStack, Rancher, Zabbix, Ansible, Open vSwitch, Hortonworks, MapR, Qlik, Tableau, Spotfire, Talend, Elasticsearch, Kibana, Cassandra, OCTO and other product and company names are trademarks of their respective owners and are named for identification and comparison only. No affiliation or endorsement is implied.