StackMAX DATA AI LAKE · Open data platform for AI
Enterprise Hadoop data platform, extended for AI.
StackMAX DATA AI LAKE is an enterprise distribution of the Apache Hadoop ecosystem, the same open stack that open data platform distributions ship: HDFS and Ozone storage, YARN, Spark, Flink, Hive, HBase, Trino, Iceberg tables, Kafka, NiFi, Kerberos, Ranger, Knox and Atlas, managed with Ambari. On top, it adds the layers AI needs as extensions of the same components, not a second platform: search and a vector database, a knowledge graph, the Context Lake and the Agent Lake.
Everything runs in your data center, sovereign cloud or provider environment, with your data, keys, identity and operations under your control.
Open source by default: Apache Hadoop, Ozone, YARN, Hive, HBase, Phoenix, Spark, Flink, Tez, Kafka, NiFi, Oozie, Airflow, Iceberg, Ranger, Knox, Atlas and Ambari, with OpenSearch, Milvus and JanusGraph for the AI extensions. Upstream-aligned, customer-operated, co-managed or fully managed.
Architecture
One open data platform. Four AI extensions.
Storage, processing, resource management, data access, security and orchestration, component by component. The four AI extensions sit on top and reuse them: the lake stores their data, YARN and Kubernetes run their jobs, Ranger and Atlas govern their output.
Hybrid retrieval with citations
Multi-hop reasoning, impact analysis
Identity-aware, cited, budgeted evidence
Governed agents, tools and memory
Component names are the upstream projects as packaged in each StackMAX DATA AI LAKE release. No deployment needs every component; the validated selection is documented in its design.
The AI extensions
Same stack, extended.
An open data platform is a data platform, not an AI engine. StackMAX DATA AI LAKE keeps it that way: the extensions reuse the storage, compute, catalog and security of the stack, and models remain replaceable consumers behind a gateway.
Search and Vector Database
Keyword, semantic and hybrid retrieval over documents, tables and events, with citations.
Extends: Spark on YARN for embedding jobs · Ozone or Ceph S3 for index segments · Ranger tags for filtering · OpenSearch and Milvus as the engines
Knowledge Graph
Entities, relationships, dependencies and rules for multi-hop questions, impact analysis and graph-enhanced retrieval.
Extends: HBase as the graph store through JanusGraph · Atlas lineage and classifications as graph input · Spark and Flink for entity extraction
Context Lake
The minimum necessary evidence for a user, task, model and policy, assembled from SQL, search, vectors and the graph, ranked, compressed and cited.
Extends: Hive and Trino SQL · Kafka and Flink for freshness · Knox single sign-on · Ranger policies for entitlement
Agent Lake
Agents, tools, prompts and memory as governed services, with the model gateway that routes to private or approved models.
Extends: HBase and Iceberg for registries and memory · Hive Metastore as agent tools · Ranger audit for every agent action · Kubernetes for the runtime
| AI extension | Builds on | What it adds | Governed by |
|---|---|---|---|
| Search and vector database | Spark jobs on YARN chunk and embed documents and tables; Ozone or Ceph S3 holds index segments; Hive and Iceberg tables hold the sources. | OpenSearch for keyword and hybrid search, Milvus for vector indexes at scale, an embedding lifecycle that records source snapshot, model and version, and a Retrieval API with re-ranking and citations. | Ranger tags carried from source tables to every chunk and vector; Knox single sign-on; Atlas lineage from table to index. |
| Knowledge graph | HBase as the storage backend, with the Atlas lineage, classifications and glossary the platform already maintains. | JanusGraph on HBase as the property graph, OpenSearch as its index, entity and relationship pipelines, ontology support, and graph expansion inside the Retrieval API. | Ranger policies on entity types and sources; Atlas classifications as graph labels; every edge carries its source and timestamp. |
| Context Lake | Hive and Trino for facts, Kafka and Flink for live state, Hive Metastore and Atlas for what exists and who owns it, Knox for the caller's identity. | A Context API that assembles identity-aware, task-relevant, fresh and policy-filtered evidence, ranks and compresses it, and returns provenance, policy decision and a token budget. | Ranger entitlements evaluated per request; Atlas classifications drive masking and exclusion; context expiry rules; audit to SMX.IQ monitoring. |
| Agent Lake | HBase and Phoenix for registries, Iceberg for memory and traces, Kafka for events, YARN or Kubernetes for the runtime. | Agent, tool and prompt registries, a harness with sandboxed tools, approvals, rollback and a kill switch, governed memory, durable workflows on Temporal, and the model gateway. | Ranger policies per agent identity; Ranger audit of every tool call and data access; Atlas lineage from memory to source. |
| Model access | The same Kerberos and Knox identity, the same Ranger audit, GPUs from node-labeled YARN nodes or the SMX.IQ AI platform. | A gateway that routes to private models first and to approved public providers only by policy, with metering per tenant and application. | Per-tenant routing policies; prompt and response retention per application. |
Every derived asset, from a chunk or vector to a graph edge, a context record or an agent memory, stays linked to its source table, Iceberg snapshot, Ranger policy and retention state, so corrections and deletions propagate from the source outward.
Data and AI, simplified
Everything needed to deliver data and knowledge as a service.
Teams register the data they own. StackMAX DATA AI LAKE coordinates storage, cataloging, processing, indexing, retrieval, context and access behind the scenes, on one stack.
Open Data Platform
Raw and curated data in open formats on storage you own: HDFS and S3-compatible Ozone, with Hive and Iceberg tables, Hive Metastore and Atlas for catalog and lineage, versioning and time travel. Query it with SQL and Spark without copying it into a proprietary warehouse.
Ingestion, Streaming and Processing
Batch files, change data capture and event streams with NiFi and Kafka; enriched in flight with Flink; transformed at scale with Spark on YARN; queried through Hive and Trino.
Search, Vector Database and Knowledge Graph
Documents, tables and events become keyword indexes, vector embeddings and a graph of entities and relationships, computed on the cluster and stored on the same platform.
Context Lake and Agent Lake
Identity-aware, fresh, ranked and cited context for models and agents, plus agents, tools, prompts and memory managed as governed services with approvals, rollback and a model gateway.
Identity and Governance
Separate tenants, queues, projects and workspaces with roles, attributes, classification, retention and policy. Enterprise single sign-on through Knox and one Ranger audit trail across data, retrieval and agent actions.
One platform, many interfaces
Use it through the console, Ambari, SQL and notebooks, S3 and Kafka, the Retrieval and Context APIs, or OpenTofu/Terraform.
A data platform first
Five questions an AI-ready data platform must answer.
Before a model is trained or an agent is given a tool, the platform has to answer for its data. The enterprise Hadoop stack answers these for tables; the extensions make the same answers hold for chunks, vectors, graph edges, context records and agent memory.
Where did the data come from?
Atlas captures lineage from NiFi ingestion through Hive, Spark and Iceberg. The chain extends to the embedding job, the index, the graph edge and the context record, so an answer traces back to a source table and snapshot.
Who can access it?
Kerberos authenticates, Knox signs on, Ranger decides: dataset grants, column masking, row filters and tag-based policies. The same tags travel with every chunk and vector.
What did it look like at training time?
Iceberg snapshots give every training run, embedding batch and evaluation a snapshot identifier, so a result can be reproduced.
Did sensitive data enter the pipeline?
Atlas classifications and Ranger masking keep classified columns out of embeddings and prompts by policy; deletions propagate to indexes, graph and memory.
Where does the data reside?
On premises, in a sovereign private cloud or in an air-gapped site, with no runtime dependency on an external service. Keys stay in Ranger KMS.
Where does the model fit?
Outside the lake, as a replaceable consumer. The lake owns the data, the context and the audit trail, whichever model is used.
The platform supports these controls; it does not make a system compliant by itself. The assessment maps regulatory obligations, such as the EU AI Act's data governance and record-keeping articles, to the controls each workload needs.
How it works
One lifecycle, from source data to governed context.
Govern
Inventory and classify sources in Atlas; assign owners, Ranger policies, retention and authorized purpose before anything is embedded.
Ingest
Load batch, stream and CDC data with NiFi and Kafka; validate, catalog and store it in open zones on HDFS and Ozone; transform with Spark.
Index
Extract entities, create embeddings, build OpenSearch and Milvus indexes and update the JanusGraph knowledge graph, as jobs on the cluster.
Contextualize
Filter by identity and Ranger policy; rank, compress and cite the evidence; enforce freshness and expiry.
Serve and observe
Deliver context to applications, models and agents; trace, evaluate, correct and delete with propagation, observed in SMX.IQ monitoring.
Why a data lake
All your data, without remodeling it first.
A data lake is a central place to store all your data, whatever its source or format, structured or unstructured. Tools from the extended Hadoop family then extract value quickly. Because all data is welcome, a lake answers the integration problems of a traditional enterprise data warehouse (EDW), especially as organizations move to mobile and cloud applications and IoT.
Unlimited kinds of data
Store structured and unstructured data, from CRM records to social media posts.
No need for every answer upfront
Store raw data now and refine it as your understanding and insight improve.
No limit on how you query
Use a variety of tools to find out what the data means.
No more silos
Democratized access, with a single, unified view of data across the organization.
The business case.
Warehouses are rigid, expensive to change and usually need IT for any manipulation or enrichment of data. Lakes remove most of that friction, which is why almost every industry has a use case.
Freedom from a single data model
Store everything from blog posts to product reviews. Data does not have to be consistent: the same kind of information can arrive in very different formats, with no schemas defining the integration points.
Streaming data
Streaming has moved from rare cases such as IoT sensors and market data to everyday data such as social media. A lake takes streams as naturally as files.
The right tool for the task
With Spark or MapReduce, preparing data in a warehouse can take longer than the analysis itself. In a lake there is little prep, and schema-on-read lets each query define its own schema.
Easier access
Bring together ever larger volumes for analytics, or simply keep them for future use. Modeling waits until the data is used, which opens better operational insight and discovery.
Scalability
Big data sits at the intersection of volume, variety and velocity. Warehouses hit architectural limits; Hadoop-based lakes scale to petabytes cost-efficiently.
Less dependence on IT
With a governed lake, business users can enrich, clean up and explore data themselves instead of queuing for IT at every step.
Three traits of a true data lake.
A single shared repository
One repository, typically HDFS or Ozone, which for smaller or virtualized clusters can run on StackMAX virtual machines with block storage (SAN). Data is kept in its original form, with changes and context captured across its lifecycle, which helps compliance, audit and provenance. In a warehouse, transformed and aggregated data is hard to piece back together.
Orchestration and job scheduling
Workload execution is a prerequisite for enterprise Hadoop. YARN provides resource management and a central platform for consistent operations, security and data governance across clusters, so analytic workflows get the data and the compute they need.
Applications and workflows that act on the data
Easy user access is a hallmark of a lake: structured, semi-structured and unstructured data is loaded as is. Data owners can consolidate customer, supplier and operations data, removing technical and even political roadblocks to sharing it.
Data warehouse and data lake, side by side.
| Attribute | Enterprise data warehouse | Data lake |
|---|---|---|
| Schema | Schema-on-write: data is modeled before it is loaded. | Schema-on-read: structure is applied when data is queried. |
| Data | Cleansed and conformed to an enterprise model. | Raw in its native form, with curated zones on top. |
| Scale and cost | Large volumes at moderate cost. | Very large volumes at low cost on commodity hardware. |
| Access | Standard SQL and BI tools. | SQL, notebooks, developer programs, APIs and BI tools. |
| Workload | Batch, plus many concurrent interactive users. | Batch, streaming and interactive queries on the same data. |
| Efficiency | Complex integration, but efficient use of CPU and I/O. | Complex processing, but very efficient use of storage and compute at low cost. |
| Agility | Changing models and mappings is slow and often needs IT. | Models and queries can change as understanding improves. |
| Strengths | Transform once, use many. Clean, secure data. One enterprise-wide view. High concurrency, consistent performance, fast responses. | Changes the economics of storing large volumes. Scales out across many servers. Any tool. Analysis can start as soon as data arrives. Structured and unstructured content in one store. |
The lake does not replace the warehouse where a governed, high-concurrency reporting layer is the right tool. It removes the need to model everything before you can use it.
Why storage wastes less.
In a warehouse, combining a 200-field source with a 400-field source means adding 400 new columns to the first table. Rows from each source leave the other source's columns empty: the sparse table problem. A lake stores each source as it is, so no empty rows or columns are created.
source A source B empty cells, wasted space
No empty rows or columns.
Illustration scaled down; the example in the text uses 200 and 400 fields.
Use cases
One governed platform, many workloads.
Almost every industry has a data lake use case. These are the ones we see most often.
Customer 360
Bring CRM, support, web and product data into one governed view of each customer.
IoT and 5G telemetry
Ingest device and network streams at scale and analyze them as they arrive.
Streaming ETL
Capture, enrich and deliver events in flight instead of waiting for nightly batches.
Event-driven microservices
Use Kafka event streams as the backbone between services and the lake.
Fraud detection
Combine live events with history to flag suspicious patterns quickly.
Mainframe augmentation
Offload and analyze mainframe data on open storage without disrupting core systems.
Clinical data lake
Hold clinical and operational records under strict masking, access and audit controls.
And more
Tell us about your workload and we will map it to the platform.
Deriving value
Data as a service for your users.
The purpose of a lake is to serve users, so they need to answer a few questions without waiting for IT:
- What is in the lake? (the catalog)
- What is the quality and profile of the data?
- What does the metadata say about it?
- How can I enrich, clean up and aggregate it myself?
- How can I annotate and tag it for others?
Browse, select, explore, share back
A rich metadata catalog on top of the lake lets users browse available data, filter it and pick what they need, as in an online shop. They get a sandbox for exploratory analysis and feed results back into the lake for others to reuse. Strict onboarding rules, naming conventions and role-based access keep it safe.
Proven in production
Case study: a connected-vehicle data lake.
OCTO Telematics runs IoT vehicle telematics: it began in the insurance sector and is moving into fleet telematics and smart mobility. Its data lake had to cost less to own and be flexible enough to support new applications and offers.
- €3.6M per year in licensing
- 46 compute nodes
- Vendor lock-in
- Third-party managed service
- Legacy technology, no automation
- €0.9M total cost of ownership
- 16 compute nodes (65% fewer)
- No vendor lock-in
- Managed by the platform provider, with 24/7 monitoring and support
- Automated, modern stack, with a 1:1 open-source migration
What replaced what.
| Function | Before | After |
|---|---|---|
| Event streaming | Kafka from a commercial distribution | Open-source Apache Kafka |
| Hadoop data platform | Commercial Hadoop distribution (HDP) | Open-source Apache Hadoop |
| Kubernetes | Cloud provider's managed Kubernetes service | Automated Kubernetes with Rancher |
| Monitoring | VMware vRealize Operations | Zabbix with Ansible automation |
| Storage | VMware vSAN and cloud object storage | Ceph distributed storage |
| Networking | VMware NSX | Open vSwitch |
| Disaster recovery | VMware vCenter SRM | Disaster-recovery module of the platform |
The platform was delivered as modules (data lake, virtualization, disaster recovery, storage, Kubernetes and monitoring) across two European data centers, then moved to a lower-cost local cloud provider. Results are specific to this customer and environment.
Console included, platforms optional
Start with the lake. Add private models and more cloud services when you need them.
With StackMAX DATA AI LAKE
- SMX.IQ console for data services, Ambari for the cluster
- Open data platform: storage, streaming, processing, SQL, governance
- Search, vector database and knowledge graph extensions
- Context Lake and Agent Lake with the model gateway
- Tenants, queues, projects, workspaces, roles and SSO
Add SMX.IQ AI platform
- Private model hosting on your GPUs
- Token-metered model APIs
- Training and fine-tuning on lakehouse data
- Hybrid routing to approved public providers by policy
Add SMX.IQ cloud platform
- More modules in the same console
- Multi-service catalogs and tenant management
- FinOps, metering, billing and payments
These modules are optional and not required to deploy or operate StackMAX DATA AI LAKE.
Migration paths and support packs
A path for every starting point.
Separately scoped support for assessment, migration, cutover and managed operations.
- Cloudera CDP, CDH, HDP and legacy Hadoop: the same Apache APIs, so Hive tables, HBase applications, Spark jobs, Ranger policies and Atlas metadata carry over.
- Azure HDInsight and Amazon EMR: the same open stack on your own hardware, with ADLS and S3 data landed on Ozone or Ceph S3.
- Databricks, Snowflake and cloud data warehouses: Iceberg tables with Spark and Trino, one workload group at a time.
- Proprietary search, SaaS vector databases and graph services: re-indexed from governed tables, with policies attached from the start.
Software supply chain assurance
An SBOM with every release.
Every supported release includes an upstream SBOM and a release SBOM covering components, container images, dependencies, licenses, vulnerabilities, provenance and signed artifacts.
SPDXCycloneDXComponents and imagesLicensesVulnerabilitiesProvenanceSigned artifactsSOC 2-aligned audit evidence
Technical deep dive
Zones, functions, technologies and deployment.
The second page covers every kind of source data, the data lake architecture in zones, the four basic functions with their technologies, advanced capabilities, and where the platform runs.
FAQ
Frequently asked questions
What is StackMAX DATA AI LAKE?
An open-source data platform for analytics and AI, delivered and supported as one product. Its base is an enterprise distribution of the Apache Hadoop ecosystem. On that base are four AI extensions that reuse the same components: search with a vector database, a knowledge graph, the Context Lake and the Agent Lake. It runs in your data center, sovereign cloud or provider environment, and can be designed, validated, supported and operated for you.
Is it a Hadoop distribution? How does it relate to Cloudera?
Yes. The base is an enterprise Apache Hadoop distribution built from current upstream releases, with the fixes needed to keep the components interoperable in production. It is the same project set that Cloudera CDP, CDH and HDP estates run, so Hive tables, HBase applications, Spark jobs, Ranger policies, Atlas metadata and Kerberos integration carry over with version-upgrade testing rather than a rewrite.
How are the AI features extensions rather than a second platform?
Each extension reuses what the platform already has. Spark on YARN computes embeddings; Ozone or Ceph S3 holds vector index segments; HBase is the graph backend; Hive, Trino, Kafka and Flink feed the Context Lake; Kerberos and Knox authenticate every call; Ranger tags travel from source tables to every chunk, vector, edge and agent action. There is no separate data copy, identity or audit trail for AI.
How is it different from a data lake or a lakehouse?
A data lake stores raw data cheaply. A lakehouse adds open table formats, transactions and SQL. StackMAX DATA AI LAKE keeps both and adds semantic search, a knowledge graph, identity-aware context assembly and governed agents and memory, with every derived asset linked to its source, version, policy and retention state.
Where do the models run?
Outside the lake, as replaceable consumers. Spark MLlib runs classical machine learning on the cluster. Private language and embedding models run on your GPUs through the SMX.IQ AI platform, and a model gateway routes to them and, where policy allows, to approved public providers, with every request logged and metered.
Why both a vector database and a knowledge graph?
A vector database finds what is similar in meaning, which suits document search, RAG and recommendations. A knowledge graph records how entities are connected and what depends on what, which suits impact analysis, root cause, compliance and planning. StackMAX DATA AI LAKE scores keyword, vector and graph results together and returns them with citations.
Does our data leave the environment? Are public AI services used?
No, by default. Data, indexes, graphs, context and memory stay inside the deployment boundary, encrypted with keys you control. The model gateway uses approved public providers only where an explicit policy allows it.
Can it run in an air-gapped environment?
Yes. Every layer runs inside the boundary with local package, image, chart and model repositories. Updates arrive only as signed, scanned and approved offline bundles.
Can a service provider offer it as a multi-tenant managed service?
Yes. Tenants, projects and workspaces are isolated at the storage, queue, namespace, index, graph, context and agent levels, with per-tenant keys, quotas and audit trails, and usage events for every service.
Who operates it?
You choose: customer-operated with vendor support, co-managed, fully managed, or a dedicated SRE Pod. Every deployment includes a validated architecture, a documented release and upgrade policy, SBOMs for every release and 24x7 support options.
Build what the world depends on.
Planning a cloud for government, financial services, pharmaceuticals, healthcare, energy, transport, industry, service-provider operations or private AI? Define the architecture, security boundary and operating model your environment requires.
Or email us directly at webinquiries@smxiq.com
Apache, Hadoop, HDFS, Ozone, YARN, Hive, HBase, Phoenix, Tez, Spark, Flink, Kafka, NiFi, Oozie, Airflow, Livy, Zeppelin, Superset, Ranger, Knox, Atlas, Ambari, Iceberg, ZooKeeper, Trino, Ceph, OpenSearch, Milvus, JanusGraph, Temporal, Kerberos, Cloudera, Databricks, Snowflake, Microsoft, Azure, Amazon, AWS, Google Cloud, IBM, VMware, Red Hat, OpenStack, Rancher, Zabbix, Ansible, Open vSwitch, Hortonworks, Talend, Elasticsearch, Kibana, Cassandra, OCTO and other product and company names are trademarks of their respective owners and are named for identification and comparison only. No affiliation or endorsement is implied.