Encrypt at Bronze, query at Gold, train in MLflow, serve at the endpoint, share across workspaces, all under a single customer-managed key.
Enigmata Cipher for Databricks installs from the Databricks Marketplace as a Unity Catalog-registered library. It exposes Enigmata Cipher’s three pillars as PySpark UDFs and SQL functions usable in any notebook, Workflow, Delta Live Tables pipeline, MLflow training run, or Model Serving endpoint. Bind your encryption context to a customer-managed key and the same context flows from raw data ingestion to served model, without ever decrypting in compute. Your plaintext never needs to leave your premises. Run Enigmata Cipher’s signed confidential-VM image on-prem to encrypt your data, and upload only ciphertext to a Unity Catalog Volume. Your Databricks workspace sees ciphertext, never plaintext.
Accuracy parity for MLflow runs trained on encrypted features vs. plaintext baselines on the published benchmark.
External network calls from cluster nodes; all key material flows through the existing Databricks-managed CMK path.
One encryption context spans the entire lakehouse: same key, same tokens, same UDF API at every stage.
Encrypted columns and key contexts register as governed assets; lineage from raw data to served model flows automatically.
What this unlocks for you
One context, the whole lakehouse
Encrypt once at the Bronze layer in your DLT pipeline; Silver, Gold, MLflow training, Vector Search indexing, and Model Serving all consume the same encrypted columns under the same key context. No "encrypt for training, decrypt for serving" gymnastics.
Delta Sharing without leaks
Share encrypted Delta tables with partner workspaces; their workspace can run analytics, join with their own data, and train models on the encrypted columns without ever holding your CMK. Cross-workspace clean rooms that don’t require a Clean Room.
Vector Search on encrypted embeddings
Encrypt your embedding columns with the Enigmata Cipher SDK before indexing; Vector Search ranks results the same way it would on plaintext, with the original embeddings never written to the index.
MLflow + Model Serving parity
The MLflow run records its encryption context; the served endpoint reuses it; inputs at inference time match the encrypted shape the model trained on. No "the model was trained on different data than it sees in production" silent drift.
Three pillars, preserved
Text: equality preserved across the lakehouse
Delta `MERGE` keys, `JOIN` predicates, `GROUP BY` clauses, and Spark DataFrame `.filter(col("x") == "y")` all work on encrypted columns. Identical plaintext always produces the same encrypted token.
Numeric: order preserved for time-series and window queries
`ORDER BY`, range filters, percentile aggregates, and Spark window functions work on encrypted numbers. Photon’s vectorized predicate pushdown still applies.
Vector: cosine ranking preserved in Databricks Vector Search
Encrypted embeddings index into Vector Search; the Enigmata Cipher SDK's similarity function returns the same ordering Databricks’ built-in cosine would on plaintext. RAG over Bricksturbo-served LLMs works on encrypted documents.
How customers install it
Subscribe to Enigmata Cipher for the Lakehouse in the workspace you train and serve from. The Marketplace install registers the library and a cluster init script under Unity Catalog.
Run the bundled setup notebook to register the library’s service principal against your existing workspace CMK. No new KMS plane to manage.
Define an encryption context with the Enigmata Cipher SDK, naming it and binding it to your CMK. On-prem: the signed Enigmata Cipher confidential-VM image attests to your KMS, derives the seed, and writes the wrapped manifest to a Unity Catalog Volume so clusters can read it. In-workspace alternative: the library reads the wrapped manifest from the same Volume and unwraps it via your CMK on each cluster cold-start.
On-prem: encrypt your source data with the confidential-VM image and write ciphertext directly to your Bronze Delta tables. Every downstream notebook, Workflow, DLT pipeline, MLflow run, and Model Serving endpoint reads ciphertext from there. In-workspace alternative: call the Enigmata Cipher SDK's encrypt UDF on a column in a notebook, a Workflow Python task, a DLT pipeline expectation, an `mlflow.pyfunc` model, or a Model Serving custom container. Same API everywhere.
Where it fits
Best fit: enterprise teams running the Databricks Lakehouse end-to-end (Bronze → MLflow → Model Serving) where the same encryption context needs to follow the data through every stage, with optional Delta Sharing to partner workspaces.
Financial Services
Lakehouse for fraud and AML; encrypted customer identifiers join across Bronze → Silver → Gold and feed MLflow-trained scoring models served on Model Serving endpoints.
Healthcare
DLT pipelines on ePHI; clinical embeddings in Databricks Vector Search for RAG over patient narratives, all encrypted in the protected representation; consent-revocation triggers a Delta `MERGE` against encrypted patient IDs.
Telecom + Energy
Subscriber and grid data in regional Databricks workspaces, sovereignty-bound; encrypted Delta Sharing replaces cross-region data movement.
Retail + CPG
Loyalty data shared with brand partners via Delta Sharing; partner trains attribution models on encrypted columns in their workspace without ever holding your CMK.
How it compares
The honest framing of every other option.
| Approach | What you get |
|---|---|
| Unity Catalog column masking | Strong governance at query time; the data is still plaintext inside compute clusters during training and Model Serving inference. |
| Databricks CMK alone | Encrypts workspace storage at rest; data is decrypted inside compute clusters and visible in plaintext to running jobs and notebook outputs. |
| Synthetic feature generation | Models drift from production behavior; inference still needs the real values and you’ve added a maintenance burden for the synthesis pipeline. |
| PySpark `crypto.aes_encrypt()` UDF | Opaque ciphertext; loses every property except equality on identical inputs in deterministic mode, and that mode is discouraged in the Spark docs. |
| Enigmata Cipher | Lakehouse-native, single context across Delta + MLflow + Model Serving + Vector Search + Delta Sharing, CMK-keyed via the existing workspace key plane. |
Key capabilities
- PySpark UDFs and Unity Catalog SQL functions from the Enigmata Cipher SDK that encrypt text, numeric, and vector columns, plus a keyless cosine-similarity compare over encrypted vectors.
- Customer-managed CMK via the workspace’s existing Databricks-managed KMS path. No parallel key plane.
- Native Marketplace library: single subscribe, single workspace install, available across every cluster, job, and Model Serving endpoint.
- Three pillars preserved across the lakehouse: equality in Delta `MERGE`, order in time-series window queries, cosine in Databricks Vector Search.
- Unity Catalog integration: encrypted columns + key contexts register as governed assets with lineage flowing through Bronze → Silver → Gold automatically.
- MLflow integration: the encryption context is registered as a model artifact, so Model Serving endpoints load the same context that trained the model.
Technical detail
How the library flows one encryption context across the lakehouse.
Enigmata Cipher for Databricks installs as a Unity Catalog-registered library plus a cluster init script via Databricks Marketplace. The library exposes PySpark UDFs registered as Unity Catalog functions so they’re callable from PySpark, SparkSQL, dbt-databricks, Workflows, DLT, and any downstream consumer with USE FUNCTION privilege. Where the encryption step physically runs is the customer’s choice: on-prem, the signed Enigmata Cipher confidential-VM image reads plaintext inside your facility and uploads ciphertext to a Unity Catalog Volume. Plaintext never enters Databricks. As an in-workspace alternative, the library reads its seed from a Unity Catalog Volume entry wrapped by the workspace CMK and unwraps it once per cluster startup inside the Databricks runtime; both paths produce identical ciphertext. Encrypted text and numeric values return as STRING; encrypted vectors return as a STRUCT matching the same shape every Enigmata Cipher integration emits. MLflow integration registers the encryption context as a model artifact so the same context loads at Model Serving time, ensuring the served model never sees a different schema than it trained on.
- Marketplace library distribution: single subscribe, single workspace install, available across every cluster and Model Serving endpoint.
- PySpark UDFs + Unity Catalog functions: callable from PySpark, SparkSQL, dbt-databricks, Workflows, DLT.
- CMK integration: leverages the existing Databricks-managed CMK path; no parallel key-management plane.
- Encrypted vector cells round-trip as `STRUCT<bytes: BINARY, bit_length: SHORT, context_fingerprint: BINARY, recovery_id: STRING>`, with `recovery_id` always null on the no-LUT path.
- MLflow + Model Serving integration: the encryption context travels with the model from training run to served endpoint, recorded as an MLflow run tag and queryable through the standard MLflow run API.
What it needs
A Databricks workspace with Unity Catalog enabled, CMK configured for the workspace, and Databricks Runtime ML or Photon. The Marketplace install handles dependency wiring via a cluster init script.
Who holds the key
Run encryption on-prem and your KMS releases the seed only to an attested, unmodified Enigmata Cipher confidential-VM image. Plaintext never leaves your facility and the seed never reaches Databricks. Run encryption in your workspace as the alternative and your Databricks Customer-Managed Key (CMK) never leaves your KMS (AWS KMS, Azure Key Vault, or GCP Cloud KMS, whichever your workspace already uses): the library calls the CMK once per encryption context to derive a seed, AEAD-wraps it, and stores the wrapped manifest as a Unity Catalog Volume entry, versioned and governed alongside your other lakehouse assets. The seed is in memory only on Databricks compute; we never see it.
How it’s billed
Databricks Marketplace billing. Library subscription: monthly per workspace. UDF invocations consume standard DBUs on whichever cluster the UDF runs (no per-row Enigmata Cipher meter; the marketplace fee is workspace-based). CMK, Unity Catalog, and Model Serving charges remain on the customer’s Databricks bill.
The offer
Prove it in 60 days.
Prove it in 60 days. Pick the Databricks Workflow you can’t stand up because the data can’t sit in a cluster in plaintext. We’ll wire Enigmata Cipher into your Bronze ingestion, follow it through to an MLflow-trained model on Model Serving, and ship the parity benchmark + Unity Catalog lineage proof that legal needs to sign off.
- Encryption-parity benchmarkMLflow run comparison: plaintext baseline vs. encrypted features on the customer’s target metric, captured as a comparison view in the MLflow UI.
- CMK custody attestationDiagram of the customer’s KMS → workspace CMK → encryption-seed flow, showing the customer’s KMS key never leaves their KMS.
- Unity Catalog lineage reportAuto-generated lineage from Bronze through Model Serving showing the encryption context as a first-class lineage edge alongside table and column lineage.
- Delta Sharing parity reportMulti-workspace share scenario showing encrypted-column queries succeeding in a recipient workspace with no CMK access.