Generative AI: What It Is, Why It Matters, and How Databricks Lakehouse Enables its Enterprise-Wide Use Case

10 min readJul 24, 2023

There is no doubt Generative AI has the potential to transform modern enterprises by enabling new business opportunities, efficiencies, and customer experiences. According to recent CEO surveys, almost 80% of CEOs believe AI is likely to significantly enhance business efficiencies in their organization[1]. However, Generative AI also presents challenges for technology leaders who want to use it for their enterprises. They need to find the best use cases, ensure the quality and security are not compromised, and integrate Generative AI with their existing data and systems.

In this blog post, we will show how technology leaders can overcome these challenges and harness the power of Generative AI for their enterprises. We will introduce a framework for implementing Generative AI use cases with Databricks Lakehouse Platform, a unified platform for data and AI. We will also share some of the enterprise-wide use cases of Generative AI and how the Databricks Lakehouse Tech Stack supports them.

How to Harness the Power of Generative AI for Your Enterprise Generative AI

Generative AI is one of the most exciting and disruptive technologies of our time. However, Generative AI also poses significant challenges and opportunities for technology leaders who want to leverage it for their enterprises. Below we will explore some of the key trends and insights technology leaders should know as they adopt Generative AI within their enterprises.

Don’t wait. Scale now. Generative AI is ready for prime time and real-world examples include enhanced customer service, automated fraud alerts, optimized workforce scheduling, and intelligent automation.
Generative AI will transform everyone’s jobs. Nearly every knowledge worker can benefit from collaborating with Generative AI.[2] All knowledge work will change regardless of your workforce’s technical background.
Responsible AI is both a necessity and an advantage. Adopting responsible AI principles for Generative AI is not only about risk management. It will also give you a competitive edge.
A new Build vs Buy debate has emerged with the ability to fine-tune foundational models or build foundational models from scratch with your enterprise’s proprietary information.
Culture may determine who wins and loses. Your organization’s cultural principles will determine your success with generative AI.
Data Governance and Privacy is everything. As AI itself gets commoditized in the long run, your proprietary data is what will become your strategic asset.
Efficiency Matters, Effectiveness Transforms. Generative AI is already proving to be useful for solving efficiency problems (i.e., generating a lot of content quickly). However, the arc of Generative AI’s enterprise evolution is trending toward effectiveness as businesses look for competitive advantages by leveraging AI to transform how they operate [4].
Enterprise Generative AI requires more advanced capabilities in the form of either specialized models trained on real-world data relevant to the problem or fine-tuned models, knowledge bases containing insights, problem-centric prompts, and other techniques to produce effective outputs at scale.[4]

What are some challenges of implementing Generative AI in enterprises?

Some of the challenges of implementing Generative AI in enterprises are:

Data and IP protection — Enterprises need to safeguard their sensitive data and IP from being exposed or exploited by Generative AI tools. These tools could leak, access, or take advantage of your secrets without your consent or knowledge.[5]
Customization requirements. Enterprises need to fine-tune base foundation models and engineer prompts for their proprietary data and unique needs. Generative AI models need to be tailored and optimized for specific domains, tasks, and audiences, and aligned with the enterprise’s goals, values, and brand.
Observability and reliability. Broadly trained base models are black boxes and often produce false, harmful, or unsafe results that require continuous evaluation and monitoring by experts. Generative AI models need to be transparent, explainable, and accountable, and provide accurate, relevant, and consistent outputs.
Security and safety risks. Generative AI models can pose threats to data privacy, security, and integrity, as well as to human dignity, rights, and well-being. Generative AI models need to be secured from unauthorized access, misuse, or manipulation and comply with ethical and legal standards and regulations.
Data management challenges. Generative AI models require large amounts of high-quality, diverse, and representative data to train and fine-tune. Data management challenges include collecting, storing, processing, labeling, cleaning, governing, and ensuring the quality of data.
Cost and resource allocation. Generative AI models are computationally intensive and expensive to train and run. Cost and resource allocation challenges include choosing the right infrastructure, platform, tools, and services, as well as optimizing the model size, performance, and efficiency.
Talent acquisition and retention. Generative AI models require skilled and experienced professionals to design, develop, deploy, and maintain them. Talent acquisition and retention challenges include finding, hiring, training, and retaining qualified data scientists, engineers, analysts, and domain experts.

How to Build a Tech Stack for Generative AI

To integrate and manage Generative AI models in your enterprise technology architecture, you need to orchestrate how they interact with each other and with existing AI and machine learning (ML) models, applications, and data sources. To make the best use of Generative AI, your tech stack should have these five key elements:

Context management and caching to provide models with the right data from enterprise sources. This helps the model understand the situation and generate better outputs. Caching stores the answers to common questions for faster and cheaper responses.
Policy management to regulate who can access enterprise data. This prevents unauthorized access to sensitive data.
Model hub, which stores trained and approved models. It also keeps track of model checkpoints, weights, and parameters.
Prompt library, which has optimized commands for the Generative AI models, and versions of prompts as models change.
MLOps platform, with enhanced MLOps features, to handle the complexity of Generative AI models.

How does Databricks Lakehouse Platform fit into this Tech Stack?

Databricks provides a unified data and AI platform that combines the best elements of data lakes and data warehouses to enable efficient and secure AI and BI directly on vast amounts of data stored in data lakes. It is built on open source and open standards, such as Delta Lake, Apache Spark, MLflow, and TensorFlow.

Databricks Lakehouse Platform can help enterprises get value from Generative AI by providing a data-centric approach to AI, with built-in capabilities for the entire AI lifecycle and underlying monitoring and governance. Below is an architectural representation of the Databricks capabilities and features which help with each of the tech stack components mentioned earlier.

Databricks Lakehouse Tech Stack for Generative AI

Let's take a deeper look at each of the capabilities in the architecture above.

Use existing models or train your own model using your data

Vector Search for indexing: Vector Embeddings let organizations use Generative AI and LLMs for many use cases, such as customer support bots or search and recommendation experiences. Databricks Vector Search helps teams index their data as embedding vectors and do fast vector similarity searches in real time. With Databricks Vector Search, developers can use embedding search to improve their Generative AI responses. It automatically creates and updates vector embeddings from files in Unity Catalog. It also integrates with Databricks Model Serving. Developers can also apply query filters to deliver better results.
Curated models, optimized for high performance and cost: Databricks has made it easy to start with Generative AI for various use cases by publishing a curated list of open-source models in Databricks Marketplace. These include MPT-7B and Falcon-7B models for instruction-following and summarization, and Stable Diffusion for image generation. Lakehouse AI features like Databricks Model Serving are optimized for these models to ensure fast and efficient performance. You can save time by using models curated by Databricks experts for common use cases, instead of researching the best open-source Generative AI models yourself.
AutoML support for LLMs: Databricks AutoML simplifies fine-tuning LLMs with a low-code approach. AutoML lets non-technical users fine-tune models easily with your data, and helps technical users do the same faster. Customers can fine-tune LLMs securely with their own enterprise data and keep the resulting model generated by AutoML, without sharing data with a third party. The model can also be easily shared, governed, served, and monitored within an organization, thanks to MLflow, Unity Catalog, and Model Serving integrations.

Monitor, evaluate, and log your model and prompt performance

Lakehouse Monitoring: This lets you monitor both your data and AI quality at the same time. The service tracks profile and drift metrics on your assets, alerts you when something goes wrong, creates quality dashboards to share with your organization, and helps you find the root cause by linking data-quality alerts across the lineage graph. Built on Unity Catalog, Lakehouse Monitoring gives you deep insights into your data and AI assets to ensure they are high quality, accurate, and reliable.
Inference Tables: This lets you log the requests and responses to serving endpoints to Delta tables in your Unity Catalog. This lets you monitor your model quality in near real-time, and use the table to find data points that need to be relabeled for fine-tuning your embeddings or other LLMs.
MLflow for LLMOps: Improved the MLflow evaluation API to track LLM parameters and models. Built-in prompt engineering tools to choose the best prompt template. MLflow records each prompt template evaluated for later use.
MLflow Prompt Tools: Users can compare different models based on prompts, using no-code visual tools. The prompts are tracked in MLflow.

Securely serve models, features, and functions in real-time

Model Serving, GPU-powered and optimized for LLMs: Databricks Lakehouse AI offers GPU model serving which is optimized for the top open-source LLMs. Our optimizations deliver best-in-class performance, making LLMs run much faster on Databricks. These performance improvements help teams reduce costs and scale endpoints to handle traffic.
Feature & Function Serving: Organizations can avoid online and offline skew by serving both features and functions. Feature and Function Serving does fast, on-demand computations behind a REST API endpoint to serve machine learning models and power LLM applications. When used with Databricks Model Serving, features are automatically joined with the inference request–making data pipelines simpler.
AI Functions: Data analysts and data engineers can use LLMs and other machine learning models in an interactive SQL query or SQL/Spark ETL pipeline. With AI Functions, an analyst can do sentiment analysis or summarize transcripts–if they have the permissions in the Unity Catalog and AI Gateway.
MLflow AI Gateway: With MLflow AI Gateway, organizations can manage credentials for SaaS models or model APIs centrally and provide access-controlled routes for querying. These routes can be integrated into different teams’ workflows or projects. Developers can also swap out the backend model easily to optimize cost and quality and switch between LLM providers. MLflow AI Gateway also supports prediction caching to track repeated prompts and rate limiting to control costs.

Manage Data & Governance

Unified Data & AI Governance: Unity Catalog provides complete governance and lineage tracking of both data and AI assets in one unified experience. This means the Model Registry and Feature Store are now part of the Unity Catalog, letting teams share assets across workspaces and manage their data and AI together.
MLflow AI Gateway: As organizations use OpenAI and other LLM providers, they face challenges with managing rate limits and credentials, controlling costs, and tracking what data is sent externally. The MLflow AI Gateway, part of MLflow 2.5, is a workspace-level API gateway that lets organizations create and share routes, which can be set with different rate limits, caching, cost attribution, etc. to manage costs and usage.
Databricks CLI for MLOps: This improved version of the Databricks CLI helps data teams set up projects with infra-as-code and speed up production with integrated CI/CD tools. Organizations can use “bundles” to automate AI lifecycle components with Databricks Workflows.

MosaicML’s Acquisition By Databricks: The acquisition of MosaicML by Databricks will help enterprise customers in several ways. Here are some of the benefits:

Enterprise customers will be able to build, own, and secure their own Generative AI models with their own data, without relying on external SaaS models or services. This will give them more control, privacy, and security over their data and models, and enable them to create custom solutions.
Enterprise customers will be able to leverage MosaicML’s state-of-the-art large language models (LLMs), such as MPT-7B and MPT-30B, as a foundation for their Generative AI applications. They will also be able to fine-tune these models with their own data in a cost-effective way, using MosaicML’s automatic optimization of model training which provides 2x-7x faster training.
Enterprise customers will be able to access Databricks’ unified data and AI platform, which enables efficient and secure AI and BI directly on vast amounts of data stored in data lakes. They will also be able to use Databricks’ new features and capabilities for Generative AI, such as vector search, lakehouse monitoring, GPU-powered model serving optimized for LLMs, MLflow 2.5 with LLM capabilities such as AI Gateway and Prompt Tools, and more.

Generative AI Use-Cases mapped to Databricks Lakehouse Capabilities

Generative AI is transforming the way businesses operate, from automating tasks to creating new insights. But how can you leverage Generative AI for your specific use case? And what are the technical challenges and solutions involved? For the purpose of this blog, we will categorize the many possible use cases with Generative AI into four categories:

Changing the way knowledge workers operate.
Helping relationship managers keep up with the pace of public information and data.
Enhancing customer experience by freeing up customer support representatives’ time for higher-value activities.
Accelerating the pace at which R&D is performed.

In the table below, we share some examples of use cases in each category and how Generative AI can be applied to different domains, what technical skills and solutions are needed, and how the Databricks Lakehouse platform can help you achieve your goals.

References/Additional Reading

This is a personal blog. Any views or opinions represented in this blog are personal and belong solely to the blog owner and do not represent those of people, institutions or organizations that the owner may or may not be associated with in professional or personal capacity, unless explicitly stated.