A hybrid data platform sounds modern. Without decision logic, it soon sounds expensive.

You have budget approval. The team is ready. The roadmap for the hybrid data platform is signed. Databricks has been chosen as the central building block for SAP and non-SAP data. Then comes the moment when the first technical question is raised out loud: "Do we migrate the data – or federate it?"

And that's exactly where the real problem begins. Because without clear decision logic, a modern platform quickly turns into a costly patchwork. In this article, we'll show you why the question of migrating from SAP BW/4HANA to Databricks doesn't start with a blanket answer – it starts with the right decision architecture.

Why data platforms become expensive without decision logic

A hybrid business analytics platform with Databricks promises a lot: integration of SAP data via SAP ODP and Databricks, connections to non-SAP sources, central governance with Unity Catalog, scalable AI and ML capabilities. But in practice, many projects fail not because of the technology – but because of missing guardrails for decision-making.

Typical symptoms:

• Data redundancy: The same SAP tables are copied multiple times because no one defined where which data should live.
• Exploding costs: Storage, compute, and network traffic grow uncontrollably because every business unit builds its own pipelines.
• Governance gaps: Permissions, lineage, and audit trails are inconsistent because no clear responsibility was defined.
• Long time-to-value: Use cases get stuck in pilot phases because the architecture doesn't scale and every new project starts from scratch.

The root cause? Technical feasibility is often confused with strategic clarity. A platform may technically be capable of everything – but without decision logic, there's no guidance on when which architecture makes sense.

The difference between a platform and a platform strategy

A platform is a tool. A platform strategy defines how that tool is used in the context of your data landscape, your governance requirements, and your business goals.

Specifically, this means:

• When is data physically moved (lift-and-shift)?
• When does data stay at the source and get federated (e.g., via foreign catalogs)?
• Which data belongs in which layer (bronze, silver, gold)?
• Who is responsible for data quality, freshness, and governance?
• How are costs made transparent and attributed to teams?

Without answers to these questions, every platform – no matter how modern – becomes a black box.

Decision check: Lift-and-shift vs. federation

The central architectural decision when modernizing the SAP data platform is: should SAP data be physically copied to Databricks – or should it stay in SAP and be made accessible via federation?

Both approaches have their merits. What matters is making the right choice for the specific use case.

What does lift-and-shift mean?

With lift-and-shift, SAP data (e.g., from BW/4HANA, S/4HANA, or via SAP ODP into Databricks) is extracted, transformed, and persisted in Databricks as Delta tables. The data is physically moved and permanently resides in the lakehouse architecture.

Advantages:

• Full control over data modeling, transformations, and performance
• Optimal integration with other data sources (non-SAP, cloud, streaming)
• Independence from SAP systems for analytics workloads
• Scalable foundation for AI, ML, and advanced analytics

Challenges:

• Initial migration effort (mapping, pipelines, testing)
• Ongoing synchronization and data freshness
• Higher storage and compute costs for large data volumes
• Governance must be ensured across system boundaries

What does federation mean?

With federation, the data stays in SAP. Databricks accesses the source systems directly via foreign catalogs or connectors. Data is read on demand – not permanently copied.

Advantages:

• Minimal data movement – the single source of truth stays in SAP
• Lower storage costs
• Faster implementation for simple reporting use cases
• No synchronization issues

Challenges:

• Dependency on SAP system performance and availability
• Limited transformation and modeling options
• Latency for complex queries or large data volumes
• More difficult integration with non-SAP data and AI workloads

When to use which approach

The decision depends on several factors:

• Data volume: For very large SAP data volumes, federation can save costs – provided queries are infrequent.
• Freshness requirements: Real-time requirements favor federation or streaming pipelines; batch analyses favor lift-and-shift.
• Transformation needs: Complex business logic, enrichment, and joins with non-SAP data require physical data storage.
• Governance: If data is meant to be reused across different domains, lift-and-shift with clear ownership rules is often the better choice.
• Cost: Lift-and-shift incurs higher storage costs; federation puts load on SAP systems and can cause licensing and performance issues.
• Performance: Analytics workloads on Delta Lake are generally significantly faster than queries against SAP source systems.

In practice, most companies opt for a hybrid architecture: critical, frequently used data is migrated; rarely used or highly dynamic data is federated.

Export pipelines in practice: From SAP to Delta

If you decide on lift-and-shift, the question becomes: how does the data get from SAP to Databricks cleanly and efficiently?

Proven integration paths

1. SAP ODP → cloud storage → Delta Lake

SAP Operational Data Provisioning (ODP) is the modern standard for data extraction from SAP. ODP supports extractors, CDS views, and DataSources, and delivers data incrementally or as a full load.

Typical flow:

• Data is extracted from SAP via ODP
• Export to cloud storage (Azure Data Lake, AWS S3, Google Cloud Storage)
• Databricks reads the data via Auto Loader or structured pipelines
• Transformation and persistence as Delta tables

Advantage: High flexibility, scalable, well-documented, SAP-native interface.

2. SAP SLT → cloud storage → Delta Lake

SAP Landscape Transformation Replication Server (SLT) is suited for real-time or near-real-time replication of tables.

Advantage: Very low latency, good for operational use cases.

Challenge: Higher operational effort, licensing costs, greater load on source systems.

3. SAP Datasphere / Business Data Cloud → Databricks

SAP Datasphere offers native integration with Databricks via Delta Sharing. Data can be provided as curated data products and used directly in Databricks.

Advantage: Semantic enrichment, governance on the SAP side, simple provisioning.

Challenge: Additional SAP licensing costs, dependency on the SAP roadmap.

Avoid tool sprawl

Avoid the mistake of introducing a new tool for every use case. Define 2–3 standard integration paths and use them consistently. This reduces complexity, operational effort, and training needs.

Mapping tips: What goes wrong in SAP migrations

Technical integration is one side of the coin. The other is correctly mapping SAP data types, formats, and semantics to Delta Lake. Typical pitfalls lurk here:

1. Decimal formats and precision

SAP works with specific decimal formats (e.g., CURR, QUAN with currency and unit-of-measure fields). This information is often lost – or misinterpreted – when exporting to Databricks.

Solution: Define clear mapping rules for decimal places, scale, and precision. Use metadata from SAP (e.g., DDIC) and document rounding rules.

2. Date, time, and timezone

SAP frequently stores date fields as YYYYMMDD (string) or DATS. Timestamps are often in UTC – but not always. During migration to Databricks, errors arise when timezone information is missing or inconsistent.

Solution: Standardize date/time formats to ISO 8601 or timestamp types. Clarify timezone handling and document conventions in the data catalog.

3. Leading zeros

SAP key fields (e.g., customer or material numbers) often contain leading zeros. When exported as numeric fields, these are lost – with serious consequences for joins and lookups.

Solution: Export key fields as string types. Define clear data types in Unity Catalog and use constraints to ensure consistency.

4. Character sets and special characters

SAP systems often use specific character sets (e.g., SAP code page 1100). When exporting to UTF-8, umlauts, special characters, or Asian scripts can be transferred incorrectly.

Solution: Test character set conversions early. Use SAP-native export functions that support UTF-8.

5. NULL values vs. empty strings

In SAP, NULL values and empty strings are often handled differently. In Databricks, this can lead to unexpected results in aggregations and filters.

Solution: Define a consistent NULL-handling strategy and document it in your data governance framework.

Unity Catalog in the target picture: Governance from day one

A successful migration from SAP BW/4HANA to Databricks doesn't end with moving the data. It begins with a clear governance architecture – and this is where Unity Catalog plays a central role.

Why Unity Catalog?

Unity Catalog is the central governance layer in Databricks. It enables:

• Cataloging: Structuring data into catalogs, schemas, and tables
• Permissions: Fine-grained, role-based access control
• Lineage: Automatic tracking of data flows and transformations
• Auditing: Logging of all access and changes
• Metadata management: Central management of descriptions, tags, and ownership

Catalog structure for SAP migrations

Define a clear catalog structure early on. Example:

• Catalog "SAP": All SAP source data (bronze layer)
• Catalog "Curated": Transformed, enriched data (silver layer)
• Catalog "Business": Business-curated data products (gold layer)
• Catalog "Sandbox": Experimental workspaces for data science

Within each catalog, you can structure schemas by domain (e.g., finance, supply chain, sales) or by source system (e.g., SAP ERP, SAP BW).

Managed vs. external tables

Deliberately decide whether tables should be created as managed (Databricks manages data and metadata) or external (data resides in your own storage).

Managed tables offer simpler governance and lifecycle management – but are less flexible in multi-cloud scenarios. External tables allow more control over storage but require additional governance mechanisms.

Roles and responsibilities

Define clear ownership rules:

• Data owners: The business unit responsible for data quality and business semantics
• Data stewards: Technical responsibility for pipelines, transformations, and metadata
• Data consumers: Teams and users with read access

Use tags in Unity Catalog to mark ownership, sensitivity (e.g., GDPR-relevant data), and lifecycles (e.g., archiving after 7 years).

Migration factory roadmap: From assessment to cutover

A structured SAP data platform modernization follows a clear phase model. Here's a proven roadmap:

Phase 1: Assessment

Goal: Create transparency about the current state and migration scope

Activities:

• Inventory all SAP data sources (BW, S/4HANA, DataSources, CDS views)
• Analyze data volume, freshness requirements, and dependencies
• Evaluate use cases (reporting, analytics, AI/ML)
• Decide lift-and-shift vs. federation per data source
• Plan costs and resources

Outcome: Prioritized migration list, architecture sketch, business case

Phase 2: Pilot

Goal: Prove feasibility and generate learnings

Activities:

• Migrate 2–3 representative data sources
• Build the first pipelines (e.g., SAP ODP with Databricks)
• Set up Unity Catalog (catalogs, schemas, permissions)
• Test mapping rules (decimals, dates, zeros)
• Validate performance, cost, and data quality

Outcome: Working proof-of-concept, validated architecture, documented lessons learned

Phase 3: Migration factory

Goal: Scale the migration across all data sources

Activities:

• Industrialize pipelines (templates, automation)
• Parallelize migration in waves (by priority)
• Continuous quality assurance and testing
• Build monitoring and alerting
• Train teams (data engineers, analysts, business users)

Outcome: Fully migrated data landscape, production pipelines, established governance

Phase 4: Cutover and operations

Goal: Transition to steady-state operations and decommission legacy systems

Activities:

• Final cutover (switch from old to new system)
• Deactivate or archive legacy systems
• Establish support and operations processes
• Continuous optimization (performance, cost, governance)
• Expand with new use cases (AI, GenAI, self-service)

Outcome: Stable, scalable platform in production

Cost traps and clean cost allocation

A hybrid data platform can quickly become expensive – if costs aren't made transparent and controlled.

Typical cost drivers

• Storage: Data volume grows uncontrollably because no one deletes old or redundant data
• Compute: Inefficient queries, missing cluster policies, and uncontrolled workloads drive up compute costs
• Network traffic: Data movement between regions or cloud providers causes high transfer costs
• Licenses: SAP licenses (e.g., for SLT, Datasphere) and Databricks licenses add up

Cost allocation with tags

Use tags in Unity Catalog and cloud storage to transparently attribute costs:

• Team tags: Which team is the owner?
• Project tags: Which project or use case is the table assigned to?
• Lifecycle tags: Is the table in production, in testing, or deprecated?
• Sensitivity tags: GDPR-relevant, confidential, public?

With these tags, you can evaluate costs per team, project, or domain – and allocate budgets fairly.

Automated lifecycle policies

Define rules for data lifecycles:

• Bronze data: archive after 90 days
• Silver data: archive after 2 years
• Gold data: long-term retention, but regular quality checks

Automate archiving and deletion via Databricks jobs or cloud-native lifecycle policies.

What has been your experience with hybrid data platforms? What challenges do you see when integrating SAP and non-SAP data? We look forward to your feedback and the exchange of ideas.

Join our free webinar "3 common migration mistakes from SAP to Databricks – and how to avoid them" to learn how to implement your data platform modernization in a structured, cost-efficient way. Register now and secure your spot!