Data integration without a data graveyard: CDC, Event Streaming and ETL compared for ERP/CRM/warehouse

Anyone who connects ERP, CRM and warehouse management usually wants two things at once: processes should run end-to-end (e.g. order → picking → shipping → invoice), and data should be available for analyses (e.g. delivery capability, contribution margins, return rates). In practice this quickly becomes a balancing act between “we need it in the reports today” and “we must not destabilize the productive ERP”. This is exactly where it is decided whether data integration without a data graveyard succeeds or whether an opaque mix of CSV exports, nightly jobs, shadow tables and unexplained data copies accumulates over years.

This article compares three central approaches: ETL (Extract, Transform, Load), CDC (Change Data Capture, i.e. detecting and transferring data changes) and Event Streaming (events as a continuous data stream via a broker). The focus is not on programming details, but on architectural consequences, operational reality, data quality, security and rollout issues — as they actually occur in integration projects between enterprise systems.

Why integrations often become a data graveyard

A data graveyard rarely arises from ill intent. Typical causes are:

• Unclear system boundaries: ERP is the “source of truth” at one moment, then CRM is, and the warehouse has its own status logic. Without a defined data ownership (System of Record), conflicts are preprogrammed.
• Ad-hoc requirements: “We need a dashboard quickly” leads to direct access to the ERP; later additional queries, materialized views or copies are added. Every quick win shifts operational load and responsibilities.
• Missing contracts: interface contracts (which fields, which semantics, which versioning) are missing. Result: schema drift — fields change meaning or structure without downstream systems noticing in time.
• No operational concept: jobs run “somewhere”, credentials sit in scripts, there is no alerting on data gaps, and no one can answer whether a report is “complete”.

ETL, CDC and Event Streaming solve different parts of this problem. What matters is choosing the approach appropriate to process criticality, latency requirements and operational maturity – and operating the integration path as a product, not as a one-off project artifact.

Classifying terms clearly: ETL, CDC and Event Streaming

ETL stands for “Extract, Transform, Load”: data is taken from source systems, transformed (e.g. cleaned, aggregated, mapped) and loaded into a target system, often a data warehouse. Classically this happens in a batch-oriented way, e.g. nightly or hourly.

CDC (Change Data Capture) describes mechanisms that detect changes to data and transfer them as deltas: new/updated/deleted records. CDC can be implemented via timestamps, triggers or — operationally often the cleanest — via the database transaction logs. The goal is usually near real-time, without constantly performing full extracts.

Event Streaming means publishing events (e.g. “order released”, “goods receipt posted”) as a continuous stream over a Message Broker (e.g. Kafka-like systems or service-bus concepts). Consumers subscribe to events and process them at their own pace. Important: an event is not automatically “the whole truth” of the data, but often a state change with context.

Comparison along the questions that really matter in operation

Latency: How fast must data actually be?

For many ERP reports, „last night“ data is sufficient. For operational control in the warehouse, „5 minutes old“ can already be too late (e.g. with tight stock levels). The guidance is:

• ETL provides predictable update windows, but by design is not „real-time“.
• CDC is useful when you want to mirror data changes quickly into reporting or search systems without re-modeling the business logic.
• Event Streaming is appropriate when processes need to react promptly (e.g. generate shipping labels, update customer status, trigger notifications).

A common mistake is to demand „real-time“ everywhere. Real-time increases complexity in monitoring, error handling and data consistency. It makes sense to classify: Which data are operational (process-critical), which analytical (reporting-critical), which archival (audit/compliance)?

Consistency: What happens in partial failures?

In distributed integrations, partial failures are normal: network interruptions, timeouts, locks, maintenance windows. The key question is whether your approach handles these robustly.

• ETL typically runs in batches. If a run fails, the target dataset is often consistent „up to time X“ and stale thereafter. That is acceptable for reporting in many cases, provided it is transparent.
• CDC transmits deltas. If the process stalls, a backlog forms. That is manageable, but you must measure lag (delay) and alarm on thresholds.
• Event Streaming shifts failures to the consumers. For this you need idempotency (processing multiple times without side effects), retry strategies and a Dead-Letter-Queue (repository for messages that cannot be processed); otherwise errors remain „silent“ and surface only in the business domain.

Consistency is also a domain question: must „order + line items + reservations“ arrive as a package, or is eventual consistency (later reconciliation) sufficient? The higher the package dependency, the more you need transaction boundaries and clear ordering rules.

Load and risk for the ERP: what is stressed and how?

Many integration issues are in reality performance and locking problems in the source system. The ERP is an OLTP system (Online Transaction Processing): many small transactions, high write load, sensitive indexes.

• ETL often pulls large volumes. Without proper time windows, read replicas or targeted extract tables, ETL can throttle the ERP.
• CDC via logs is usually gentler because it uses the „already existing“ change stream. Trigger-based CDC, by contrast, can extend write paths and is a risk on heavily loaded tables.
• Event Streaming avoids direct read load when events originate from the application itself. If events are generated „from the database,“ you are close to CDC again — with similar trade-offs.

Rule of thumb: If the ERP is already tightly sized today, integration should not begin with additional full extracts. Often it pays to decouple first, e.g. via CDC into a separate reporting or integration schema, and only then perform transformations.

ETL in practice: good for reporting, risky as process glue

ETL is the entry point in many companies because it is conceptually tangible: „We extract data, prepare it, load it into the DWH.“ For classic BI requirements this remains sensible.

Strengths of ETL

• Predictability: Night runs or hourly runs are well controllable and fit maintenance windows.
• Transformation logic centralized: Cleansing, mapping, historization (e.g. Slowly Changing Dimensions) are established in the DWH context.
• Auditability: With run IDs, row counts and checksums you can trace what was loaded when.

Typical risks and „data graveyard“ patterns

• Proliferation of direct access: The more analyses are based directly on extracted tables, the more „unofficial data products“ emerge.
• Schema drift without early warning: When fields change in the ERP, this is often only noticed at the next run — or worse: not at all, because null values „slip through“.
• Batch windows become tight: Data volume grows, runtime increases, eventually ETL collides with backups, reorgs or nightly ERP job chains.

Concrete example: A warehouse requires a daily report „items without inventory but with open orders“. As an ETL report that’s okay. If this report, however, is used as the basis for operational scheduling, a 24-hour delay suddenly becomes business-critical. ETL then becomes process glue — and that is rarely stable.

CDC: The pragmatic path to deltas and near-real-time

CDC is often the „sweet spot“ when you want to bring data from ERP/CRM/warehouse in a timely manner into search systems, a Data Warehouse or integration databases, without rethinking every piece of domain logic as an event model.

CDC variants and their operational implications

• CDC via timestamps/high-watermark: You read „everything since the last timestamp“. That’s simple, but susceptible to subsequent corrections, time drift and missing delete events.
• Trigger-based CDC: Changes additionally write into change tables. That’s functionally straightforward, but increases write load and requires clean permissions as well as maintenance on schema changes.
• Log-based CDC: Changes are derived from the transaction log. That is often more performant and closer to the truth, but requires careful configuration, because log retention, backups and maintenance jobs suddenly become integration-relevant.

Important for admins: CDC is not „set and forget“. You must monitor lag, define resync procedures (e.g. rebuild individual tables) and determine how long change history is retained in the target.

What CDC does particularly well

• Relief from full extracts: After an initial snapshot, only deltas are processed.
• Clear separation OLTP vs. analytics: Reporting can run on a separate database or a warehouse, without burdening the ERP.

Practical example: A CRM needs to know daily whether a customer has open deliveries, without running complex queries constantly in the ERP. CDC mirrors relevant tables or views into an integration database; the CRM reads from there. Result: fewer load spikes in the ERP, and queries can be indexed deliberately.

Event Streaming: When processes need to react — and you accept ownership

Event streaming is especially worthwhile when you want not only to copy data but to orchestrate process reactions: status changes, notifications, follow-up tasks, integrations with partners. An event is a „thing that has happened“ — including a timestamp, identifiers and the minimally necessary context.

Strengths of Event Streaming

• Decoupling: Producer and consumer do not need to be available at the same time. That reduces fragility during maintenance windows.
• Scaling across consumers: Multiple systems can use the same event (e.g., CRM, shipping, BI) without the ERP having to deliver separately for each target.
• Transparency in the flow: With good monitoring you can see throughput, backlog and error rates per consumer.

Risks and typical false assumptions

• „We send events, then data quality will be fine“: Events also transport incorrect states if upstream validations are missing. Data quality remains a domain responsibility.
• Idempotency is forgotten: Duplicate events happen (retry, network, rebalancing). Consumers must tolerate duplicate processing, e.g. via unique event IDs and „already processed“ checks.
• Schema and version management: Event messages are interface contracts. Without versioning and a deprecation plan you get chaos — only faster.
• Ordering is not free: Many brokers provide ordering only within defined partitions/keys. It must be clear at the domain level which key (e.g., order ID) guarantees ordering.

Concrete scenario: An outbound goods posting is recorded in the warehouse. The ERP should invoice, the CRM should update the customer status, and the tracking portal should provide shipping information. Event streaming can decouple this cleanly. But if invoicing must strictly occur before the status change, you need either process coordination (e.g., saga/choreography) or clear rules about who the orchestrator is. Otherwise states can „flicker“.

Decision guide: Which approach fits which objective?

In integration projects the wrong foundational decision is expensive. A practical classification:

If your primary goal is reporting and analytics

• Starting point: ETL or ELT (load first, transform later in the target system) – with clear execution schedules.
• When data freshness requirements increase: CDC as the data feed into the warehouse, ETL/ELT for transformation and modeling.

If your goal is operational, near-real-time synchronization

• Starting point: CDC for table/object mirroring, plus lightweight services for validation and conflict resolution.
• When true reaction chains are required: Event Streaming, but only with defined ownership and operational responsibility per consumer.

If your goal is process coupling between ERP/CRM/warehouse

• Starting point: Event Streaming or message-based integration, supplemented with return channels (acknowledgements) and error paths.
• ETL here only for secondary flows (e.g. daily reconciliations, archive, BI), not as a trigger for operational actions.

Important: In reality it is rarely an „either-or.“ Many stable architectures combine: events for processes, CDC for data provisioning and ETL/ELT for reporting models.

Architectural consequences you should clarify early

Data ownership and Golden Record questions

Who is allowed to change what? A „Golden Record“ is the domain-authoritative data record for an object (customer, item, order). If multiple systems write, you need conflict rules: priorities, manual reconciliation or MDM approaches (Master Data Management). Without these rules, integration becomes a constant „Why are the data different?“ ticket.

Error handling as design, not as an afterthought

Whether ETL, CDC or Event Streaming: you need defined error classes. A three-part classification is recommended:

• Technical errors (timeout, network, temporary locks): automatic retry with backoff.
• Semantic errors (mandatory field missing, unknown status): into quarantine/dead-letter, with ticketing capability.
• Process conflicts (sequence violated, double booking): business-level resolution process, often with manual decision.

Without a quarantine mechanism you end up with „integration shows green, but individual cases are missing.“ That is the fastest route to the data graveyard, because nobody knows anymore which data state is „true.“

Monitoring, alerting and traceability

For IT management and operations concrete questions matter: How many records/events per hour? How large is the backlog? Which interface causes the most retries? ETL needs run monitoring (start/end, row counts), CDC needs lag metrics, Event Streaming needs consumer lag and dead-letter rates. This includes logs with correlation (e.g. order ID), so that support cases do not end in screenshots.

Security and compliance: data copies are a responsibility

Integration produces copies. Copies mean new attack surfaces and new retention questions. Typical items that arrive too late in projects:

• Least Privilege: ETL and CDC accounts should only read what is necessary. For event producers/consumers, service accounts with minimal rights are mandatory.
• Secrets handling: Passwords in scripts or task schedulers are a classic. Better: centralized secrets management or at least proper rotation and audit.
• GDPR and deletion: If data is deleted/blocked in the ERP, it must be clear what happens in the DWH/Data Lake/Stream. CDC must represent deletion events, ETL needs deletion or anonymization logic.

Rollout and migration: How to avoid Big-Bang integrations

Especially with evolved processes a gradual transition is more stable. A practical approach:

1. Inventory: Which data flows exist (incl. Excel, SFTP, direct DB accesses)? Which are process-critical?
2. Stable target state per domain: e.g. „Stock status comes from WMS, order status from ERP, customer communication from CRM“.
3. Parallel operation with reconciliation: CDC/ETL initially run in „shadow“ mode; results are compared against the current state (delta reports, spot checks).
4. Cutover with fallback: For operational integrations: switch to the event/CDC source, but with a clear fallback level (e.g., read-only queries or a temporary batch).
5. Clean-up: Disable old jobs, revoke accesses, establish documentation and ownership. Without this step the data graveyard remains, only with new decoration.

Expectation management is important: an integration is never „finished.“ New fields, new processes, new locations — all of that affects data flows. Successful teams therefore define a maintenance mode: versioning, tests, approvals, monitoring adjustments.

Conclusion: Data integration without a data graveyard requires technology — and operational clarity

ETL remains a solid tool for reporting, provided you have schedules, data contracts and growth of batch windows under control. CDC is often the pragmatic path to up-to-date data, relieves source systems and creates a clean separation between OLTP and analytics. Event Streaming is powerful when processes must react and multiple systems consume events — but it requires consistent error handling, versioning and ownership per consumer.

In practice the decisive question is not „which technology is modern“, but: Which latency and reliability do our processes need — and which operational capability can we sustain long-term? If you clarify that early, integrations can be built to grow without rotting.

If you want to modernize your integrations between ERP, CRM and warehouse in a structured way — including an operational concept, data contracts and a migration path — talk to us:

For this topic, Change Data Capture (Cdc) and ERP integration are also important. This article contextualizes these aspects clearly and shows what matters in day-to-day operations.

Discuss a project or modernization initiative with Net-Base.