Apache Airflow with Shopware: Scalable Orchestration for Modern E-Commerce

Apache Airflow is an open-source platform that lets developers define data pipelines as code, run them on a schedule and monitor them. For Shopware shops, and for UK merchants syncing stock, prices and orders across several systems, it becomes relevant once imports, reconciliations and reports grow too complex for cron jobs.

Airflow describes workflows as DAGs (directed acyclic graphs): each node is a task, and the edges define order and dependencies. DAGs are written in Python, a scheduler starts them on a schedule or on a trigger, workers execute the tasks, and a web interface shows runs, durations, errors and logs. Retries on failure, notifications and backfills for missed time windows are built in.

Originally created at Airbnb, Airflow is now an Apache project and a standard tool in data engineering teams. Managed versions are available from Astronomer, AWS (MWAA) and Google Cloud (Cloud Composer).

For recurring jobs, Shopware ships with the scheduled task mechanism and the message queue, and for events there is the Flow Builder. Airflow is worth it when workflows reach beyond Shopware and several systems are involved in a defined order:

• Product imports with pre-processing: load supplier data, clean it, enrich it with PIM data, convert images, then write it to Shopware via the Sync API.
• Stock and price reconciliation from several sources (ERP, warehouse, marketplaces), with rules on which source takes priority.
• Reporting and data warehouse: load orders, customers and product data into BigQuery, Snowflake or PostgreSQL overnight and prepare reports.
• Fulfilment chains: hand orders over to logistics partners such as Royal Mail, Evri or DPD UK, collect their responses, set the status in Shopware and flag exceptions to customer service.
• Machine learning pipelines: train recommendations or demand forecasts and feed the results back as product data or tags.

For individual simple workflows, Airflow is overkill; the Flow Builder, a scheduled task in a plugin or a platform such as n8n will do.

Airflow talks to Shopware via the Admin API. A typical stock sync in simplified form:

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime, timedelta

def load_stock(): ... # fetch stock levels from the ERP
def clean_up(): ... # check SKUs, remove duplicates
def write_shopware(): ... # POST /api/_action/sync in batches of 500 products

with DAG("shopware_stock_sync", start_date=datetime(2026, 1, 1),
schedule="*/15 * * * *", catchup=False,
default_args={"retries": 3, "retry_delay": timedelta(minutes=5)}) as dag:
load = PythonOperator(task_id="load", python_callable=load_stock)
check = PythonOperator(task_id="check", python_callable=clean_up)
write = PythonOperator(task_id="write", python_callable=write_shopware)
load >> check >> write

The DAG runs every 15 minutes, retries failed tasks three times and shows every run in the interface. Shopware credentials are kept in Airflow connections or a secrets backend, never in the code.

• Hosting: Airflow needs a scheduler, web server, workers and a metadata database. Docker Compose on your own server is enough to get started; for production, managed services are the better choice.
• Idempotency: every task must be able to run several times without causing damage. Fixed IDs and upserts via the Sync API make sure retries do not create duplicates.
• API load: write large data volumes in batches and move indexing to the queue so the shop stays fast during the run.
• Monitoring: alerts for failed runs by email, Slack or Teams; keep an eye on run times to spot growing data volumes early.
• Data protection: only pull customer data into pipelines when the purpose requires it, and write logs without personal data, in line with UK GDPR.

• Shopware scheduled tasks and message queue: for jobs inside the shop, implemented as a plugin.
• Flow Builder and webhooks: for event-driven workflows without a schedule.
• n8n, Make, Zapier: for connections between SaaS tools without programming.
• Prefect, Dagster, Temporal: modern alternatives to Airflow with a different focus; Prefect and Dagster in data engineering, Temporal for long-running, reliable workflows.
• AWS Step Functions, Google Cloud Workflows: orchestration within the respective cloud, with no infrastructure of your own to run.

Apache Airflow is the right choice when data jobs around Shopware become complex, multi-step and business-critical, and a team wants to maintain them as code. For small shops, running it is too much effort; Shopware’s built-in tools or a no-code platform are enough there. We support UK merchants remotely from Bremen and Prishtina with the decision and build the connection to the Admin API, see Shopware plugin development. Our Shopware dictionary explains more terms.