Automating data workflows is key to efficient data integration, and Azure Data Factory (ADF) enables this through triggers. Instead of manually starting pipelines, you can use triggers to automate execution by schedule or event. Two of the most popular options are the Schedule Trigger and Tumbling Window Trigger in ADF, which streamline time-based and batch processing.
- Schedule Trigger: Executes pipelines at predefined times or intervals.
- Tumbling Window Trigger: Executes pipelines in consecutive, non-overlapping time windows—ideal for batch and time-based processing.
This guide walks you through creating both types, with practical examples and tips.
1. What is a Schedule Trigger ?
A Schedule Trigger lets you run a pipeline at designated times, similar to a cron job. Configuration options include:
- Running at a specific time of day
- Recurring execution every X minutes, hours, or days
- Setting start and end dates
Common use cases: Daily ETL processes, weekly data updates, or hourly incremental loads.
Steps to Create a Schedule Trigger in ADF
Step 1 — Open Trigger Menu:
Inside ADF Studio, open your pipeline, click Add Trigger, and select New/Edit.
Step 2 — Create New Trigger:
Create a new trigger, name it DailyTrigger, and set the type to Schedule.
Step 3 — Configure Recurrence:
Pick the start date, specify the time zone, and choose the recurrence. For example, set it to run every day at 7 AM IST.
Step 4 — Publish and Enable:
Click OK and then Publish All to save changes. Finally, go to Manage → Triggers and enable your DailyTrigger.
Your pipeline is now set to run automatically on your chosen schedule.
2. What is a Tumbling Window Trigger ?
A Tumbling Window Trigger divides time into fixed, non-overlapping intervals (windows). Each pipeline run processes data for one window, with no overlaps or missed intervals. This is perfect for time-series data, logs, or batch ingestion tasks.
Example:
A 1-hour tumbling window starting at midnight will trigger runs for:
- 12:00–01:00
- 01:00–02:00
- 02:00–03:00
- …and so on.
Steps to Create a Tumbling Window Trigger in ADF
Step 1 — Create Trigger:
Open your pipeline, click Add Trigger, and select New/Edit. This time, choose Tumbling Window Trigger.
Step 2 — Configure Windows:
First, provide a name for your trigger. Select the schedule as Tumbling Window. Next, set the start time and define the recurrence. For example, if you select every 1 hour, the pipeline will run once for each hourly window.
Step 3 — Advanced Options:
You can add a delay for late-arriving data, set retry counts for failures, and use system variables like windowStartTime and windowEndTime within your pipeline for filtering data.
Step 4 — Publish and Enable:
Once the trigger is configured, publish and enable it. Your pipeline will now run for each hourly window, ensuring no overlaps and complete data coverage.
Your pipeline will now run for each window, ensuring consistent, reliable processing.
3. Key Differences Between Schedule Trigger and Tumbling Window Trigger in ADF
| Feature | Schedule Trigger | Tumbling Window Trigger |
|---|---|---|
| Execution Style | Runs at fixed times | Runs in sequential time windows |
| Overlap | Can overlap if configured | Never overlaps |
| Use Cases | Daily/weekly batch jobs, simple scheduling | Time-series, event log processing, incremental data loads |
| Backfill Capability | Not supported | Supports backfilling missed windows |
4. Best Practices
- Use Schedule Triggers for straightforward, recurring tasks (e.g., daily reports).
- Choose Tumbling Window Triggers for time-partitioned or backfillable data processing.
- Always set a retry policy for resilience.
- Monitor pipeline activity in the Monitor tab to ensure triggers are functioning as intended.
Conclusion
Azure Data Factory (ADF) offers powerful automation capabilities through the use of triggers, allowing you to transition from manual interventions to fully automated, enterprise-grade data workflows.
- Schedule Triggers are ideal for running jobs at predetermined times, such as daily or weekly batch loads, ensuring tasks execute consistently without manual oversight.
- Tumbling Window Triggers provide dependable, time-based batch processing with non-overlapping intervals and built-in support for backfilling missed runs, making them perfect for scenarios requiring strict data consistency and completeness.
Selecting the appropriate trigger type enables you to automate workflows, improve efficiency, and minimize the risk of human error in critical data processes. Additionally, ADF triggers can be seamlessly integrated with monitoring, alerting, and retry policies to create highly resilient and scalable data pipelines, further enhancing operational reliability.
Chapter 1:- Create a Schedule Trigger in ADF
📌 Watch the full video here: https://www.youtube.com/watch?v=pTjBYBSB89I
Chapter 2:- Create a Tumbling Window Trigger in ADF
📌 Watch the full video here: https://www.youtube.com/watch?v=9XVfPg08HZQ