ADF Course Content
1) Azure Data Factory
- What is Azure Data Factory?
- Azure Data Factory Architecture
- Azure Data Factory Portal UI
- Top-level concepts
- Pipelines
- Activities
- Linked services
- Datasets
- Triggers
- Data Flows
- Integration Runtimes
2) Pipeline
- What is a Pipeline?
- Create a new pipeline
- Organize pipelines into folders
- Debug pipeline
- Publish pipeline
- Parameters / Pipeline Parameters
3) Linked Service
- What is a Linked Service?
- Create a Linked Service for –
- BLOB
- SQL Database
- SQL Server
- Data Lake Storage Gen1
- Azure Data Lake Storage Gen2 etc
- Parameters / Linked Service Parameterization
4) DataSets
- What is a Data Set?
- Create a Data Set for –
- Avro, Binary, CSV, Excel, JSON, ORC, Parquet, XML in BLOB/ADLS Gen1/ADLS Gen2.
- Table in SQL Database, SQL Server, Oracle Database etc
- Parameters / Data Set Parameterization
5) Activities
- Wait
- Variables
- Create a variable
- Set variable
- Append variable
- Copy Data
- General
- Source
- Sink
- Mapping
- Settings
- User Properties
- Copy file(s) from one BLOB Container to another Container
- One file from a folder
- All files from a folder
- All files and folders recursively from a folder
- Copy data / file from BLOB to SQL Database / ADLS Gen2
- As CSV, TSV, Parquet, Avro, ORC etc.
- Databricks Notebook
- Azure Function
- Lookup, Stored Procedure
- Get Metadata, Delete
- Execute Pipeline
- Validation, Fail
- Iteration & Conditionals
- Filter
- ForEach
- If Condition
- Switch
- Until
6) What is a Trigger?
- Types
- Schedule
- Tumbling window
- Storage Events
- Triggers with Parameters
7) Integration Runtime (IR)
- Azure AutoResolveIntegrationRuntime
- Azure Managed Virtual Network
- Self-Hosted
- Linked Self-Hosted
8) Source control
- Git configuration
- ARM Template
- Export / Import
- Azure Devops Repos
9) Global parameters
10) Credentials
11) Monitoring ADF Jobs
12) Alerts
13) Send Failure Notifications using Logic Apps
14) Data Flows
- What is Data Flow?
- Mapping Data Flow
- Data Flow Debug
- Transformations
- Filter, Aggregate, Join
- Conditional Split, Derived Column
- Exists, Union, Lookup, Sort,
- GroupBy, Pivot, Unpivot, Flatten etc.
- Flatten, parase, stringify
- Filter sort, alterrow,asset
- flowlet
- Validate Schema, Schema Drift
- Remove Duplicate Rows using Mapping Data Flows in Azure Data Factory
15) Azure Devops
- Repos
16) SDLC
17) Agile Methodology
18) ADF Interview Questions
19) ADF Resume Preparation
20) End TO End ADF Project
21) ADF Exercises
- Create variables using set variable activity
- How to use if condition using if condition activity
- Iterating files using for loop activity
- Creating linked services, Data sets
- Copy activity – blob to blob
- Copy activity – blob to azure SQL
- Copy activity – pattern matching files copy
- Copy activity – copy the filtered file formats
- Copy activity – copy multiple files from blob to another blob
- Copy activity – Delete source files after copy activity
- Copy activity – using parameterized data sets
- Copy activity – convert one file format to another file format
- Copy activity – add additional columns to the source columns
- Copy activity – filter files and copy from one blob to another
- Delete the files from blob with more than 100KB
- How to use getmetdata activity
- Bulk copy tables and files
- How to integrate keyvault in ADF
- How to set up integration run time
- Copy data from on premises to azure cloud
- How to use databricks activity activity and pass paraemeters to it
- How to use scheduling trigger
- How to use tumbling window trigger
- How to use event based trigger
- How to use with Activity
- How to use Until Activity
- Dataflows – select the rows
- Dataflows – Filter the rows
- Dataflows – join Transformations
- Dataflows – union Transformations
- Dataflows – look up Transformations
- Dataflows – window functions transformations
- Dataflows – pivot, unpivot transformations
- Dataflows – Alter rows transformations
- Dataflows – Removing Duplicates transformations
- How to pass parameters to the pipeline
- How to create alerts and rules
- How to set global parameters
- How to import and export ARM templates
- How to integrate ADF with Devops
- How to use Azure devops Repos
- How to send mail notifications using logic apps
- How to monitor the pipelines
- How to debug the pipelines
- How to schedule pipeline using triggers
- How to create trigger dependency
- How to one pipeline in another pipeline