Azure Synapse Analytics Training Content
Introduction & Overview
- Azure Synapse Analytics Overview
- Anzure Synapse Analytics Architecture
- Create Azure Free Account for Synapse
Overview of pools in Synapse Analytics
- Dedicated SQL pools
- Serverless SQL pool
- Apache Spark pools
- Data Explorer pools
Using Azure Synapse Analytics to Query Data Lake
- Creating Azure Synapse Analytics Workspace
- Uploading Sample Data into Data Lake Storage
- Exploring Azure Synapse Workspace and Studio
- Querying a Data Lake Store using serverless SQL pools in Azure Synapse Analytics
- Creating a View for CSV Data with a Serverless SQL Pool
Azure Storage Account Integration with Azure Synapse
- Copy multiple files from blob to blob using wildcard file options
- Copy multuple folders from blob to blob using dataset parameters
- Get File Names from Folder Dynamically and copy latest file from folder
Azure Synapse Triggers
- Schedule Trigger in Azure Synapse
- Event Based Trigger in Azure Synapse
Azure SQL Database integration with Azure Synapse
- Azure SQL Databases – Introduction __ Relational databases in Azure
- Copy data from SQL Database to ADLS Gen2 using table, query and stored procedure
- Overwrite and Append Modes in Copy Activity in Azure Synapse
- Use Foreach loop activity to copy multiple Tables- Step by Step Explanation
Incremental Load to Azure Synapse in Azure Synapse
- Incremental Load or Delta load from SQL to blob Storage in Azure Synapse
- Multi Table Incremental Load or Delta load from SQL to to Azure Synapse
- Incrementally copy new and changed files based on Last Modified Date
Logging and Notification and keyvault integration __ Azure Logic Apps
- Log Pipeline Executions to SQL Table using Azure Synapse
- Custom Email Notifications and keyvault integration with Linked Service
- Send Error notification with logic app
- Use Foreach loop activity to copy multiple Tables with pipeline logs logic and notifications
Deep dive into Copy Activity in Azure Synapse
- Load data from on premise sql server to Azure Synapse
- Copy Data from sql server to to Azure Synapse with polybase & Bulk Insert
- Copy Data from on-premise File System to Azure Synapse
- Loop through REST API copy data 2 ADLS Gen2 Linked Service Parameters
Data Flows Introduction
- Azure Data Flows Introduction
- Setup Integration Runtime for Data Flows
- Basics of SQL Joins for Azure Data FlowsServerless SQL Pool Demo
- Joins in Azure DataFlowsDedicated SQL Pool Demo
- Difference Between Join vs.Lookup Transformation& Merge Functionality Spark Pool Demo
- Dataflows – select the rows
- Dataflows – Filter the rows
- Dataflows – join Transformations
- Dataflows – union Transformations
- Dataflows – look up Transformations
- Dataflows – window functions transformations
- Dataflows – pivot, unpivot transformations
- Dataflows – Alter rows transformations
- Dataflows – Removing Duplicates transformations
Spark Pool Introduction in Azure Synapse
- Spark Introduction and components
- Spark Architecture
- Create notbook and notebook option and create notebook in different langauges
- MSSparkUtils for file system
- MSSparkUtils for for creating notebook parameters
- Magic commands and calling one synapse notebook from another and returning output of synapse notebook
- Configure keyvault in azure synapse notebook
- Different ways to connect to ADLSGen2 from synapse notebook
- Different ways to connect to Blob from synapse notebook
- Different ways to connect to Azure SQL Database from synapse notebook
- Different ways to connect to on premise SQL Server from synapse notebook
- Optimization while Reading and writing CSV files from Azure Synapse
- Reading and writing parquet files from Azure Synapse
- Reading and writing JSON files from Azure Synapse
- Reading and writing avro and orc files from Azure Synapse
- Reading and writing EXCEL files from Azure Synapse
- Different ways to create RDD in synapse notebook
- Different ways to create dataframes in synapse notebook
- When to use repartition and coalesce
- Joins in Synapse Notebook
- Broadcast Joins in Synapse Notebook and configuration of spark for optimization
- what is catalyst optimiser and skewness issue in spark
- Optimization techniques in pyspark
- Implementing SCD1 in Synapse Notebook
- Implementing SCD2 in Synapse Notebook
- Executing synapse notebooks from synapse pipleines with input and output parameters
Project: End to End DataMigration using Synapse Analytics