Author name: Ritu Negi

Data Science

Send a result set of a Snowflake query as email in Tabular format using ADF

Sending an email notification is the traditional way of checking if the integration has been done properly or not. In ADF for sending an email, we need to configure the logic app in the Azure portal. We can send several notifications on respective events like on Success, Failure or Completion of a pipeline. We can send a result set of a query as an attachment as well. But when we need to send a result set of a query in an email as a body, we don’t get the right format. Here I am explaining how can we show the result set with multiple records in tabular format in an email body using some simple steps. So, for this, we have a query whose result set needs to be sent by email. Here I am using Snowflake as a database but this query can be used in SQL Server too. Now to get it in a single record and give it an HTML table format, we need to modify it as: I have used <tr> and <td> tags in between to give it a tabular format and got it in a single record to process it in the Lookup activity in ADF in a single go else we need to use For each activity to process each record. Now if the logic app is already configured in the Azure portal, then create a pipeline in ADF and take a lookup activity as below and use the above query in its settings:   I have already configured a pipeline for Success Email Notification using Web Activity like the below:   Now add this pipeline to our pipeline as below: In the pipeline PL_SendEmailSuccess settings, write the code to the parameter which is mentioned as Body in LogicApp as below: Now we will execute the pipeline. Though the output of lookup looks messy as below:   But email we get is properly formatted in table form: Conclusion: This type of task can be used to send the status report of the jobs which run on a daily basis, to send an alert on significant metrics, or to get the result of ETL audits with rejected records. So instead of downloading attachments, users can have a quick check using this approach.

Data Science

Power Automate as Bridge to Write-back on Synapse Using Power Apps

 The Write-Back functionality enables end-users to give input and update the data in the data warehouse directly from the BI report. In addition, this helps BI professionals to have a rich user experience in deploying solutions for budgeting, forecasting, planning, and adding comments.  This functionality can be achieved using Power Apps. There is no connector for Azure Synapse in PowerApps, therefore, the write-back to Synapse using Power Apps does not work directly.   This blog is to show the process of write-back on Azure Synapse using Power Apps while making Power Automate a bridge.   Power Automate has a Synapse connector that allows the changes in the database. For this we need a Synapse SQL table in which the data will be stored (Created WriteBackTable table), a power app form that has some input fields and a submit button. Then we will create a flow in power automate that will be then called in Power Apps Submit Button.     Here, the WriteBackTable is being used as a base table on which the Fact table will be created to show the Daily and Monthly Goal for different department in each month. The Department, Monthly Goal and Number of working days will be entered by the users using PowerApps. Below is the script for WriteBackTable:         The RowId is the primary key for unique Department and Month Year, which can be used further to decide weather the record will be inserted or updated. Initially there is no record in the table so the data will be inserted:         Below is the complete flow which is explained further in detail:         For creating the flow, first we need to click on Create and select the Instant Cloud Flow:       Then we need to select the trigger of this flow as PowerApps and give our flow a name and then click on Create:     After that, we need to initialize a variable that we call from PowerApps:     We can Name the variable as we want and when we click on the value, the Dynamic content option pops up and we need to click on Ask in PowerApps:     This step will be repeated as many times as we have input fields in our PowerApps form, for me, its 4 so used 4 Initialize Variables as below.     Once we will initialize all the variables then we will add action Execute a SQL query (V2) under Azure SQL Data Warehouse to see if there are any records in the Synapse table for the key columns entered by PowerApps.   We are using “Azure SQL Data Warehouse” connector that allows us to connect to Azure Synapse and run SQL queries and stored procedures to perform CRUD (create, retrieve, update and delete). But it is available only in Power Automate and not in Power Apps and that is why we are using Power Automate as a bridge here:     In this option, we need to create a connection by clicking the three dots on the top right corner and then click on Add new connection:       Then we need to insert credentials for Azure Synapse Dedicated SQL pool using Basic username and password.       Once you will establish the connection, the Server Name and Database will come in the drop-down list and you can select them and write your query as above.   Note: We need to define the parameters and the data types that we need in our query. Here we need Department and MonthYear as the key columns (on the basis a record will be inserted/updated) so two parameters are created. Then assign the value by selecting the corresponding variable (coming from the PowerApps) from the Dynamic Content:       After this step we need to decide if the record will be inserted or updated in the table so we need to add a new step Condition:       In this step, we need to write an expression to know if the select query in the previous step returns any data or not. Here the expression is as below:   empty(outputs(‘Execute_Select_from_Synapse’)?[‘body/resultsets/Table1’])   But instead of copying the above line of code, you can run the flow and see the output for this step and note in which set you are getting the result, and then use the empty function in Dynamic content and select that set:       Now if the condition is True, i.e., the result set is empty then we need to insert the record else the update will be done. Therefore, for this we need to use the Execute a SQL query (V2) again and use the insert and update query:     After the flow is completed, go to PowerApps and click on the Submit button in the form on which we want to call the Power Automate flow and then click on Action and then Power Automate:     Then select your flow and give the parameters as below:     This App can be called using Power BI and whenever the user will enter values in the form and click on the submit button. The flow will then get triggered and data will be inserted/updated in the Synapse Database.     Conclusion: The PowerApps does not have any connector to Azure Synapse and hence there is no direct way to write back to the synapse database but using Power Automate as a bridge, we can use this functionality without having any trouble.

Azure, Data/Performance Analytics, Decision Making, Expertise Data Warehousing, SQL

Analyze at the speed of thought with Azure Synapse

Azure Synapse Analytics is an infinite analytics service that unites data integration, enterprise data warehousing, and big data analytics. Using either server-less or dedicated resources at scale, you have the freedom to query data your way. Here we will be comparing the performance at different scales.   Azure Synapse Performance Comparison   In order to compare the performance of different data volume tables with different scale, we have below set of tables/views: [dbo].[Sales_Order_2Years] [dbo].[Sales_Order_4Years] [dbo].[Sales_Order_6Years] [dbo].[Sales_Order_8Years] [dbo].[Sales_Order_10Years] Process of Scaling the Azure Dedicated SQL Pool   Default Azure Dedicated SQL pool is set on DW100c specification. To change the scaling on the higher specification, we can click on the scale button as shown in the below screenshot of the Azure Dedicated Pool window of Azure Portal.       In the above screenshot, we have changed the scale to DW200c, in the highlighted section it is showing all the pricing details. Once, change clicked on the save button on the same window to set the specification to DW200c.   SQL used in the process of performance analysis   1. Select count(*) from Sales_Order_2Years;   2. Select DISTINCT Part_Code from Sales_Order_2Years;   3. Select sum(cast(Month_Primary_Amount as float)) from Sales_Order_2Years;     4. Select sum(cast(Month_Primary_Amount as float)) ,Part_Code from Sales_Order_2Years group by Part_Code ;     5. Select sum(cast(Month_Primary_Amount as float)),Part_Code ,Organization_Name from Sales_Order_2Years group by Part_Code, Organization_Name;     6. Select sum(cast(Month_Primary_Amount as float)),Part_Code FROM Sales_Order_2Years group by Part_Code order by Part_Code DESC ;   7. Select sum(cast(Month_Primary_Amount as float)),sum(cast(Primary_Quantity as float )),Part_Code,Organization_Name From Sales_Order_2Years group by Part_Code,Organization_Name order by Part_Code desc;   Similarly, executed queries with other tables having different volumes.   Performance Comparison on Difference DP Specifications Here following are the comparison among the different DP Specifications with different data sets of Volume in millions.   Conclusion: The above comparison clearly shows that when there is a large volume of data (in Millions) and the specification is higher on the dedicated Azure Pool, the performance increases. However, it doesn’t get better on low data volume whether the specification is higher or lower.   Scale Up Dedicated SQL Pool using Pipeline   Note: To allow the Synapse workspace to call the REST API, we need to give the Synapse workspace access to the SQL Pool. In the Access control (IAM) of the SQL Pool, assign the contributor role to your Synapse Workspace. Now, to scale up the SQL Pool using pipeline, we need to create a pipeline and set the parameters as below:   Above are the generic Parameters used within the Pipeline. Subscription Id: Subscription Id of Synapse Workspace ResourceGroupName: Name of the ResourceGroup of the used Synapse Workspace   Region: Location of the Synapse Workspace ServerName: Synapse Workspace Name DatabaseName: Name of the dedicated SQL Pool Command: Resume performance level: To level which we need to ScaleUp   Then in the pipeline, we need a Web Activity which will get the status of the SQL Pool whether it is Pause or Resume, an IF Condition which will check if the status is Resume or not, and again a Web Activity which will then Scale Up the SQL Pool.     Here is the setting of check status web activity:   Here is the URL value: @concat(‘https://management.azure.com/subscriptions/’,pipeline().parameters.SubscriptionId,’/resourceGroups/’,pipeline().parameters.ResourceGroupName,’/providers/Microsoft.Synapse/workspaces/’,pipeline().parameters.ServerName,’/sqlPools/’,pipeline().parameters.DatabaseName,’?api-version=2019-06-01-preview’)   Method = GET Resource =https://management.azure.com/   Now in If Condition, we will check if the status is not Resume and if it is not i.e. condition is true then in its Activity we will resume the status by assigning a Web Activity for it.   Here the expression in IF Condition is : @and(equals(activity(‘Check Azure Synapse Analytics Status’).output.properties.status,’Paused’), equals(pipeline().parameters.Command,’Resume’)) To add Web Activity we need to click on the pencil icon besides True as below and drag a Web Activity to that window from the left pane:       In Settings we need to give the below information: URL Value: @concat(‘https://management.azure.com/subscriptions/’,pipeline().parameters.SubscriptionId,’/resourceGroups/’,pipeline().parameters.ResourceGroupName,’/providers/Microsoft.Synapse/workspaces/’,pipeline().parameters.ServerName,’/sqlPools/’,pipeline().parameters.DatabaseName,’/’,pipeline().parameters.Command,’?api-version=2019-06-01-preview’)   Method: Post Body: {} Resource =https://management.azure.com/   Once the status is set to Resume(or if it is confirmed that it is already resumed) the last Web Activity will Scale up the SQL Pool as per the given parameter. URL Value: @concat(‘https://management.azure.com/subscriptions/’,pipeline().parameters.SubscriptionId,’/resourceGroups/’,pipeline().parameters.ResourceGroupName,’/providers/Microsoft.Synapse/workspaces/’,pipeline().parameters.ServerName,’/sqlPools/’,pipeline().parameters.DatabaseName,’/’,’?api-version=2019-06-01-preview’)   Method: Patch Header: Name = Content-Type Value= application/json Body: {“sku”: {“name”: ‘@{pipeline().parameters.PerformanceLevel}’}} Resource =https://management.azure.com/   You can schedule the pipeline before running ETLs in order to load the data and increase the performance by scaling up the SQL Pool and then in the same way you can create a pipeline to scale it down as before and schedule it after the load is done.   Azure Synapse offers an enterprise-grade analytical engine for datasets of any size. It offers a flexible pricing model to enable businesses to start small and scale up based on their need. If you would like to explore Azure Synapse for your business, Accelerant offers a free demo environment customized to your data needs. Reach us at info@ymz.633.myftpupload.com for more information.               Author: Ritu Negi Technical Lead, Accelerant

Activate Your Free 15-Day Infominer Trial

X
Scroll to Top