Let's talk Azure
Posts Tags Categories Series Showcase
Let's talk Azure

Aggregator pattern with Microsoft Fabric and GraphQL

Divakar Kumar  included in  category Azure and series DurableMultiAgents
     2010 words   10 minutes 
https://dev-to-uploads.s3.amazonaws.com/uploads/articles/16apao2ttb9kxazhvktc.jpg
Series -

How can Microsoft Fabric help address common challenges in microservice patterns?

In this blog, I explore how GraphQL APIs in Microsoft Fabric can help address the communication challenges that often arise in service-to-service interactions within microservices or event-driven architectures, particularly for users who are already leveraging Microsoft Fabric to consolidate data from multiple sources. If you’ve been following my blogs, you might already be familiar with this project that I’m currently working on. While I haven’t started explaining the Durable MultiAgent project in detail, this blog stands independent of that series. You don’t need prior knowledge of what I’ve built so far. Instead, I’ll focus entirely on integrating Microsoft Fabric and setting up database mirroring across various data sources.

Here’s what I’ll cover:

  1. Aggregator Pattern
  2. GraphQL
  3. Database Mirroring
  4. Setting Up GraphQL in Microsoft Fabric
  5. Demo

The Aggregator Pattern is a design strategy used in microservices architectures to handle the challenge of aggregating data from multiple services. Instead of a client making multiple API calls to different services, the aggregator acts as a single point of contact.

Ref: https://learn.microsoft.com/en-us/azure/architecture/patterns/gateway-aggregation

Key Benefits:

  1. Reduced Network Overhead: The aggregator fetches and combines data from multiple sources, reducing the number of API calls the client needs to make.
  2. Simplified Client Logic: Clients only need to interact with one service instead of managing the complexity of multiple APIs.
  3. Improved Performance: By consolidating multiple calls into a single request, the overall response time can be improved, especially for high-latency services.

GraphQL is a query language and runtime for APIs, designed to offer clients exactly the data they need, no more and no less. Unlike traditional REST APIs, where fixed endpoints return predefined data structures, GraphQL allows clients to specify the shape of the response they want.

A GraphQL server serves as a middleware that provides a unified interface to interact with diverse data sources like SQL databases, NoSQL stores, REST APIs, or files. It operates on top of an HTTP server and processes requests using a schema that defines the structure of available data and operations. Clients send queries, mutations, or subscriptions in a JSON format to the server, specifying exactly what data they need. The server translates these requests into the appropriate data source-specific calls, fetches the data, and returns the results in a structured JSON response. This approach minimizes over-fetching or under-fetching of data, offering a more efficient and flexible alternative to traditional REST APIs.

The GraphQL API exposes this functionality, allowing developers to interact with multiple data sources through a single endpoint. Its schema provides a clear contract, ensuring clients know what data and operations are available. Developers can use queries to fetch specific data, mutations to modify it, and subscriptions to receive real-time updates. This abstraction eliminates the need for developers to understand the complexities of underlying data sources, enabling faster development with better security and performance. By decoupling the client from backend intricacies, GraphQL simplifies integration and supports modern application needs.

  1. Flexible Data Retrieval: Clients can request only the fields they need, reducing over-fetching or under-fetching of data.
  2. Single Endpoint: A single GraphQL endpoint can replace multiple REST endpoints, simplifying API management.
  3. Strongly Typed Schema: GraphQL APIs are defined by a schema, making them self-documenting and easier to understand for developers.
  4. GraphQL shines in microservice architectures by serving as a unified API layer. It can query multiple services simultaneously, aggregate their responses, and return a single, cohesive result to the client. This aligns well with the Aggregator Pattern.

Database Mirroring in Fabric is a feature that enables the replication of entire databases and tables from various data sources into Microsoft Fabric’s OneLake. This allows organizations to consolidate data from disparate systems into a unified analytics platform, supporting near real-time data availability.

Key Benefits of Database Mirroring:

  1. Simplified Data Integration: Eliminates the need for complex ETL pipelines by providing seamless data replication into OneLake.
  2. Near Real-Time Data Sync: Ensures up-to-date data availability for analytics by automatically syncing changes from the source systems.
  3. Low-Cost and Low-Latency Solution: Offers a cost-efficient and fast mechanism to replicate data into an analytics-ready format.
  4. Broad Compatibility: Supports various data sources like Azure SQL Database, Cosmos DB, and Snowflake, enabling integration across platforms.
  5. Analytics-Ready Data: Converts replicated data into Parquet format for easy consumption by Fabric services, such as Power BI, Spark, and data engineering tools.
  6. Secure and Collaborative Access: Includes features like Row-Level Security (RLS) and Object-Level Security (OLS), ensuring controlled access and secure decision-making.
  7. End-to-End Usability: Fully managed service with tools like SQL analytics endpoints and semantic models for querying and analysis.
  8. Cross-Database Queries: Allows T-SQL queries that join data across mirrored databases and warehouses, simplifying analytics workflows.

This architecture make use of database mirroring to synchronize data from microservices, such as Booking, Flight, Weather, and Passenger services, with corresponding SQL databases. Each microservice operates independently in a containerized environment, ensuring modularity and scalability. The mirrored databases allow seamless replication of transactional data, enabling a consistent and reliable source for downstream processing. This mechanism is critical for ensuring real-time availability of data while reducing the load on the primary transactional systems, thus maintaining high performance and fault tolerance.

Microsoft Fabric is integrated into this system to process and analyze the mirrored data using its Landing Zone and Change Data Capture (CDC) capabilities. The CDC processor continuously tracks changes in the mirrored databases and transforms them into Delta Parquet files stored in OneLake, which serves as a unified data lake for analytics. A GraphQL API layer built on top of the processed data provides clients with a flexible query interface, enabling precise data access through queries and mutations. This integration streamlines data flow from transactional systems to analytical endpoints, fostering a robust data ecosystem for real-time insights and decision-making.

  • Register an application in Microsoft Entra to obtain:

    • clientId
    • clientSecret
    • tenantId

  • Add required API permissions:

    • Under Power BI service, select Delegate Permission.
    • Choose the Item.ExecuteAll permission.

  • Configure client credentials:

    • Use the client credentials flow for backend service consumption.
    • Redirect URIs are not required since this is a backend service setup.
  • Sign up for a free Microsoft Fabric trial:
    • Use your official account associated with the same tenant where your workloads are hosted.
    • Ensure you have a Power BI license. If you don’t have one, sign up for a Power BI license before starting the trial.
  1. In Microsoft Fabric, create a new workspace.
  2. Under +New Item, select Mirrored Azure SQL Database.
  1. Configure connection settings:
    • Choose SPN (Service Principal Name).
    • Provide the clientId, tenantId, and clientSecret obtained earlier.
  2. Grant permissions in Azure SQL Database:
    • Run the following commands in your Azure SQL database: 
      1
2
3
4
5

      CREATE USER [EntraAppRegisteredName] FROM EXTERNAL PROVIDER;

ALTER ROLE db_datareader ADD MEMBER [EntraAppRegisteredName];
ALTER ROLE db_datawriter ADD MEMBER [EntraAppRegisteredName];
GRANT CONTROL TO [EntraAppRegisteredName];
  3. Once connected, the initial database mirroring process will begin. It may take some time initially, but subsequent replication will be seamless, providing a near real-time integration experience.
  1. Under +New Item, select API for GraphQL.
  2. Provide a name for the API and create it.
  3. Once the API is created, start adding data sources.
  1. Data sources to mirror:
    • Mirror Azure SQL Database.
    • Mirror Azure Cosmos DB (ensure the Database Mirroring feature is enabled in the Cosmos DB account before proceeding).
  2. Steps to add data sources:
    • Click on Select Data Source.
    • Choose Single Sign-On (SSO) for authentication.
      • Avoid using saved credentials as they won’t work when consuming the GraphQL endpoint in a backend application.

Once the data sources are added, you can begin setting up relationships to enable cross-table and cross-database queries.

  1. Open the Relationship Manager:

    • Select the desired table.
    • Click on Manage Relationships.
    • In the popup window, choose to Create New Relationship.

  2. Define the Relationship:

    • Choose the Type of Relationship:
      • One-to-One
      • One-to-Many
      • Many-to-Many
      • Many-to-One
    • Specify the From Type and select the source table.
    • Select the fields from the source table.

  3. Map to the Target Table:

    • Specify the To Type and select the target table.
    • Map the respective fields between the source and target tables.

  4. Finalize the Relationship:

    • Once the fields are mapped, click Create Relationship to save.

After defining the relationships, your data model is ready for performing cross-table and cross-database queries seamlessly.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

query GetBookingDetails($passengerId: String!, $flightNumber: String!) {
  passengers(filter: { Id: { eq: $passengerId } }) {
    items {
      Id
      FirstName
      LastName
      Email
      Phone
    }
  }
  flightListings(filter: { FlightNumber: { eq: $flightNumber } }) {
    items {
      FlightId
      DepartureAirportCode
      DestinationAirportCode
      Duration
      FlightNumber
      DepartureTime
      Price
      AvailableSeats
      AircraftType
      airlines {
        items {
          AirlineId
          Name
          Code
          Country
          City
          LogoUrl
        }
      }
    }
  }
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

using Microsoft.Extensions.Configuration;
using Microsoft.Identity.Client;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using TravelService.MultiAgent.Orchestrator.Interfaces;

namespace TravelService.MultiAgent.Orchestrator.Services
{
    public class FabricAuthService : IFabricAuthService
    {
        private readonly string _clientId;
        private readonly string _clientSecret;
        private readonly string _authority;

        public FabricAuthService(IConfiguration configuration)
        {
            // Fetch Client ID, Client Secret, and Tenant ID from configuration
            _clientId = configuration["Fabric:ClientId"];
            _clientSecret = configuration["Fabric:ClientSecret"];
            _authority = $"https://login.microsoftonline.com/{configuration["Fabric:TenantId"]}";
        }

        public async Task<string> GetAccessTokenAsync()
        {
            // Create a confidential client application using MSAL
            var app = ConfidentialClientApplicationBuilder.Create(_clientId)
                .WithClientSecret(_clientSecret)
                .WithAuthority(_authority)
                .Build();

            // Define the required scope for Microsoft Fabric (Power BI)
            var scopes = new[] { "https://analysis.windows.net/powerbi/api/.default" };

            // Acquire the token and return it
            var result = await app.AcquireTokenForClient(scopes).ExecuteAsync();
            return result.AccessToken;
        }
    }
}

This simplifies authentication with Microsoft Fabric APIs by leveraging Azure AD’s client credentials flow. It securely retrieves client credentials (Client ID, Client Secret, and Tenant ID) from the configuration, uses the Microsoft Authentication Library (MSAL) to acquire an access token for the required API scopes (e.g., Power BI), and provides this token to enable seamless interaction with Microsoft Fabric services for tasks like database mirroring or GraphQL API integration.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

public async Task<dynamic> FetchBookingDetailsAsync(string passengerId, string flightId)
{
   var token = await _authService.GetAccessTokenAsync();
   using var client = new GraphQLHttpClient(_graphqlUri, new NewtonsoftJsonSerializer())
   {
      HttpClient = { DefaultRequestHeaders = { Authorization = new AuthenticationHeaderValue("Bearer", token) } }
   };
   client.Options.IsValidResponseToDeserialize = response => response.IsSuccessStatusCode;

   var query = new GraphQLHttpRequest
   {
      Variables = new { passengerId, flightId },
      Query = @"
              query GetBookingDetails($passengerId: String!, $flightId: String!) {
                  passengers(filter: { Id: { eq: $passengerId } }) {
                      items { Id, FirstName, LastName, Email, Phone }
                  }
                  flightListings(filter: { FlightNumber: { eq: $flightId } }) {
                      items {
                          FlightId, DepartureAirportCode, DestinationAirportCode, Duration, FlightNumber,
                          DepartureTime, Price, AvailableSeats, AircraftType,
                          airlines { items { AirlineId, Name, Code, Country, City, LogoUrl } }
                      }
                  }
              }"
   };

   var response = await client.SendQueryAsync<dynamic>(query);
   return response.Data;
}

The FetchBookingDetailsAsync method fetches booking-related details for a passenger and a flight by querying a GraphQL API. It first retrieves an access token using the FabricAuthService for secure authentication. It then creates a GraphQL client configured with the API endpoint and sets the authentication token in the HTTP headers. The method sends a GraphQL query that retrieves passenger details (e.g., name, email, phone) and flight details (e.g., flight number, departure time, price, airline information) based on the provided passenger ID and flight ID. The response data is returned dynamically, enabling seamless integration of booking and flight information.

Updated on 2024-12-25
Read markdown
 .Net Core, Asp.net, Csharp, Azure Open Ai, Azure Cosmos Db, Azure Sql, Azure Log Analytics, Application Insights, Azure Monitor, Microsoft Fabric, Azure Aad, GraphQL, Microservice
Back | Home