AI Counselor - HackTogether: The Microsoft .NET 8 Global Hack

1. About this blog

It’s another day, another hackathon! 🌐 This time, it’s Microsoft’s HackTogether: The Microsoft .NET 8 Global Hackathon! 💻✨ Get ready for an exciting coding adventure! 🤖👩‍💻

🚀 Few key takeaways

• How to build cloud-native apps using .NET 8
• Introduction to .NET Aspire
• How to build IoT edge solution with Raspberrypi
• How to integrate AI and make intelligent applications
• How to train custom models for classifying face emotions using Azure CustomVision

2. Inspiration

In the fast-paced realm of work, we often neglect our mental well-being, giving rise to various challenges. To address this, We’ve introduced a solution that leverages AI.

• The AI Counselor app goes beyond traditional emotion tracking tools. It harnesses the power of artificial intelligence to offer real-time counseling tailored to the unique emotional challenges individuals encounter in their professional journeys.

• This innovative app seamlessly integrates into users’ daily routines, discreetly capturing emotional nuances through IoT edge and custom vision technology. As users navigate the ups and downs of their workday, the AI Counselor analyzes emotional patterns and delivers personalized counseling strategies directly to their mailbox at the end of the day.

• The app doesn’t merely identify emotions; it takes a proactive approach by providing actionable insights and well-being recommendations. This empowers users to navigate their professional landscapes with resilience and emotional intelligence.

• The AI Counselor represents a paradigm shift in promoting mental health in the workplace. It serves as a virtual counselor that is not only empathetic but also equipped with the precision of artificial intelligence, fundamentally reshaping how we approach emotional well-being in our professional lives.

3. Prerequisite

Setting up Raspberrypi and USB camera

• Install Raspbian 9 or above
• Refer here for instruction to setup the hardware

At the end of this setup you should have hardware that is supported for IoT Edge application

• Plug USB camera to Raspberrypi

Azure IoT Hub

• Create Azure IoT Hub, Free plan should be sufficient for running this project
• Follow the steps here https://learn.microsoft.com/en-us/azure/iot-hub/iot-hub-create-through-portal
• In the left pane, select Devices from the menu, then select Add Device.
• On the Create a device page, provide the following information:
• Create a descriptive Device ID, for example my-edge-device-1 (all lowercase). Copy this Device ID, as you’ll use it later.
• Check the IoT Edge Device checkbox.
• Select Symmetric key as the authentication type.
• Use the default settings to auto-generate authentication keys, which connect the new device to your hub.
• Select Save.

Install IoT Edge

Now we need to prepare the linux virtual machine for IoT Edge. Refer to this article https://learn.microsoft.com/en-us/azure/iot-edge/how-to-provision-single-device-linux-symmetric?view=iotedge-1.4&tabs=azure-portal%2Cdebian

Azure Custom Vision

• Follow steps mentioned here to create a new project in Custom Vision (https://learn.microsoft.com/en-us/azure/iot-edge/tutorial-deploy-custom-vision?view=iotedge-1.4)
• We downloaded face emotions dataset from hugging face and classified emotions based on the tags (https://huggingface.co/datasets)
• Next we trained them in CustomVision
• Finally , we exported the model by selecting Dockerfile for the platform and Linux for the version

IoT Edge Solution

• Install VS code along with Azure IoT Edge and Azure IoT Hub extensions, for creating IoT Edge solution
• Follow this article https://learn.microsoft.com/en-us/azure/iot-edge/tutorial-deploy-custom-vision?view=iotedge-1.4

.NET Aspire

.NET Aspire is an opinionated, cloud ready stack for building observable, production ready, distributed applications. .NET Aspire is delivered through a collection of NuGet packages that handle specific cloud-native concerns. Cloud-native apps often consist of small, interconnected pieces or microservices rather than a single, monolithic code base. Cloud-native apps generally consume a large number of services, such as databases, messaging, and caching.

• Install .NET 8.0 SDK (https://dotnet.microsoft.com/en-us/download/dotnet/8.0)
• Use Visual studio installer

4. Architecture

Components used in architecture:

• Azure IoT Edge This device-focused runtime facilitates the deployment, operation, and monitoring of containerized Linux workloads. It is an integral part of Azure IoT Hub, allowing the scaling and management of IoT solutions from the cloud. The AI Counselor system consists of two modules: the CameraCaptureModule and the FaceEmotionRecognizerModule. The former captures images from a USB camera and sends them to the latter via HTTP. The FaceEmotionRecognizerModule classifies images based on emotions using Azure Custom Vision, and the results are then sent back to the CameraCaptureModule.

• Azure IoT Hub This component enables reliable and secure communication between millions of IoT devices and a cloud-hosted solution backend. IoT Edge modules are registered in Azure IoT Hub, and data is routed to Azure Service Bus.

• Azure Service Bus As a fully managed enterprise message broker, Azure Service Bus is used to decouple applications and services from each other. In the AI Counselor solution, all telemetries from IoT Edge devices are routed to Azure Service Bus, which initiates the ingestion service by listening to the queue.

• Azure Container Registry This service allows the building, storing, and management of container images and artifacts in a private registry for various container deployments. The IoT Edge Agent pulls the images of the two Edge modules from Azure Container Registry.

• Azure Container Apps This serverless platform minimizes infrastructure concerns and reduces costs for running containerized applications. Backend services and a frontend app in the AI Counselor solution are hosted within the Azure Container Apps environment.

• Azure OpenAI he Azure OpenAI Service provides REST API access to powerful language models, including GPT-4 and GPT-3.5-Turbo. This service generates a counseling summary at the end of the day, delivered directly to the user’s mailbox.

• Azure Cosmos DB As a fully managed NoSQL and relational database, Azure Cosmos DB is used to store telemetries from Azure Service Bus via the Counselor service. It offers single-digit millisecond response times, automatic scalability, and guaranteed speed at any scale. Emotion-related reasons are also stored in Azure Cosmos DB.

• Twilio The Twilio SMS API is employed to send messages to users if attention is required.

• Postmark This email delivery service is used to send mail summaries. It is chosen over Sendgrid from Twilio due to account block concerns.

• Azure Application Insights Azure Application Insights is a crucial component for monitoring and optimizing application performance. It serves as an application performance management (APM) service, offering a comprehensive view of an application’s health and user interactions.

• Azure Log Analytics Workspace Azure Log Analytics Workspace acts as a centralized repository and analytics engine for log data generated by Azure resources and applications. It facilitates effective log data collection, analysis, and visualization for monitoring and troubleshooting purposes.

• Azure Monitor Azure Monitor serves as a comprehensive monitoring solution for Azure resources, applications, and infrastructure. It unifies telemetry data collection, analysis, and action to ensure the availability, performance, and reliability of applications.

5. Data Flow

1. The CameraCapture Edge Module enables continuous capture from a USB camera at a rate of one frame per second.
2. The captured images are subsequently transmitted to the FaceEmotionRecognizer Edge Module through an HTTP call.
3. The FaceEmotionRecognizer module returns the probability of facial emotions to the CameraCapture module.
4. The CameraCapture module is responsible for aggregating emotions obtained from the FaceEmotionRecognizer module.
5. Once a specified threshold is reached, the aggregated telemetry data is then pushed to the Azure IoT Hub.
6. The data, through route endpoints are subsequently pushed from Azure IoT Hub to Azure Service Bus.
7. The ingestion Service listens for messages in the Azure Service Bus and, upon their placement, initiates an HTTP call to the Counselor service.
8. The Counselor service stores the telemetry data into Azure Cosmos DB.
9. Simultaneously, if a specific emotion persists for more than n seconds, the ingestion service, following the defined business logic, initiates an HTTP call to the Notification service. Subsequently, the Notification service sends an SMS to the user, prompting them to provide a reason in the Frontend app.
10. The ingestion service is scheduled to run daily at 12 AM, triggering the Counselor Service. The Counselor Service then initiates an HTTP call to the Azure OpenAI service, including the telemetry data collected throughout the day.
11. The Azure OpenAI service furnishes a counseling summary, and this information is subsequently transmitted to the ingestion service.
12. With the counseling summary in place, the ingestion service initiates an HTTP call to the Notification service, which, in turn, dispatches an email template containing the summary to the user.

6. AI Counselor Frontend

.NET Aspire Dashboard

Landing Page

Track Emotions Page

7. AI Counseling SMS Notification

Users receive SMS notifications only when their attention is necessary. Upon receiving the SMS, users can visit the AI Counselor portal to provide reasons for emotions that persisted for more than n seconds.

The WorkerService manages the logic and initiates an HTTP call to the Notification Service when the specified condition is met. Subsequently, the Notification Service, in turn, places a call to the Twilio SMS API to send a message to the user.

8. AI Counseling Email summary

The counseling summary is structured into four key elements:

1. General Emotion Overview: The summary initiates by offering a broad overview of the emotions experienced throughout the day.
2. Contextual Explanations: Following the emotion overview, the summary provides possible explanations, taking into account the context and reasons shared by the user through our app.
3. Action Points for Overcoming Challenges: Subsequently, actionable points are presented to assist users in navigating and overcoming any challenges identified during the emotional analysis.
4. Motivational Quote Conclusion: The counseling summary concludes with a carefully selected motivational quote, personalized to the user’s emotions. This aims to provide a positive and uplifting note, wrapping up the email.

8.1 Prompt used

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

var chatCompletionsOptions = new ChatCompletionsOptions()
{
    DeploymentName = _deploymentId,
    Messages =
{
    new ChatMessage(ChatRole.System, "System"),
    new ChatMessage(ChatRole.System, $"Act as an expert counselor and provide me counseling summary based on the series of emotions that I went through this day, where timestamp start and end is in unixtimestamp format." +
    $" If you want to specify time period in this glimpse of summary do specify time period in human readable date format (not in timestamp). Provide summary along with some possible explanations to it, if reason is not explicitly provided. " +
    $"Feel free to ignore emotions that are not to be concerned about, like neutral emotions. Also provide the summary in a email friendly format with <hr> <b> <br/> tags wherever relevant. " +
    $"If needed, provide me some action points for me to follow with <ul> <li> tags. End the summary with a famous quote that will be relevant for my summary, use a different style and border for quotes." +
    $"Here is the emotion: \\n"),
    new ChatMessage(ChatRole.User, emotionsTimeSeries),
},
    MaxTokens = _maxTokens,
    Temperature = _temperature,
};

9. Analyzing summary

• Telemetry data collected from IoT Edge devices is stored in Azure Cosmos DB, along with reasons captured from the Frontend app, as illustrated in the image below:

• When a user provides the reason for their fear, such as “I was asked to present a topic in front of all team members,” the AI Counselor incorporates this information into its summary. The actionable advice provided is: “Prepare well in advance for the presentation to overcome the fear of public speaking.”

• Additionally, the AI Counselor offers motivation to the user by acknowledging that feeling fear when presenting in front of a large group is natural. It emphasizes the commonality of this experience and encourages the user to practice and prepare thoroughly in order to overcome this fear.

• The summary concludes with an inspirational quote: “Happiness is not something ready-made. It comes from your own actions.” - Dalai Lama

10. Deployment of Aspire application

• Navigate to your AppHost folder.
• Execute the following command to initialize your app:

1

azd init

• Choose “Use code in the current directory” to scan the app code in the current directory.
• After detecting the services, azd will generate the necessary files for hosting in Azure Container Apps.
• Confirm the generated options.
• Choose which services to expose; in this case, we exposed the “aicounselorfrontend” service.
• Provide the environment name as “aicounselor.”
• The tool will generate the required files to run the app on Azure.
• To provision and deploy, enter the following command:

1

azd up

• Select the subscription and location.
• The deployment process will commence, deploying all the services into the Azure Container App environment.

After provisioning the resources for further updates in code try below command:

1

azd deploy

11. Resources in Azure

Above deployment will create all of below services in Azure under a resource group “rg-aicounselor”

12. Source code

Github - AI Counselor

Sponsor