Cuata in Action - Teams Meeting Assistant and Browser Automation Workflows

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

1. Meeting Notifier triggers (scheduled meeting about to start)
2. OpenCV checks if your face is visible on webcam
3. If you're NOT present:
   → Cuata opens Teams calendar
   → Searches for the meeting by title
   → Clicks "Join"
   → Mutes microphone
   → Starts transcription
   → Takes screenshots at intervals
4. When you return or meeting ends:
   → Stops transcription
   → Sends you a summary with key screenshots

1
2
3
4
5
6
7
8

1. Teams Agent detects you're in an active meeting
2. OpenCV checks if your face is visible on webcam
3. If you've LEFT:
   → Starts listening and transcribing
   → Takes screenshots of slides/shared content
4. When you return or meeting ends:
   → Stops transcription
   → Sends you a summary of what you missed

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

chatHistory.AddSystemMessage(
"""
You are a helpful assistant that can help with Microsoft Teams related tasks.
You can help listening to the meetings while the user is away.

Goal: Always follow the master strategy to pick the right strategy to follow.

**Master strategy:**

1. Capture the screen and verify if the meeting is open.
2. If the meeting is open, go to strategy 2.
3. If the meeting is not open, go to strategy 1.
""");

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

**Strategy 1:**

1. Open Microsoft Teams.
2. Search for "calendar" and click on it.
3. Take a screenshot and verify if the calendar is open.
4. Search for the meeting with the title that happens today and click on it.
5. Take a screenshot and verify if the popup opens with Join button.
6. Search for "join" button and click on it.
7. Take a screenshot and verify if "Join now" button appears.
8. In the meeting popup, search for "Join now" button and click on it.
9. Take a screenshot and verify if you are inside the meeting.
10. Maximize the meeting window by pressing "Win" + "Up" key.
11. Search for "Mic" button and click on it to mute.
12. Take a screenshot and verify if you are muted.
13. Search for "more" button and click on it.
14. Take a screenshot and verify if more options menu is open.
15. Search for "Record and transcribe" button.
16. Take a screenshot and verify if popup with "Start transcription" opens.
17. Click on "Start transcription" button.
18. If it asks for permission, click on "Confirm" button.
19. Take a screenshot and verify if transcription started.
20. Start listening to the meeting and take notes.
21. Once meeting is over, search for "Leave" button and click on it.
22. Take a screenshot and verify if you are out of the meeting.
23. Share the notes with the user.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

**Strategy 2:**

1. Verify if transcription is already started (check right panel for transcript).
2. If transcription NOT started:
   → Search for "more" button and click on it.
   → Take a screenshot and verify if more options menu is open.
   → Search for "Record and transcribe" button.
   → Take a screenshot and verify if popup with "Start transcription" opens.
   → Click on "Start transcription" button.
   → If it asks for permission, click on "Confirm" button.
   → Take a screenshot and verify if transcription started.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

// Simplified OpenCV face detection logic
public bool IsUserPresent()
{
    using var frame = _videoCapture.QueryFrame();
    if (frame == null) return false;
    
    // Use Haar Cascade classifier for face detection
    var faces = _faceCascade.DetectMultiScale(
        frame.Convert<Gray, byte>(),
        scaleFactor: 1.1,
        minNeighbors: 3
    );
    
    bool isPresent = faces.Length > 0;
    
    // Update global state
    CuataState.Instance.IsPresent = isPresent;
    
    return isPresent;
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

CuataState.Instance.OnPresenceChanged += isPresent =>
{
    if (isPresent)
    {
        Console.WriteLine("👤 User is present. Stopping the app.");
        _recognizer.StopContinuousRecognitionAsync().Wait();
        return;
    }
    else
    {
        Console.WriteLine("🚶 User has left. Starting Teams Agent...");
        // Trigger Teams Agent workflow
    }
};

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

// Start continuous recognition
await _recognizer.StartContinuousRecognitionAsync();

// Handle recognized speech
_recognizer.Recognized += (sender, e) =>
{
    if (e.Result.Reason == ResultReason.RecognizedSpeech)
    {
        Console.WriteLine($"🗣️ Recognized: {e.Result.Text}");
        
        // Store transcript
        CuataState.Instance.AddTranscript(new TranscriptEntry
        {
            Timestamp = DateTime.UtcNow,
            Speaker = "Unknown", // Teams doesn't provide speaker ID
            Text = e.Result.Text
        });
    }
};

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

// Periodic screenshot capture
var screenshotTimer = new Timer(async _ =>
{
    var screenshot = ScreenCapture.CaptureScreen();
    var filename = $"meeting-{DateTime.UtcNow:yyyyMMddHHmmss}.png";
    
    // Save to Azure Blob Storage
    await _blobClient.UploadAsync(filename, screenshot);
    
    // Ask Azure OpenAI to describe the screenshot
    var description = await _chatService.GetChatMessageContentAsync(
        new ChatHistory
        {
            new ChatMessageContentItemCollection
            {
                new TextContent("Describe this meeting slide briefly."),
                new ImageContent(screenshot, "image/png")
            }
        }
    );
    
    CuataState.Instance.AddScreenshot(new ScreenshotEntry
    {
        Timestamp = DateTime.UtcNow,
        BlobUrl = filename,
        Description = description.Content
    });
    
}, null, TimeSpan.Zero, TimeSpan.FromSeconds(30));

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

// Orchestrator function
[Function(nameof(SummarizeMeetingOrchestrator))]
public async Task<MeetingSummary> SummarizeMeetingOrchestrator(
    [OrchestrationTrigger] TaskOrchestrationContext context)
{
    var meetingData = context.GetInput<MeetingData>();
    
    // Activity 1: Consolidate transcript
    var transcriptSummary = await context.CallActivityAsync<string>(
        nameof(SummarizeTranscriptActivity),
        meetingData.Transcripts
    );
    
    // Activity 2: Analyze screenshots
    var screenshotAnalysis = await context.CallActivityAsync<List<ImportantSlide>>(
        nameof(AnalyzeScreenshotsActivity),
        meetingData.Screenshots
    );
    
    // Activity 3: Generate final summary
    var finalSummary = await context.CallActivityAsync<string>(
        nameof(GenerateFinalSummaryActivity),
        new { transcriptSummary, screenshotAnalysis }
    );
    
    // Activity 4: Send notification to user
    await context.CallActivityAsync(
        nameof(SendNotificationActivity),
        new { summary = finalSummary, slides = screenshotAnalysis }
    );
    
    return new MeetingSummary
    {
        FinalSummary = finalSummary,
        ImportantImageUrls = screenshotAnalysis.Select(s => s.BlobUrl).ToList()
    };
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

chatHistory.AddSystemMessage(
"""
You are a powerful desktop automation assistant capable of observing the screen 
visually, reasoning about UI elements, and performing system-level actions.

You work like a human: you see the screen, think, and interact using the mouse and keyboard.

🧠 Reasoning Strategy:

1. Act like a human: observe the screen, think, and act.
2. Use screenshots to verify actions and analyze UI elements.
3. Use plugins to perform actions based on observations.
4. Always reason based on the current screen and context.
5. If you are not sure about the next action, ask for clarification.

Thought Process:

Split the user goal into multiple steps, each with a verification step.

Example 1: Search for live mint related articles
[
    {"Verify if you are already at the search result page"},
    {"If not, open Google Chrome"},
    {"Type the article to be found along with livemint in the search box"},
    {"Search for 'live mint' in the page using locate plugin and click on it"},
    {"Take a screenshot to verify if we done the previous action correctly"},
    {"Scroll through the articles and find any related to live mint"}
]
""");

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

// AI Decision: "Open Google Chrome and search for stock market news"
await ChromePlugin.OpenUrl("https://www.google.com");

// Wait for page load
await Task.Delay(2000);

// Locate search box and type query
await LocatePlugin.LocateElementInScreenshot("Search");
await MousePlugin.LeftClick();
await KeyboardPlugin.TypeText("Stock Market news");
await KeyboardPlugin.PressEnter();

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

// AI Decision: "Take a screenshot and verify we're at search results"
var validation = await ScreenshotPlugin.TakeScreenshotAndVerify(
    "Are we at Google search results for 'Stock Market news'?"
);

if (!validation.Contains("yes"))
{
    // Retry logic
    Console.WriteLine("❌ Search failed, retrying...");
}

1
2
3
4
5
6

// AI Decision: "Scroll and locate 'Motherson Sumi' in search results"
await MousePlugin.Scroll(-3); // Scroll down
await Task.Delay(1000);

await LocatePlugin.LocateElementInScreenshot("Motherson Sumi");
await MousePlugin.LeftClick();

1
2
3
4

// AI Decision: "The article is open, now summarize it"
var summary = await SummarizePlugin.SummarizePage(isFullScreen: true);

Console.WriteLine($"📄 Summary: {summary}");

1
2
3
4
5
6
7

// AI Decision: "Open Word and write the summary"
await WordPlugin.OpenWord();
await Task.Delay(2000);

await WordPlugin.WriteWord(summary);

Console.WriteLine("✅ Summary written to Word!");

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

// Start continuous speech recognition
await _recognizer.StartContinuousRecognitionAsync();

_recognizer.Recognized += async (sender, e) =>
{
    if (e.Result.Reason == ResultReason.RecognizedSpeech)
    {
        var userCommand = e.Result.Text;
        Console.WriteLine($"🎤 You said: {userCommand}");
        
        if (userCommand.ToLower().Contains("quit"))
        {
            await _recognizer.StopContinuousRecognitionAsync();
            return;
        }
        
        // Add user command to chat history
        chatHistory.AddUserMessage(userCommand);
        
        // Let Semantic Kernel orchestrate the workflow
        var response = await _chatService.GetChatMessageContentAsync(
            chatHistory,
            executionSettings: new OpenAIPromptExecutionSettings
            {
                ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions,
                Temperature = 0.2
            },
            kernel: _kernel
        );
        
        chatHistory.Add(response);
        
        // Speak the response
        await SpeakAsync(response.Content);
    }
};

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

[KernelFunction, Description("Summarizes the current screen content")]
public async Task<string> SummarizePage(
    [Description("Is the content full screen?")] bool isFullScreen)
{
    // Capture screen
    ScreenCapture sc = new ScreenCapture();
    Image img = sc.CaptureScreen();
    
    using var bmp = new Bitmap(img);
    using var ms = new MemoryStream();
    bmp.Save(ms, ImageFormat.Png);
    ms.Position = 0;
    
    var imageData = new ReadOnlyMemory<byte>(ms.ToArray());

    // Send to Azure OpenAI Vision
    var chatHistory = new ChatHistory();
    chatHistory.AddUserMessage(new ChatMessageContentItemCollection
    {
        new TextContent(isFullScreen 
            ? "Summarize everything you see on this screen in detail."
            : "Summarize the main content on this screen."),
        new ImageContent(imageData, "image/png")
    });

    var response = await _chatService.GetChatMessageContentAsync(
        chatHistory,
        executionSettings: new OpenAIPromptExecutionSettings 
        { 
            MaxTokens = 1000,
            Temperature = 0.3
        }
    );

    Console.WriteLine($"📄 Summary: {response.Content}");
    return response.Content ?? "Unable to summarize.";
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

[KernelFunction, Description("Writes summary to Microsoft Word")]
public string WriteWord([Description("The summary text")] string summary)
{
    try
    {
        var wordApp = new Microsoft.Office.Interop.Word.Application();
        wordApp.Visible = true;
        
        var doc = wordApp.Documents.Add();
        var range = doc.Range();
        
        range.Text = summary;
        range.Font.Name = "Calibri";
        range.Font.Size = 12;
        
        Console.WriteLine("📄 Summary written to Word!");
        return "Summary written to Word document.";
    }
    catch (Exception ex)
    {
        Console.WriteLine($"❌ Failed to write to Word: {ex.Message}");
        return $"Failed: {ex.Message}";
    }
}

1
2
3
4
5
6
7
8

✅ Opens Microsoft Teams
✅ Navigates to Calendar
✅ Searches for "10 AM Project Sync"
✅ Clicks "Join"
✅ Clicks "Join now"
✅ Mutes microphone
✅ Starts transcription
✅ Takes screenshot every 30 seconds

1
2
3
4

✅ Stops transcription
✅ Sends transcript + screenshots to Durable Functions
✅ Generates summary: "Discussion about Q4 roadmap, Sarah presented 3 slides on feature priorities"
✅ Desktop notification appears with summary and slide links

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

🌐 Opens Google Chrome
⌨️ Types "live mint stock market today"
🔍 Presses Enter
📸 Takes screenshot to verify search results
📍 Locates "Live Mint" link and clicks
📸 Takes screenshot to verify page loaded
📜 Scrolls through articles
📍 Locates "Stock Market Today" article and clicks
📸 Takes screenshot of article
📝 Summarizes the article using Azure OpenAI Vision
📄 Opens Microsoft Word
⌨️ Writes the summary into Word document
✅ Speaks: "Summary written to Word"

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

public class CuataState
{
    public static CuataState Instance { get; } = new CuataState();
    
    public bool IsPresent { get; set; }
    public List<TranscriptEntry> Transcripts { get; } = new();
    public List<ScreenshotEntry> Screenshots { get; } = new();
    
    public event Action<bool>? OnPresenceChanged;
    public event Action<MeetingSummary>? OnConsolidatedSummaryChanged;
    
    public void AddTranscript(TranscriptEntry entry) => Transcripts.Add(entry);
    public void AddScreenshot(ScreenshotEntry entry) => Screenshots.Add(entry);
}