Context Sharing in Activity Protocol

[sveluvil]

The Activity Protocol must support seamless sharing of context information between users and agents to improve understanding of ongoing conversations. This enables more relevant, personalized, and accurate responses, especially in multi-modal interactions. By embedding context directly in the activity message, agents can leverage contextual information such as images, custom objects, and audio/video-related context.

The Activity Protocol schema will be enhanced to include a context as entity, enabling both client and agent to exchange relevant contextual data within a conversation. This proposal is only for sharing images as context.

---

Context Object Properties

The context entity in an activity message can contain the following fields.
Images for context can be shared in two ways. Users can upload an image as an attachment, or the client application can take a screenshot of the page the user is currently on. In both cases, the images can be provided either as a URL or as base64-encoded data, which can then be passed in the images list.

• Type: "context"
• Properties
  • screenContext: An object containing information about the screen's current state, including:
    • surroundingText: Text surrounding the cursor position. Relevant for parsing out the text for context (e.g., in a code editor or document viewer).
    • images: A list of base64-encoded image or a URL that points to an image which can be used as context.

---

Handling Context Information

Context Messages

• If the context contains no elements, it should be ignored.
• Only one context entity can be included in a single activity message. If multiple context objects are present, they should be ignored by the recipient.
• If the context entity contains any syntactic errors, it should be ignored by the recipient to prevent processing invalid data.

Throttling

• Outgoing activities that contain context should be throttled to a configurable interval to prevent overloading the recipient system.

Unsupported Channels

• At present, context sharing is not supported in any channels.

---

Example Usage

  // Sending context information as part of the activity message
  {
    "type": "message",
    "text": "What pizzas are available for $150?",
    "entities": [
      {
        "type": "context",
        "screenContext": {
          "surroundingText": "Details of the product on the page",
          "images": ["iVBORw0KGgoAAAANSUhEUgAAAAUA...", "https://example.com/product-page"] // a list of base64-encoded image or URL that should be used as context.
        }
      }
    ]
  }

[gurubhg]

Hi @sveluvil, Thanks for putting together the initial proposal for context sharing.

After evaluating the current Activity Protocol spec, the multimodal streaming proposal (#416), and the earlier context‑sharing draft, here is detailed feedback and suggested improvements.

1. Use value + valueType for Primary Context (instead of entities)

The proposal currently embeds the entire context object inside an entity, but entities are intended for metadata and may be ignored by receivers. Context is carried in:

• value - structured context object
• valueType - MIME-style identfier

{
  "type": "event",
  "name": "context.share",
  "valueType": "application/vnd.microsoft.activity.context.url+json",
  "value": {
    "url": "https://contoso.com/product/42",
    "title": "Product 42"
  }
}

This aligns the model with the multimodal pattern in #416.

2. Context Sharing Pattern (non-streaming)

event -> context.share

{
  "type": "event",
  "name": "context.share",
  "valueType": "application/vnd.microsoft.activity.context.selection+json",
  "value": {
    "text": "selected text",
    "offset": 182,
    "length": 41
  }
}

context -> message

{
  "type": "message",
  "text": "Here’s the screen section.",
  "valueType": "application/vnd.microsoft.activity.context.screen+json",
  "value": {
    "contentUrl": "data:image/png;base64,...",
    "region": { "x": 24, "y": 160, "w": 900, "h": 280 }
  }
}

3. Consent/Ack via command + commandResult

Rather than embedding consent flow as events, it is cleaner and more consistent with existing lifecycle patterns to use command and commandResult.

// Context request
{
  "type": "command",
  "name": "context.request",
  "value": { "types": ["screen"], "frequency": "continuous" }
}

// User/Client Accept
{
  "type": "commandResult",
  "replyToId": "cmd1",
  "value": { "status": "accepted" }
}

4. Reuse Streaming Semantics from #416 - Media.Start/Chunk/End

For continuous context sharing (screen, cursor, live image feed, etc.), we should reuse the exact streaming event model defined in #416. This avoids creating new streaming semantics.

{
  "type": "event",
  "name": "Media.Start",
  "valueType": "application/vnd.microsoft.activity.mediastart+json",
  "value": {
    "mediaType": "image",
    "contentType": "image/png",
    "profile": "context/screen"
  },
  "entities": [
    {
      "type": "streamInfo",
      "streamId": "abc123",
      "streamType": "context",
      "streamState": "Streaming",
      "streamSequence": 0
    }
  ]
}

{
  "type": "event",
  "name": "Media.Chunk",
  "valueType": "application/vnd.microsoft.activity.context.screen+json",
  "value": {
    "contentUrl": "data:image/png;base64,..."
  },
  "entities": [
    {
      "type": "streamInfo",
      "streamId": "abc123",
      "streamType": "context",
      "streamState": "Streaming",
      "streamSequence": 1
    }
  ]
}

{
  "type": "event",
  "name": "Media.End",
  "valueType": "application/vnd.microsoft.activity.mediaend+json",
  "entities": [
    {
      "type": "streamInfo",
      "streamId": "abc123",
      "streamType": "context",
      "streamState": "Final",
      "streamSequence": 2
    }
  ]
}

5. Introduce Typed valueType Profiles

To keep the proposal clean and predictable across channels, please define the following explicit profiles:

• application/vnd.microsoft.activity.context.url+json
• application/vnd.microsoft.activity.context.selection+json
• application/vnd.microsoft.activity.context.screen+json
• application/vnd.microsoft.activity.context.media+json
• application/vnd.microsoft.activity.context+json (generic fallback)

These provide predictable, channel‑safe envelopes without schema changes.

Overall summary:

1. Move context out of entities -> into value + valueType
2. Use unified carriers:
   • context.share
   • message
   • context.request/commandResult
3. Reuse Media.* for streaming
4. Introduce typed context valueTypes

[sveluvil]

Hi @gurubhg,

Thanks for the detailed review comments. I went through the review comments and I have a few queries.

The scope of the current work item is limited to sharing a screenshot of the web page the user is currently in as context. Therefore, we will take up point 4 (streaming context) in the future and restrict the current proposal to sharing the screenshot as context. Additionally, I have concerns about token conception with streaming context.

Using Consent/Ack via command + commandResult for taking consent
We already have implemented consent for saving the transcript using events. Hence, moving to command + commandResult will introduce backward compatibility issues. Additionally, supporting both approaches will be messy. Hence, I recommend we follow the current approach of using events for taking consent.

Context Sharing Pattern (non-streaming)
I have a query related to usage of valueType and value for sharing context.
If we ever have to send a message with multiple valueTypes, it will not be possible with current approach. How will we handle it?

From the #416 proposal and our discussions, I understand that valueTypes are meant to be used with events. Hence, I am doubtful about using valueTypes with the activity type 'message'. Therefore, I believe sending context in entities is better. It is also extensible for future use cases, except for streaming context (point 4).

So, if we are not planning for streaming context in the near future, I recommend we go with the entities approach. Otherwise, we will have to rethink this.
When we plan to implement streaming context, how about taking a hybrid approach: streaming context via events + valueTypes, and normal context sharing via entities?
Please share your thoughts.