gateway/cookbook/monitoring-agents/Llama_Agents_with_Telemetry.ipynb

Monitoring Llama Agents with Portkey

llama-agents is an agent framework for building, iterating, and productionizing multi-agent systems, including multi-agent communication, distributed tool execution, human-in-the-loop, and more!

One of the major challenges of building AI agents is the lack of visibility into key performance metrics such as:

• Number of API Requests: How many requests are being made to LLMs.
• Token Usage & Costs: How many tokens are being consumed and the cost for each agent run.
• Latency: How long is the agent taking to complete a set of tasks.

Portkey is an open source AI Gateway that helps you manage access to 250+ LLMs through a unified API while providing visibility into

✅ cost,
✅ performance, and
✅ accuracy metrics.

This notebook demonstrates how you can bring visibility and flexbility to Llama Agents using Portkey's AI Gateway.

In [ ]:

!pip install -qU llama-agents llama-index portkey-ai

In [ ]:

import nest_asyncio
nest_asyncio.apply()

In [ ]:

import os
from google.colab import userdata

os.environ["OPENAI_API_KEY"] = userdata.get('OPENAI_API_KEY')

Llama Agents Setup

Quickstart

Since Portkey is fully compatible with the OpenAI signature, you can connect to the Portkey AI Gateway through the OpenAI client.

• Set the base_url as PORTKEY_GATEWAY_URL
• Add default_headers to consume the headers needed by Portkey using the createHeaders helper method.

In [ ]:

from llama_index.llms.openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders
from google.colab import userdata


gpt_4o_config = {
    "provider": "openai",
    "api_key": userdata.get('OPENAI_API_KEY'),
    "override_params": { "model":"gpt-4o" }
}

gpt_4o = OpenAI(
    api_base=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(
        api_key=userdata.get('PORTKEY_API_KEY'),
        config=gpt_4o_config
    )
)

llama3_groq_config = {
    "provider": "groq",
    "api_key": userdata.get('GROQ_API_KEY'),
    "override_params": {"model":"llama3-70b-8192"}
}

llama3 = OpenAI(
    api_base=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(
        api_key=userdata.get('PORTKEY_API_KEY'),
        config=llama3_groq_config
    )
)

Creating Agents

In [ ]:

from llama_agents import (
    AgentService,
    AgentOrchestrator,
    ControlPlaneServer,
    SimpleMessageQueue,
)

from llama_index.core.agent import ReActAgent
from llama_index.core.tools import FunctionTool
from llama_index.llms.openai import OpenAI


# create an agent
def get_the_secret_fact() -> str:
    """Returns the secret fact."""
    return "The secret fact is: A baby llama is called a 'Cria'."


tool = FunctionTool.from_defaults(fn=get_the_secret_fact)

agent1 = ReActAgent.from_tools([tool], llm=llama3)
agent2 = ReActAgent.from_tools([], llm=llama3)

# create our multi-agent framework components
message_queue = SimpleMessageQueue()
control_plane = ControlPlaneServer(
    message_queue=message_queue,
    orchestrator=AgentOrchestrator(llm=gpt_4o),
)

agent_server_1 = AgentService(
    agent=agent1,
    message_queue=message_queue,
    description="Useful for getting the secret fact.",
    service_name="secret_fact_agent"
)

agent_server_2 = AgentService(
    agent=agent2,
    message_queue=message_queue,
    description="Useful for getting random dumb facts.",
    service_name="dumb_fact_agent"
)

Launching Agents

In [ ]:

from llama_agents import LocalLauncher

# launch it
launcher = LocalLauncher(
    [agent_server_1, agent_server_2],
    control_plane,
    message_queue,
)
result = launcher.launch_single("What is the secret fact?")

print(f"Result: {result}")

INFO:llama_agents.message_queues.simple - Consumer AgentService-86ae75c9-9d5f-4ddf-88ea-c67d79506506: secret_fact_agent has been registered.
INFO:llama_agents.message_queues.simple - Consumer AgentService-d2c4ba62-3a1f-4f08-8fdc-10c49d7f5140: dumb_fact_agent has been registered.
INFO:llama_agents.message_queues.simple - Consumer 230c00bf-9ff1-4770-a92b-84c0cfb7e290: human has been registered.
INFO:llama_agents.message_queues.simple - Consumer ControlPlaneServer-2578d1b4-9576-4f4e-8107-c8a6829ca7a3: control_plane has been registered.
INFO:llama_agents.services.agent - secret_fact_agent launch_local
INFO:llama_agents.services.agent - dumb_fact_agent launch_local
INFO:llama_agents.message_queues.base - Publishing message to 'control_plane' with action 'new_task'
INFO:llama_agents.message_queues.simple - Launching message queue locally
INFO:llama_agents.message_queues.base - Publishing message to 'secret_fact_agent' with action 'new_task'
INFO:llama_agents.message_queues.simple - Successfully published message 'control_plane' to consumer.
INFO:llama_agents.message_queues.simple - Successfully published message 'secret_fact_agent' to consumer.
INFO:llama_agents.message_queues.base - Publishing message to 'control_plane' with action 'completed_task'
INFO:llama_agents.message_queues.base - Publishing message to 'human' with action 'completed_task'
INFO:llama_agents.message_queues.simple - Successfully published message 'control_plane' to consumer.
INFO:llama_agents.message_queues.simple - Successfully published message 'human' to consumer.

Result: A baby llama is called a 'Cria'.

Telemetry with Portkey

Part 1: Logs

For the above agent run, we have initiated two agents utilizing the GPT-4o and Llama 3 models. The detailed logs for this request are available on the Portkey dashboard, providing comprehensive insights into the activities of each agent.

Sample Logs

Part 2: Analytics

The insights from this request can be viewed on the Portkey dashboard. We are able to infer tokens used and number of API requests. You can also review many other metrics from the dashboards like cost, latency, etc.

Sample Dashboard

In [ ]: