gateway/cookbook/monitoring-agents/CrewAI_with_Telemetry.ipynb

Monitoring CrewAI agents with Portkey

CrewAI is an AI agent framework that lets you create and orchestrate role-playing, autonomous AI agents. You can can create agents, assign them goals and have them create and complete tasks to achieve these goals.

One of the key challenges of building AI agents is the lack of visibility into key performance metrics such as:

• Cost Analysis: How much does it cost to run CrewAI agents for your project.
• Token Usage: How many tokens are being consumed for each agent run.
• Latency: How long is the agent taking to complete a set of tasks.

Portkey is an open source AI Gateway that helps you manage access to 250+ LLMs through a unified API while providing visibility into

✅ cost
✅ performance
✅ accuracy metrics

This notebook demonstrates how you can bring visibility and flexbility to CrewAI agents using Portkey's AI Gateway.

Setup

First, let's install both CrewAI and Portkey AI SDKs in our notebook.

In [ ]:

!pip install -qU crewai portkey-ai

To monitor your agents, connect the LLMs to the Portkey AI Gateway through the ChatOpenAI interface.

• Set the base_url as PORTKEY_GATEWAY_URL
• Add default_headers needed by Portkey using the createHeaders helper method.

To start, get your Portkey API key by signing up here. (Click the profile icon on the bottom left, then click on "Copy API Key")

In [4]:

from crewai import LLM
from portkey_ai import createHeaders, PORTKEY_GATEWAY_URL


gpt_llm = LLM(
    model="gpt-4o",
    base_url=PORTKEY_GATEWAY_URL,
    api_key="dummy", # We are using Virtual key
    extra_headers=createHeaders(
       api_key="YOUR_PORTKEY_API_KEY",
       virtual_key="YOUR_VIRTUAL_KEY", # Enter your OpenAI Virtual key from Portkey
    )
)


anthropic_llm = LLM(
    model="claude-3-5-sonnet-latest",
    base_url=PORTKEY_GATEWAY_URL,
    api_key="dummy", # We are using Virtual key
    extra_headers=createHeaders(
        api_key="YOUR_PORTKEY_API_KEY",
        virtual_key="YOUR_VIRTUAL_KEY", # Enter your Anthroipc Virtual key from Portkey
        config="YOUR_PORTKEY_CONFIG_ID", # All your model parmaeters and routing strategy
        trace_id="llm2"
    )
)

Simple CrewAI Agent

In [5]:

from crewai import Agent, Task, Crew, Process

# Define your agents with roles and goals

coder = Agent(
    role='Software develoepr',
    goal='Write clear - concise code on demand',
    backstory='An expert coder with a keen eye for software trends.',
    llm = gpt_llm
)


# Create tasks for your agents
task1 = Task(
    description="Define the HTML for making a simple website with heading- Hello World! Portkey is working! .",
    expected_output="A clear and concise HTML code",
    agent=coder
)

# Instantiate your crew with a sequential process
crew = Crew(
    agents=[coder],
    tasks=[task1],
)

# Get your crew to work!
result = crew.kickoff()
print("######################")
print(result)

WARNING:opentelemetry.trace:Overriding of current TracerProvider is not allowed

######################
```html
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Simple Website</title>
</head>
<body>
    <h1>Hello World! Portkey is working!</h1>
</body>
</html>
```

Multi Agent Process with CrewAI

In [ ]:

from crewai import Agent, Task, Crew, Process

# Define your agents with roles and goals
product_manager = Agent(
    role='Product Manager',
    goal='Define requirements for a software product',
    backstory="You are an experienced Product Manager skilled in defining clear and concise requirements.",
    llm = gpt_llm
)

coder = Agent(
    role='Software Developer',
    goal='Develop software based on the provided requirements',
    backstory="You are a skilled software developer proficient in coding robust and efficient applications.",
    llm = anthropic_llm
)

# Create tasks for your agents
task1 = Task(
    description="Define the key requirements and features for a classic ping pong game. Be specific and concise.",
    expected_output="A clear and concise list of requirements for the ping pong game",
    agent=product_manager
)

task2 = Task(
    description="Based on the provided requirements, develop the code for the classic ping pong game. Focus on gameplay mechanics and a simple user interface.",
    expected_output="Complete code for the ping pong game",
    agent=coder
)

# Instantiate your crew with a sequential process
crew = Crew(
    agents=[product_manager, coder],
    tasks=[task1, task2],
    verbose=1,
)

# Get your crew to work!
result = crew.kickoff()
print("######################")
print(result)

# Agent: Product Manager
## Task: Define the key requirements and features for a classic ping pong game. Be specific and concise.


# Agent: Product Manager
## Final Answer: 
1. Game Interface:
   - A simple 2D graphical interface displaying the ping pong table, net, paddles, and ball.
   - Scoreboard displaying current scores for both players.
   - Option to pause and resume the game.

2. Game Mechanics:
   - Physics engine to simulate realistic ball movement, including speed, spin, and bounce.
   - Collision detection for ball interaction with paddles and boundaries.
   - Two paddles controlled by players or AI opponents.

3. Control Options:
   - Keyboard controls: arrow keys or WASD for paddle movement.
   - Option for mouse control for more intuitive paddle movement.
   - Touch controls for mobile or touchscreen devices.

4


# Agent: Software Developer
## Task: Based on the provided requirements, develop the code for the classic ping pong game. Focus on gameplay mechanics and a simple user interface.


# Agent: Software Developer
## Final Answer: 
Here's a complete implementation of a simple ping pong game using Python and Pygame, which covers the requirements outlined:

```python
import pygame
import sys

# Initialize Pygame
pygame.init()

# Constants
WIDTH, HEIGHT = 800, 400
WHITE = (255, 255, 255)
BLACK = (0, 0, 0)
BALL_RADIUS = 10
PADDLE_WIDTH, PADDLE_HEIGHT = 10, 60
PADDLE_SPEED = 5
BALL_SPEED_X, BALL_SPEED_Y = 4, 4
FPS = 60

# Set up display
window = pygame.display.set_mode((


######################
Here's a complete implementation of a simple ping pong game using Python and Pygame, which covers the requirements outlined:

```python
import pygame
import sys

# Initialize Pygame
pygame.init()

# Constants
WIDTH, HEIGHT = 800, 400
WHITE = (255, 255, 255)
BLACK = (0, 0, 0)
BALL_RADIUS = 10
PADDLE_WIDTH, PADDLE_HEIGHT = 10, 60
PADDLE_SPEED = 5
BALL_SPEED_X, BALL_SPEED_Y = 4, 4
FPS = 60

# Set up display
window = pygame.display.set_mode((

Telemetry with Portkey

Part 1: Logs

For the above agent run, we have initiated two agents utilizing the GPT-3.5 and Llama 3 models. The detailed logs for this request are available on the Portkey dashboard, providing comprehensive insights into the activities of each agent.

Part 2: Analytics

The insights from this request can be viewed on the Portkey dashboard. We are able to infer the following:

1. Total Tokens Used: 3.6K tokens
2. Total Number of Requests: 11 requests; Llama 3 (6 requests), GPT-3.5 (5 requests)

You can also review many other metrics from the dashboards like cost, latency, etc.