PearWiseSDK Integration

Welcome to PearWiseAI, the tool that simplifies LLM system evaluation through AI-powered scoring. This guide will walk you through the simple concepts in PearWiseSDK and how to integrate it with your techstack.

Install PearWise Python SDK

Before diving into PearWise, ensure you have the PearWise Python SDK installed. Open your terminal or command prompt and run the following command:

pip install pearwise

Create a Session

Sessions aggregate multiple Interactions, providing a broader context for model evaluation. Evaluate consistency and effectiveness over a series of Interactions.

quickstart.py

from pearwise import PearWise

# Initialise PearWise API
api_key = "YOUR_API_KEY"
pear = PearWise(api_key)

existing_session_id = None

# Create Session. If session_id = None, we will create a new session
session = pear.session("ANY_MODEL_NAME", id=existing_session_id)

Define Interactions

Create interactions in the session. Interactions represent pairs of model inputs and outputs. They serve as the fundamental units for evaluation.

quickstart.py

# Create Interaction
interaction = session.interact()

model_input = "What is the Pythagoras Theorem?"
def my_model(input):
    # Your model logic here
    return "A^2 + B^2 = C^2"

# Add model input to interaction
interaction.input(model_input)

# Add model output to interaction
model_output = my_model(model_input)
interaction.output(model_output)

Score the Interaction

Interactions and Sessions in PearWise are Scorables, allowing you to attach multiple scores for nuanced evaluation. Scores can be both continuous and discrete. For this example we will use

quickstart.py

# When user gives it a thumbs down
interaction.score("USER_FEEDBACK", -100)

# When domain expert scores the output based on a rubrics
interaction.score("RUBRICS_A", 100)

Log the Interaction

quickstart.py

# submit the interaction and receive session_id
interaction_id, session_id = interaction.log()
print(f"Interaction {interaction_id} in Session {session_id} submitted successfully.")

View Scores on Webapp

Visit our webapp to get an overview of model performance with known data.

Conclusion

And thats it! You are fully integrated with PearWise!