Overview
The Comet platform has extensive LLMOps functionality powered by a specialized SDK, this SDK is refered to as the LLM SDK and is open-sourced at comet-llm.
For more information on LLM support in Comet, see Track LLMs.
Note
The LLM SDK is under active development. If there are any features you would like to see implemented, reach out on Github
Install LLM SDK¶
To use the LLM SDK, downoad and install using:
pip install comet-llm
The full reference documentation is available at LLM SDK reference
Use the LLM SDK to log prompts and responses¶
The LLM SDK supports logging prompts with it's associated response as well as any associated metadata like token usage. This can be achieved through the function log_prompt
:
import comet_llm
comet_llm.log_prompt(
prompt="Answer the question and if the question can't be answered, say \"I don't know\"\n\n---\n\nQuestion: What is your name?\nAnswer:",
prompt_template="Answer the question and if the question can't be answered, say \"I don't know\"\n\n---\n\nQuestion: {{question}}?\nAnswer:",
prompt_template_variables={"question": "What is your name?"},
metadata= {
"usage.prompt_tokens": 7,
"usage.completion_tokens": 5,
"usage.total_tokens": 12,
},
output=" My name is Alex.",
duration=16.598,
)
Use the LLM SDK to log chains¶
The LLM SDK supports logging a chain of executions that may include more than one LLM call, context retrieval, or data pre- or post-processing.
First start a chain with its inputs:
import comet_llm
comet_llm.start_chain({"user_question": user_question})
For each step in the chain, you can create a Span object. The Span object keeps track of the input, outputs, and duration of the step. You can have as many Spans as needed, and they can be nested within each other. Here is very simple example:
with comet_llm.Span(
category="YOUR-SPAN-CATEGORY", # You can use any string here
inputs=INPUTS, # You can pass any object in that dict as long as it can be dumped in JSON
) as span:
YOUR_CODE_HERE
span.set_outputs(outputs=OUTPUTS) # You can pass any object in that dict as long as it can be dumped in JSON
Here is a more realistic example of Spans including nesting of them:
def retrieve_context(user_question):
# Retrieve the context
with comet_llm.Span(
category="context-retrieval",
name="Retrieve Context",
inputs={"user_question": user_question},
) as context_span:
context = get_context(user_question)
context_span.set_outputs(outputs={"context": context})
return context
def llm_call(user_question, context):
prompt_template = """You are a helpful chatbot. You have access to the following context:
{context}
Analyze the following user question and decide if you can answer it, if the question can't be answered, say \"I don't know\":
{user_question}
"""
prompt = prompt_template.format(user_question=user_question, context=context)
with comet_llm.Span(
category="llm-call",
inputs={"prompt_template": prompt_template, "prompt": prompt},
) as llm_span:
# Call your LLM model here
result = "Yes we are currently open"
usage = {"prompt_tokens": 52, "completion_tokens": 12, "total_tokens": 64}
llm_span.set_outputs(outputs={"result": result}, metadata={"usage": usage})
return result
with comet_llm.Span(
category="llm-reasoning",
inputs={
"user_question": user_question,
},
) as span:
context = retrieve_context(user_question)
result = llm_call(user_question, context)
span.set_outputs(outputs={"result": result})
Then finally end your chain and upload it with:
comet_llm.end_chain(outputs={"result": result})
Upcoming features¶
The LLM SDK is under active development, we are currently planning on implementing:
- Support for logging prompt and response embeddings
Feel free to reach out on Github with any and all feature requests !