Track Large Language Models¶
When it comes to building applications using Large Language Models, a lot of time is spent working on prompt engineering rather than training models. This new workflow requires a different set of tools that Comet is developing under the umbrella of LLMOps.
The tools available for Large Language Models fall under two categories:
- LLM projects: Dedicated views specifically for analyzing prompts, responses and chaings. The functionality has been specifically built to allow you to analyze tens of thousands of prompts.
- LLM panels: Visualizations that can be used with Experiment Management to view prompts and chains, usefull when projects contain both fine-tuning and prompt engineering use-cases.
LLM Projects¶
LLM Projects differ from standard Experiment Management projects in that they are customized to the prompt engineering workflow rather than a fine-tuning or model training workflow.
Creating an LLM project can be done in one of two ways:
- Through the Comet UI by using the
New Project
button and choosingLarge Language Model
as the project type. - By using the LLM Python SDK to log a prompt to a new project.
Once the LLM project has been created, it can be accessed from the Projects page and gives you access to a number of prompt analysis tools.
Learn more about LLM projects here.
LLM Panels¶
When working with LLMs, some workflow are not just focused on prompt engineering but include a combination of fine-tuning and prompt engineering. This workflows can be tracked by using two different projects for each parts of the workflow or by using a single project and utilizing the LLM panels.
There are two LLM panels that are currently available:
- Prompt Playground: Interact with Large Language Models from the Comet UI. All your prompts and responses will be automatically logged to Comet.
- Prompt History: Track all your prompt / response pairs. You can also view prompt chains to identify where issues might be occuring.
Learn more about LLM panels here.
Example projects¶
A demo LLM project is available here: LLM demo project
A demo project with the Prompt History panel is available here: LangChain demo project.