chatrecord

Tools for exporting chat related portions of Langchain runs. 
from nbdev.showdoc import show_doc

When instantiating ChatRecord with the class methods ChatRecord.from_run or ChatRecord.from_run_id, we automatically query the parent run of the LangChain trace in LangSmith to get metadata like feedback. Additionally, if you instantiate ChatRecord with a root run or a run that is not a ChatOpenAI run type, ChatRecord will attempt to find the last ChatOpenAI in your chain and store the id in ChatRecord.child_run_id. The data for this child run (inputs, outputs, functions) is stored in ChatRecord.child_run and is of type RunData.

source

ChatRecord.from_run

 ChatRecord.from_run (run:langsmith.schemas.Run)

Collect information About A Run into a ChatRecord.

Type Details
run Run the run object to parse. 
client = Client()
_root_run = client.read_run('fbfd220a-c731-46a2-87b3-e64a477824f5')
_root_result = ChatRecord.from_run(_root_run)

source

ChatRecord.from_run_id

 ChatRecord.from_run_id (run_id:str)

Collect information About A Run into a ChatRecord.

Type Details
run_id str the run id to fetch and parse. 
_child_run_id = str(_root_run.child_run_ids[-1])
_child_result = ChatRecord.from_run_id(_child_run_id)

Tests

# test that child and root runs are related
test_eq(_root_result.flat_output, _child_result.flat_output)
test_eq(_root_result.parent_run_id, _child_result.parent_run_id)

# Test case without feedback
_parent_run_no_feedback = client.read_run('87900cfc-0322-48fb-b009-33d226d73597')
_no_feedback = ChatRecord.from_run(_parent_run_no_feedback)
test_eq(_no_feedback.feedback, [])

# Test case with feedback

#  ... starting with a child run
_child_w_feedback = client.read_run('f8717b0e-fb90-45cd-be00-9b4614965a2e')
_feedback = ChatRecord.from_run(_child_w_feedback).feedback
assert _feedback[0]['key'] == 'empty response'

# #  ... starting with a parent run
_parent_w_feedback = client.read_run(_child_w_feedback.parent_run_id)
_feedback2 = ChatRecord.from_run(_parent_w_feedback).feedback
test_eq(_feedback[0]['comment'],  _feedback2[0]['comment'])

ChatRecordSet, a list of ChatRecord

source

ChatRecordSet.from_runs

 ChatRecordSet.from_runs (runs:List[langsmith.schemas.Run])

Load ChatRecordSet from runs.

We can create a ChatRecordSet directly from a list of runs:

# from langfree.runs import get_runs_by_commit
_runs = get_runs_by_commit(commit_id='028e4aa4', limit=10)
llmdata = ChatRecordSet.from_runs(_runs)
Fetching runs with this filter: and(eq(status, "success"), has(tags, "commit:028e4aa4"))

There is a special shortcut to get runs by a commit tag which uses get_runs_by_commit for you:

source

ChatRecordSet.from_commit

 ChatRecordSet.from_commit (commit_id:str, limit:int=None)

Create a ChatRecordSet from a commit id

llmdata2 = ChatRecordSet.from_commit('028e4aa4', limit=10)
assert llmdata[0].child_run_id == llmdata2[0].child_run_id
Fetching runs with this filter: and(eq(status, "success"), has(tags, "commit:028e4aa4"))

Finally, you can also construct a ChatRecordSet from a list of run ids:

source

ChatRecordSet.from_run_ids

 ChatRecordSet.from_run_ids (runs:List[str])

Load ChatRecordSet from run ids.

_run_ids = ['ba3c0a47-0803-4b0f-8a2f-380722edc2bf',
 '842fe1b4-c650-4bfa-bcf9-bf5c30f8204c',
 '5c06bbf3-ef14-47a1-a3a4-221f65d4a407',
 '327039ab-a0a5-488b-875f-21e0d30ee2cd']

llmdata3 = ChatRecordSet.from_run_ids(_run_ids)
assert len(llmdata3) == len(_run_ids)
assert llmdata[0].child_run_id == _run_ids[0]

Convert ChatRecordSet to a Pandas Dataframe

You can do this with to_pandas()

_df = llmdata.to_pandas()
_df.head(1)
child_run_id child_run child_url parent_run_id parent_url total_tokens prompt_tokens completion_tokens feedback feedback_keys tags start_dt function_defs param_model_name param_n param_top_p param_temp param_presence_penalty param_freq_penalty
0 ba3c0a47-0803-4b0f-8a2f-380722edc2bf inputs=[{'role': 'system', 'content': 'You are... https://smith.langchain.com/o/9d90c3d2-ca7e-4c... 7074af93-1821-4325-9d45-0f2e81eca0fe https://smith.langchain.com/o/9d90c3d2-ca7e-4c... 0 0 0 [] [] [commit:028e4aa4, branch:testing, test, room:6... 09/05/2023 [{'name': 'contact-finder', 'parameters': {'ty... gpt-3.5-turbo-0613 1 1 0 0 0

Save Data

llmdata.save('_data/llm_data.pkl')
Path('_data/llm_data.pkl')

Load Data

_loaded = ChatRecordSet.load('_data/llm_data.pkl')
assert llmdata.records[0].child_run_id == _loaded.records[0].child_run_id