runs

Get lang model runs from langsmith

Get Runs

Background

Langsmith offers a convenient python client for retrieving runs. The docs go into further detail about the various options available. Some useful patterns to know are:

Getting a list of runs:

from langsmith import Client
client = Client()
project_runs = client.list_runs(project_name="<your_project>")

Getting a specific run:

from langsmith import Client
client = Client()
run = client.client.read_run("<run_id>")

Furthermore, there are various ways to filter and search runs which are described in the documentation. If these suit your needs, you may not need the utilities in this module. This module offers opinionated wrappers around the Langsmith client that retrieve runs using common patterns we have seen.

Utilities

The following functions help retrieve runs by a very specific kind of tag, as well as recent runs.

assert reformat_date('9/22/2023') == '2023-09-22'
assert reformat_date('9/2/2023') == '2023-09-02'

The idea behind get_runs_by_commit is to quickly retrieve runs that are being logged to langsmith in CI, for example if you are running offline tests automatically against your language models. For example, let’s get runs with the tag commit:4f59dcec in LangSmith (this is specific to my project).

In LangSmith, the last child is often useful to view the final call to the language model.

_child_runs = get_last_child(take(_runs, 3))
assert _child_runs[0].child_run_ids is None # the child doesn't have other children

It is often helpful to get runs in a batch in a date range:

_runs1 = get_recent_runs(start_dt='10/4/2023', end_dt='10/5/2023', limit=10)
assert len(_runs1) == 10

_runs2 = get_recent_runs(start_dt='10/3/2023', limit=10)
assert len(_runs2) == 10

_runs3 = get_recent_runs(limit=10)
assert len(_runs3) == 10

Fetching runs with this filter: and(eq(status, "success"), gte(start_time, "2023-10-04"), lte(start_time, "2023-10-05"))
Fetching runs with this filter: and(eq(status, "success"), gte(start_time, "2023-10-03"), lte(start_time, "2023-10-06"))
Fetching runs with this filter: and(eq(status, "success"), gte(start_time, "2024-02-22"), lte(start_time, "2024-02-25"))

Because I like to tag my LangSmith runs with commit SHA (see get_runs_by_commit), I also want to see the most recent commit SHAs so I know what to query!

get_recent_commit_tags()

Fetching runs with this filter: and(eq(status, "success"), gte(start_time, "2024-02-22"), lte(start_time, "2024-02-25"))
| start_dt   | commit   |   count |
|:-----------|:---------|--------:|
| 02/24/2024 | ca2232cc |     490 |

get_recent_commit_tags can also return a Pandas dataframe:

_df = get_recent_commit_tags(return_df=True)
assert _df.shape[0] >= 1

Fetching runs with this filter: and(eq(status, "success"), gte(start_time, "2024-02-22"), lte(start_time, "2024-02-25"))

Other Ways Of Getting Runs

You may also want to query runs by feedback, however there are many degrees of freedom with how you can implement feedback. Furthermore, there are many ways you can utilize tags. For these cases, we suggest using the langsmith client directly as discussed earlier.

We will continue to update this library with additional recipes should we find other common patterns that are generalizable.

Parse Data

_run = client.read_run('8cd7deed-9547-4a07-ac01-55e9513ca1cd')
get_params(_run)

{'param_model_name': 'gpt-3.5-turbo-0613',
 'param_n': 1,
 'param_top_p': 1,
 'param_temp': 0,
 'param_presence_penalty': 0,
 'param_freq_penalty': 0}

_funcs = get_functions(_run)
for f in _funcs:
    print(f['name'])

contact-finder
contact-creator
email-campaign-creator
task-creator
task-finder
human-chat
calculator
knowledge-base

_feedback = get_feedback(client.read_run('7aba254d-3812-4050-85a5-ed64af50d2f1'))
assert _feedback[0]['score'] == 0
assert _feedback[0]['key'] == 'empty response'
_feedback

[{'key': 'empty response',
  'score': 0.0,
  'value': None,
  'comment': "expected '' to have a length above 0 but got 0",
  'correction': None}]

Exporting Runs To Pandas

See the chatrecord module.