LangSmith

Weco's LangSmith integration lets you optimize code against LangSmith datasets using both code-based evaluators and dashboard-configured LLM judges, without writing shell eval scripts.

Instead of writing an evaluation command that prints metrics to stdout, you point Weco at a LangSmith dataset and evaluators. Weco handles the rest: running your target function against each example, collecting scores, and iteratively improving your code.

How it works

1. Target function: A Python function that receives dataset inputs and returns outputs. LangSmith calls this for each example in your dataset.
2. Evaluators: Scoring functions that run locally (code evaluators) and/or LLM judges configured in the LangSmith dashboard (dashboard evaluators).
3. Metric function: A function that combines all evaluator scores into a single number for Weco to optimize.
4. Optimization loop: Weco iteratively modifies your source file, re-runs evaluation against the dataset, and keeps the version that scores best.

Prerequisites

• Python 3.10+
• Weco CLI installed
• A LangSmith account
• API keys for LangSmith and your LLM provider (e.g., OpenAI)

Tutorial: Optimize an HR QA Agent

This walkthrough uses the ZephHR example, a QA agent that answers HR policy questions over fictional documentation. Weco optimizes the agent's prompts to improve answer quality, measured by a composite metric that combines correctness and helpfulness.

git clone https://github.com/WecoAI/weco-cli.git
cd weco-cli/examples/langsmith-zeph-hr-qa

python -m venv .venv
source .venv/bin/activate
pip install weco openai langsmith

python -m venv .venv
.venv\Scripts\activate
pip install weco openai langsmith

python -m venv .venv
.venv\Scripts\Activate.ps1
pip install weco openai langsmith

export OPENAI_API_KEY="sk-..."
export LANGCHAIN_API_KEY="lsv2_..."

def answer_hr_question(inputs: dict) -> dict:
    """Answer an HR policy question from the ZephHR docs."""
    question = inputs.get("question", "")

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": USER_TEMPLATE.format(docs=DOCS, question=question)},
        ],
        temperature=0.0,
        response_format={"type": "json_object"},
    )

    # ... parse and return {answer, confidence, relevant_sections}

def qa_score(scores: dict) -> float:
    """Combine correctness (binary gate) with helpfulness (1-5 signal).

    correctness * (helpfulness - 1) / 4

    - Incorrect answers always score 0.
    - Correct answers are ranked by helpfulness, normalized to 0-1.
    """
    correctness = scores.get("correctness", 0.0)
    helpfulness = scores.get("helpfulness", 1.0)
    return correctness * (helpfulness - 1.0) / 4.0

def json_schema_validity(run, example) -> dict:
    """Check that the agent output contains the required fields."""
    outputs = run.outputs or {}
    checks = {
        "answer": isinstance(outputs.get("answer"), str) and len(outputs["answer"]) > 0,
        "confidence": outputs.get("confidence") in ("high", "medium", "low"),
        "relevant_sections": isinstance(outputs.get("relevant_sections"), list),
    }
    passed = all(checks.values())
    return {"key": "json_schema_validity", "score": 1.0 if passed else 0.0, "comment": "..."}

def conciseness(run, example) -> dict:
    """Score based on answer length. Penalises empty or excessively verbose answers."""
    # Returns score: 1.0 for ≤150 words, 0.5 for ≤250, 0.0 for >250 or empty

python setup_dataset.py

weco run --source agent.py \
  --eval-backend langsmith \
  --langsmith-dataset zephhr-qa \
  --langsmith-splits opt \
  --langsmith-target agent:answer_hr_question \
  --langsmith-evaluators evaluators:json_schema_validity evaluators:conciseness \
  --langsmith-dashboard-evaluators helpfulness correctness \
  --langsmith-metric-function evaluators:qa_score \
  --additional-instructions optimizer_exemplars.md \
  --metric qa_score --goal maximize --steps 10

weco run --eval-backend langsmith

weco run --source agent.py \
  --eval-backend langsmith \
  --langsmith-dataset zephhr-qa \
  --langsmith-splits holdout \
  --langsmith-target agent:answer_hr_question \
  --langsmith-evaluators evaluators:json_schema_validity evaluators:conciseness \
  --langsmith-dashboard-evaluators helpfulness correctness \
  --langsmith-metric-function evaluators:qa_score \
  --metric qa_score --goal maximize --steps 1

weco run --eval-backend langsmith

Key concepts

Target function

Your target function is specified as module:function (e.g., agent:answer_hr_question). It receives an inputs dict from the dataset and returns a dict of outputs. LangSmith calls this function once per dataset example.

Code evaluators

Code evaluators are Python functions specified as module:function (e.g., evaluators:json_schema_validity). Each receives (run, example) and returns a dict with key, score, and optionally comment:

def my_evaluator(run, example) -> dict:
    # run.outputs contains the target function's output
    # example.outputs contains the expected output from the dataset
    return {"key": "my_evaluator", "score": 1.0, "comment": "Passed"}

Dashboard evaluators

Dashboard evaluators are LLM judges configured in the LangSmith UI. They run asynchronously after evaluation. Weco polls for their scores automatically. Specify them by name with --langsmith-dashboard-evaluators.

Metric function

A metric function combines all evaluator scores into a single number for Weco to optimize. It receives a dict of {evaluator_name: aggregated_score} and returns a float:

def my_metric(scores: dict) -> float:
    return scores["accuracy"] * scores["efficiency"]

Summary aggregation

Per-example evaluator scores are aggregated across the dataset using --langsmith-summary (default: mean). Options: mean, median, min, max.

CLI reference

All LangSmith-specific flags for weco run --eval-backend langsmith: