Fraud Detection (IEEE-CIS)

Setup

If you haven't already, follow the Installation guide. Otherwise, install the CLI:

pipx install weco

curl -fsSL https://weco.ai/install.sh | sh

irm https://weco.ai/install.ps1 | iex

pip install weco

git clone https://github.com/wecoai/weco-cli.gitcd weco-clipip install -e .

Clone the repo and install the example's dependencies:

git clone https://github.com/WecoAI/weco-cli.git
cd weco-cli/examples/fraud-detection
pip install --upgrade -r requirements.txt

Prepare the data

This one-off script downloads the IEEE-CIS data and builds a fixed, leakage-safe train/val split (100K train / 25K validation, split by time so the validation period is strictly later than training):

python prepare_data.py

It writes data/base_train_small.parquet and data/base_val_small.parquet (SHA-256 identical to the published case study). A quick sanity check should print the deterministic baseline:

python evaluate.py
# auc_roc: 0.909132

The optimization target

Weco edits two files. Each owns a separate scope, and the interface keeps them honest.

features.py — a FeatureBuilder with fit(X_train, y_train) and transform(X). fit() sees only the training split; transform() has no y argument, so it can never branch on labels:

# features.py — Weco optimizes this for the Features scope
class FeatureBuilder:
    def fit(self, X_train: pd.DataFrame, y_train: pd.Series) -> "FeatureBuilder":
        # Fit frequency / group / target encoders on (X_train, y_train) only.
        # Stash state in self.* so transform() can apply it deterministically.
        ...
        return self

    def transform(self, X: pd.DataFrame) -> np.ndarray:
        # Apply self.* state to X. Called once each on X_train and X_val.
        # No `y` here — val labels can't leak into val features.
        ...

model.py — a single train_and_evaluate function that receives pre-built feature arrays and returns validation AUC:

# model.py — Weco optimizes this for the Model scope
def train_and_evaluate(X_train, y_train, X_val, y_val) -> float:
    """Train on (X_train, y_train); return AUC-ROC on (X_val, y_val).
    Tune LightGBM, switch model class, build ensembles — features arrive
    pre-built as ndarrays, so this scope can't re-engineer features."""
    ...
    return float(roc_auc_score(y_val, y_pred))

instructions.md (passed via --additional-instructions) carries the EDA + techniques domain knowledge from the case study, plus a guardrail against silent target leakage.

The evaluator (frozen)

evaluate.py is the API enforcement boundary — Weco never edits it. It loads the data, strips isFraud and TransactionID before any feature code runs, calls fit/transform/train_and_evaluate, and prints the metric line Weco parses:

# evaluate.py (frozen)
y_train = train_df["isFraud"].astype("int32")
X_train = train_df.drop(columns=["isFraud", "TransactionID"])   # target stripped
X_val   = val_df.drop(columns=["isFraud", "TransactionID"])

fb = FeatureBuilder().fit(X_train, y_train)                     # val never seen by fit()
X_train_t, X_val_t = fb.transform(X_train), fb.transform(X_val)

auc = train_and_evaluate(X_train_t, y_train, X_val_t, y_val)
print(f"auc_roc: {auc:.6f}")

Run Weco

Pick a scope. The full pipeline (recommended) lets Weco edit both files:

weco run --sources features.py model.py \
     --eval-command "python evaluate.py" \
     --metric auc_roc \
     --goal maximize \
     --steps 200 \
     --model gemini-3.1-pro-preview \
     --additional-instructions instructions.md \
     --eval-timeout 900 \
     --log-dir .runs/full

weco run --sources features.py model.py ^
     --eval-command "python evaluate.py" ^
     --metric auc_roc ^
     --goal maximize ^
     --steps 200 ^
     --model gemini-3.1-pro-preview ^
     --additional-instructions instructions.md ^
     --eval-timeout 900 ^
     --log-dir .runs/full

weco run --sources features.py model.py `
     --eval-command "python evaluate.py" `
     --metric auc_roc `
     --goal maximize `
     --steps 200 `
     --model gemini-3.1-pro-preview `
     --additional-instructions instructions.md `
     --eval-timeout 900 `
     --log-dir .runs/full

To isolate where the gains come from, run a single scope instead:

• Features only — --sources features.py (model stays at the baseline LightGBM).
• Model only — --sources model.py (features frozen at the baseline FeatureBuilder). Headroom is small (~+0.008 AUC) — model tuning isn't where the wins live for tabular fraud.

Explanation

• --sources features.py model.py: the two files Weco may edit (independently, separately, or together).
• --eval-command "python evaluate.py": the frozen evaluator; Weco parses auc_roc: 0.xxxxxx from its output.
• --metric auc_roc --goal maximize: maximize validation AUC-ROC.
• --steps 200: optimization iterations.
• --model gemini-3.1-pro-preview: the LLM driving the optimization.
• --additional-instructions instructions.md: the EDA + techniques domain prompt.
• --eval-timeout 900: per-evaluation timeout (seconds); a full train+score pass on 100K rows can take a few minutes.