Weco DocsCLI Reference
Reference for the Weco CLI commands and options
This reference provides detailed information about the Weco CLI commands and their options.
If ever in doubt, you can use weco --help to get a list of all the commands.
Use weco <command> --help to get more information about a specific command (e.g. weco run --help).
The weco run Command
This is the primary command for starting the optimization process. It takes several arguments to configure how Weco should optimize your code.
⚠️ Warning: Code Modification
wecodirectly modifies the file specified by--sourceduring the optimization process. It is strongly recommended to use version control (like Git) to track changes and revert if needed. Alternatively, ensure you have a backup of your original file before running the command. Upon completion, the file will contain the best-performing version of the code found during the run.
Command Arguments
| Argument | Description | Required |
|---|---|---|
--source | Path to the source code file that will be optimized (e.g., optimize.py). | Yes |
--eval-command | Command to run for evaluating the code in --source. This command should print the target --metric and its value to the terminal. | Yes |
--metric | The name of the metric you want to optimize (e.g., 'accuracy', 'speedup', 'loss'). This metric name should match what's printed by your --eval-command. | Yes |
--maximize | Whether to maximize (true) or minimize (false) the metric. | Yes |
--steps | Number of optimization steps (LLM iterations) to run. | Yes |
--model | Model identifier for the LLM to use (e.g., gpt-4o, claude-3.5-sonnet). Recommended models to try include o3-mini, claude-3-haiku, and gemini-2.5-pro-exp-03-25. | Yes |
--additional-instructions | (Optional) Natural language description of specific instructions OR path to a file containing detailed instructions to guide the LLM. | No |
--log-dir | (Optional) Path to the directory to log intermediate steps and final optimization result. Defaults to .runs/. | No |
--preserve-source | (Optional) If set, do not overwrite the original --source file. Modifications and the best solution will still be saved in the --log-dir. | No |
Evaluation Requirements
The command specified by --eval-command is crucial for the optimization process. It must:
- Execute the potentially modified code from
--source - Assess its performance
- Print the metric you specified with
--metricalong with its numerical value to the terminal
For example, if you set --metric speedup, your evaluation script should output a line like:
or
Weco will parse this output to extract the numerical value (1.5 in this case) associated with the metric name ('speedup').
Performance Expectations
Weco, powered by the AIDE algorithm, optimizes code iteratively based on your evaluation results. Achieving significant improvements, especially on complex research-level tasks, often requires substantial exploration time.
The following plot from the independent Research Engineering Benchmark (RE-Bench) report shows the performance of AIDE (the algorithm behind Weco) on challenging ML research engineering tasks over different time budgets:
As shown, AIDE demonstrates strong performance gains over time, surpassing lower human expert percentiles within hours and continuing to improve. This highlights the potential of evaluation-driven optimization but also indicates that reaching high levels of performance comparable to human experts on difficult benchmarks can take considerable time (tens of hours in this specific benchmark, corresponding to many --steps in the Weco CLI).
Factor this into your planning when setting the number of --steps for your optimization runs.
The weco logout Command
This command logs you out of your Weco account. It doesn't take any arguments.