fix(data): sandbox feature-expression parser to prevent code execution (CWE-94)

[sebastiondev]

Description

ExpressionProvider.get_expression and DiskExpressionCache._uri evaluate feature-expression strings with the builtin eval:

# qlib/data/data.py (before)
expression = eval(parse_field(field))

# qlib/data/cache.py (before)
if not isinstance(eval(parse_field(field)), Feature):
    ...

parse_field (in qlib/utils/__init__.py) only performs regex rewrites — it turns $close into Feature("close") and Ref(...) into Operators.Ref(...), but it does not validate or sanitize the input. Any other Python syntax in the field string is passed straight through to eval, which happily executes it in a scope that has __builtins__ available. That is a textbook CWE-94 (Improper Control of Generation of Code / code injection) sink.

Field strings reach these functions from user-controlled workflow YAML configs (qrun config.yaml) and from dataset/handler configs shared between users — a very common qlib workflow. A malicious config such as

data_handler_config:
  fields: ["__import__('os').system('touch /tmp/pwned')"]

executes arbitrary code the moment the handler is initialized.

Motivation and Context

CWE-94 — arbitrary code execution via feature expression strings. Qlib's ecosystem revolves around sharing YAML configs (examples in examples/, community strategies, tutorials), so "the attacker controls a field string" is a realistic threat, not a theoretical one. Opening a shared config should not be equivalent to running python -c "$(cat config.yaml)".

Fix

This PR replaces eval(parse_field(field)) with a new parse_expression(field) helper in qlib/data/expression_parser.py. The helper:

1. Runs the existing parse_field regex rewrite (so the DSL still works identically).
2. Parses the resulting string with ast.parse(..., mode="eval").
3. Walks the AST and rejects anything outside a strict whitelist:
   • Literals (ast.Constant, plus legacy Num/Str/NameConstant), tuples/lists.
   • Unary +/-, binary arithmetic, boolean ops, comparisons.
   • Name nodes only if they resolve to Feature, PFeature, or Operators.
   • Attribute nodes only on Operators and only for non-dunder names.
   • Call nodes with positional and simple keyword arguments (no *args/**kwargs unpacking).
4. Evaluates the whitelisted tree with __builtins__ cleared and a fixed name table {Feature, PFeature, Operators}.

Attribute access, subscripting, comprehensions, lambdas, f-strings, __import__, dunder traversal, and any unknown identifier all raise ValueError before any code runs.

Call sites in qlib/data/data.py and qlib/data/cache.py switch from eval(parse_field(...)) to parse_expression(...). The public parse_field helper is left in place for backward compatibility with any downstream users that only want the string rewrite.

How Has This Been Tested?

• Added tests/misc/test_expression_parser_safety.py with cases covering:
  • Legitimate expressions ($close, Ref($close, -1), Mean($high-$low, 5) / $close) parse to the same objects the old eval path produced.
  • Malicious payloads are rejected: __import__('os').system('id'), ().__class__.__bases__[0].__subclasses__(), (lambda: 1)(), [x for x in range(1)], open('/etc/passwd'), Operators.__class__, Ref($close, **{'d': -1}).
• Ran the new test file locally — all pass.
• Manually re-ran a handful of existing handler configs from examples/benchmarks/ through parse_expression to confirm parity with the previous eval behavior.

Security analysis

We verified this is exploitable end-to-end: constructing a QlibDataLoader (or running qrun) with a fields entry containing __import__('os').system('...') executes the payload on the current interpreter, with the current user's privileges, at handler-initialization time — no other preconditions.

The fix removes the eval sink entirely. The AST walker refuses any node type it hasn't explicitly whitelisted, so future additions to Python's grammar can't silently widen the attack surface — a new node type will raise ValueError until it's reviewed and added. Attribute access is restricted to Operators.<name> with a dunder check, which blocks the classic ().__class__.__bases__[0].__subclasses__() escape.

Adversarial review

Before submitting, we tried to disprove this. We checked whether qlib treats configs as trusted-by-default (it doesn't — configs are routinely shared and downloaded), whether any upstream layer sanitizes field strings before they reach eval (no — parse_field is purely a regex rewrite), and whether the attacker needs privileges that would already grant equivalent access (no — reading a YAML file is not equivalent to arbitrary Python execution). We also confirmed the whitelist doesn't over-restrict: every operator in qlib/data/ops.py is reachable via Operators.<name>(...), which the whitelist permits.

Proof of Concept

# poc.py — run from a qlib checkout with a data provider configured
from qlib.data.data import ExpressionProvider

class P(ExpressionProvider):
    def expression(self, *a, **k): pass

# Before the patch, this executes os.system('touch /tmp/qlib_pwned'):
P().get_expression_instance("__import__('os').system('touch /tmp/qlib_pwned')")

Or via a workflow config:

# pwn.yaml — passed to `qrun pwn.yaml`
task:
  dataset:
    kwargs:
      handler:
        kwargs:
          data_loader:
            kwargs:
              config:
                feature: ["__import__('os').system('touch /tmp/qlib_pwned')"]

After this patch both raise ValueError: unsupported expression element ... and no code executes.

Types of changes

• Fix bugs
• Add new feature
• Update documentation

---

_{Discovered by the Sebastion AI GitHub App.}