Results from incremental policy.execute do not match full backtest

[9th-Bacar]

I am using cvxportfolio to predict cryptocurrency positions. My goal is to use selected parameters, constraints, and a cov file to predict these positions.

The project is currently under development, and my trading frequency is minute-level. I have 24 hours of returns data for ten trading pairs. I have tried the following steps:

Backtest with the first 12 hours of data, and use the holdings h from the backtest results as the initial holdings.
Subsequently, for each time step, input the latest returns data and execute the policy.execute method.
Use the resulting holdings h_plus as the initial holdings for the next execution.
However, I have found that the results obtained by iteratively executing policy.execute are not consistent with the results obtained from a full backtest using the complete 24-hour data over the same time period. There doesn't seem to be any correlation between the two. This issue has been troubling me for many days, and I hope you can provide some assistance. Thank you very much!

Partial code will be provided below.

def execute_strategy(current_holdings, market_data, policy, hyper_parameters):
    """Execute this strategy.

    :param current_holdings: Current holdings in dollars.
    :type current_holdings: pandas.Series
    :param market_data: Market data server.
    :type market_data: cvxportfolio.data.MarketData
    :param policy: Policy constructor function.
    :type policy: callable
    :param hyper_parameters: Current choice of hyper-parameters.
    :type hyper_parameters: dict

    :return: Output of the execute method of a Cvxportfolio policy.
    :rtype: tuple
    """
    _policy, _ = policy(**hyper_parameters)
    return _policy.execute(h=current_holdings, market_data=market_data)


my_returns, my_my_forecast_df = get_data(mins=1440, mode="head")
my_market_data = cvx.UserProvidedMarketData(
    returns=my_returns_df,
    min_history=pd.Timedelta("1m"),
    cash_key="cash",
)
full_policy = cvx.SinglePeriodOptimization(
    objective=(
        cvx.ReturnsForecast(r_hat=my_forecast_df)
        - config["GAMMA_RISK"] * cvx.FullCovariance(Sigma=my_cov)
        - config["GAMMA_TRADE"] * cvx.TransactionCost(a=config["FEE_RATE"], b=None)
        - config["GAMMA_DN"] * cvx.SoftConstraint(cvx.DollarNeutral())
        - config["GAMMA_HOLD"]
        * cvx.HoldingCost(
            short_fees=config["HLD_RATE"],
            long_fees=config["HLD_RATE"],
            periods_per_year=365 * 3,
        )
    ),
    constraints=[
        cvx.LeverageLimit(config["LEVERAGE_LIMIT"]),
        cvx.MaxWeights(MAX_WEIGHT),
        cvx.MinWeights(-MAX_WEIGHT),
    ],
    ignore_dpp=True,
)
full_simulator = cvx.MarketSimulator(
    market_data=my_market_data,
    cash_key="cash",
    costs=[
        cvx.TransactionCost(a=config["FEE_RATE_ACTUAL"], b=None),
        cvx.HoldingCost(short_fees=0, long_fees=0, periods_per_year=365 * 3),
    ],
    min_history=pd.Timedelta("1m"),
)

full_result = full_simulator.backtest(full_policy)
full_result.w.to_csv("full_w.csv")

my_returns, my_my_forecast_df = get_data(mins=720, mode="head")
my_market_data = cvx.UserProvidedMarketData(
    returns=my_returns_df,
    min_history=pd.Timedelta("1m"),
    cash_key="cash",
)

half_policy = cvx.SinglePeriodOptimization(
    objective=(
        cvx.ReturnsForecast(r_hat=my_forecast_df)
        - config["GAMMA_RISK"] * cvx.FullCovariance(Sigma=my_cov)
        - config["GAMMA_TRADE"] * cvx.TransactionCost(a=config["FEE_RATE"], b=None)
        - config["GAMMA_DN"] * cvx.SoftConstraint(cvx.DollarNeutral())
        - config["GAMMA_HOLD"]
        * cvx.HoldingCost(
            short_fees=config["HLD_RATE"],
            long_fees=config["HLD_RATE"],
            periods_per_year=365 * 3,
        )
    ),
    constraints=[
        cvx.LeverageLimit(config["LEVERAGE_LIMIT"]),
        cvx.MaxWeights(MAX_WEIGHT),
        cvx.MinWeights(-MAX_WEIGHT),
    ],
    ignore_dpp=True,
)
half_simulator = cvx.MarketSimulator(
    market_data=my_market_data,
    cash_key="cash",
    costs=[
        cvx.TransactionCost(a=config["FEE_RATE_ACTUAL"], b=None),
        cvx.HoldingCost(short_fees=0, long_fees=0, periods_per_year=365 * 3),
    ],
    min_history=pd.Timedelta("1m"),
)

half_result = half_simulator.backtest(half_policy)
half_result.w.to_csv("half_w.csv")

init_h = half_result.h.iloc[-1]


for i in range(0, 720):
    my_returns, my_my_forecast_df = get_data(mins=720 + i, mode="head")
    my_market_data = cvx.UserProvidedMarketData(
        returns=my_returns_df,
        min_history=pd.Timedelta("1m"),
        cash_key="cash",
    )
    policy = cvx.SinglePeriodOptimization(
        objective=(
            cvx.ReturnsForecast(r_hat=my_forecast_df)
            - config["GAMMA_RISK"] * cvx.FullCovariance(Sigma=my_cov)
            - config["GAMMA_TRADE"] * cvx.TransactionCost(a=config["FEE_RATE"], b=None)
            - config["GAMMA_DN"] * cvx.SoftConstraint(cvx.DollarNeutral())
            - config["GAMMA_HOLD"]
            * cvx.HoldingCost(
                short_fees=config["HLD_RATE"],
                long_fees=config["HLD_RATE"],
                periods_per_year=365 * 3,
            )
        ),
        constraints=[
            cvx.LeverageLimit(config["LEVERAGE_LIMIT"]),
            cvx.MaxWeights(MAX_WEIGHT),
            cvx.MinWeights(-MAX_WEIGHT),
        ],
        ignore_dpp=True,
    )

    h = init_h
    v = sum(h)
    w = h / v

    u, t, _ = execute_strategy(
        current_holdings=h, market_data=my_market_data, policy=policy
    )

    z = u / v
    w_plus = w + z
    h_plus = h + u
    init_h = h_plus
    save_result(t, w, w_plus, h, h_plus)

[9th-Bacar]

I suspect the issue is related to cvx.DollarNeutral(), because when I removed the following code:
-config["GAMMA_DN"] * cvx.SoftConstraint(cvx.DollarNeutral())
Although the results from both scenarios won't be completely identical, they will be relatively close.

[enzbus]

Yes this is expected (if I understand correctly). The important piece is when you propagate post-trade holdings at time t to initial holdings at time t+1, I believe in these lines (at the end of your script above):

    z = u / v
    w_plus = w + z
    h_plus = h + u
    init_h = h_plus

The MarketSimulator does a few more things instead, most importantly it applies the period's returns to calculate the holdings at the start of next period. The holdings and weights change. This is explained at pages 14-15 (section 2.6) of the paper. Makes sense?

[9th-Bacar]

Thank you very much for your response. The point you mentioned about "applying the period's returns to calculate the holdings at the start of the next period" was critical, and we have added relevant code for this part in our project. However, through further observation, we discovered that the root issue actually lay in the versions of cvxpy and specific solvers used during testing. In our environment, the solvers invoked for full backtests and incremental executions were inconsistent and appeared to function improperly. After updating cvxpy and explicitly specifying the solver used by the policy, the results from full backtests and incremental executions became nearly identical.

[enzbus]

Thanks for the clarification. Indeed, different solver selection and settings might change the solution, and that might be more or less pronounced depending on the formulation of the optimization program (which constraints you use, ...). Interior-point solvers (ECOS, CLARABEL, CVXOPT, various commercial ones) are standard choices, you can often set them to achieve machine numerical precision which means indistinguishable solutions across solvers, if the program has a unique solution (most examples we give have that property). First-order solvers (OSQP,SCS) use much simpler logic and hence are more stable, for example we use SCS as fallback if the default solver from CVXPY (typically an interior-point one) gives numerical errors. They are also faster in some cases. However they typically give solutions with lower precision. It is not always true that low precision solutions give worse back-tested PnL, if fact I can think of program formulations in which you'd get higher PnL with low precision, but in general the lowest numerical precision you set the more differences b/w solvers you'll observe.