Empty Option Contracts Returned if Invoked inside the Lean Environment (both Backtest and Research)

Problem Description

If I directly run the test cases in the repository, everything runs smoothly. More specifically, I tried the following code in this repository:

        // ThetaDataOptionChainProviderTests.cs
        ...
        private static IEnumerable<Symbol> UnderlyingSymbols
        {
            get
            {
                TestGlobals.Initialize();
                yield return Symbol.Create("SPY", SecurityType.Equity, Market.USA); // New Code Here
                yield return Symbol.Create("XEO", SecurityType.Index, Market.USA);
                yield return Symbol.Create("DJX", SecurityType.Index, Market.USA);
            }
        }

        [TestCaseSource(nameof(UnderlyingSymbols))]
        public void GetOptionContractList(Symbol symbol)
        {
            var referenceDate = new DateTime(2024, 03, 28);
            var optionChain = _thetaDataOptionChainProvider.GetOptionContractList(symbol, referenceDate).ToList();

            // New Code Here
            if (symbol.SecurityType == SecurityType.Equity && symbol.ID.Symbol == "SPY")
            {
                // len == 8334
                Console.WriteLine($"Number of contracts for {symbol.ID.Symbol}: {optionChain.Count} on {referenceDate}");
                Assert.That(optionChain, Has.Count.EqualTo(8334));
            }

            ...
        }

And run the test with dotnet test --filter "FullyQualifiedName=QuantConnect.Lean.DataSource.ThetaData.Tests.ThetaDataOptionChainProviderTests.GetOptionContractList"

The test case runs without an error, and the API returns the expected number of contracts. Everything works as expected here.

However, when running the code in backtesting or research, things go wrong. See the following section, which indicates an issue with the integration with the Lean repository.

The code correctly sets up the DownloaderDataProvider, as expected, and the logs confirm this:

20250206 09:44:32.340 TRACE:: Log: [OnData]   type(DataCacheProvider._dataProvider): QuantConnect.Lean.Engine.DataFeeds.DownloaderDataProvider
20250206 09:44:32.340 TRACE:: Log: [OnData]   type(_dataProvider._dataDownloader): QuantConnect.Lean.DataSource.ThetaData.ThetaDataDownloader

The ThetaDataDownloader is also correctly set in the DownloaderDataProvider, but it is likely not being called, and all the data returned is only fetched from the local disk.

Note: I didn't run lean download to fetch local data. My goal is to use Lean CLI to access the ThetaData API directly instead of QuantConnect's data source.

I have also examined Lean/Engine/HistoricalData/SubscriptionDataReaderHistoryProvider.cs, where the GetHistory method does not return meaningful data. It is invoked from BacktestingChainProvider.GetOptionSymbols.

---

Alex mentioned that this issue was caused by changes to the option contract fetching logic made four months ago. I have identified the relevant commit in the Lean repository here: https://github.com/QuantConnect/Lean/commit/16c425934218954082d97b1e2ea03228cddf4783, along with some subsequent commits. However, simply mimicking the changes in that commit does not resolve my issue. For example, in that commit:

diff --git a/Algorithm.CSharp/AddAndRemoveOptionContractRegressionAlgorithm.cs b/Algorithm.CSharp/AddAndRemoveOptionContractRegressionAlgorithm.cs
index 0a8cce770..4ae44e2c4 100644
--- a/Algorithm.CSharp/AddAndRemoveOptionContractRegressionAlgorithm.cs
+++ b/Algorithm.CSharp/AddAndRemoveOptionContractRegressionAlgorithm.cs
@@ -40,8 +40,8 @@ namespace QuantConnect.Algorithm.CSharp
 
             var aapl = QuantConnect.Symbol.Create("AAPL", SecurityType.Equity, Market.USA);
 
-            _contract = OptionChainProvider.GetOptionContractList(aapl, Time)
-                .OrderBy(symbol => symbol.ID.Symbol)
+            _contract = OptionChain(aapl)
+                .OrderBy(x => x.ID.Symbol)
                 .FirstOrDefault(optionContract => optionContract.ID.OptionRight == OptionRight.Call
                     && optionContract.ID.OptionStyle == OptionStyle.American);
             AddOptionContract(_contract);

And if I use the following code directly, the contract count is still 0:

var contracts = OptionChain(this.underlying).ToList(); // 0 contracts

This suggests an issue with the outdated integration in this repository. I am wondering if you can help resolve this issue or at least you may direct me to a 'correctly updated' DataSource repository, so that I can understand what's the correct way of integrating with the Lean Engine and make a fix for here.

Thanks!

---

For convenience, I have provided a minimal project to reproduce the issue: https://github.com/efJerryYang/ThetaDataWithLeanEmptyResultDemo

I used a fixed Lean Engine version for reproducibility:

❯ lean backtest 'Jumping Red-Orange Penguin' --image quantconnect/lean:16920 --data-provider-historical ThetaData --thetadata-subscription-plan Standard

And the logs would be like:

20250206 09:44:32.335 TRACE:: Debug: Accurate daily end-times now enabled by default. See more at https://qnt.co/3YHaWHL. To disable it and use legacy daily bars set 
self.settings.daily_precise_end_time = False.
20250206 09:44:32.335 TRACE:: StopSafely(): Waiting for 'Result Thread' thread to stop...
20250206 09:44:32.335 TRACE:: Debug: Warning: The following securities were set to raw price normalization mode to work with options: SPY...
20250206 09:44:32.335 TRACE:: Log: [Initialize] Start Date: 3/28/2024 12:00:00 AM
20250206 09:44:32.336 TRACE:: Log: [Initialize] End Date: 3/28/2024 12:00:00 AM
20250206 09:44:32.336 TRACE:: Log: [Initialize] Cash: 100000.0
20250206 09:44:32.336 TRACE:: Log: [Initialize] Before AddEquity: 0
20250206 09:44:32.336 TRACE:: Log: [Initialize] After AddEquity: 1
20250206 09:44:32.336 TRACE:: Log: [Initialize] Before AddOption: 1
20250206 09:44:32.336 TRACE:: Log: [Initialize] After AddOption: 2
20250206 09:44:32.336 TRACE:: Log: [OnData] ======================== OnData [Start] ========================
20250206 09:44:32.336 TRACE:: Log: [OnData] Time: 3/28/2024 10:00:00 AM
20250206 09:44:32.336 TRACE:: Log: [OnData] type(OptionChainProvider): QuantConnect.Lean.Engine.DataFeeds.CachingOptionChainProvider
20250206 09:44:32.337 TRACE:: Log: [OnData] Available contracts by OptionChainProvider.GetOptionContractList(underlying, ...): 0
20250206 09:44:32.337 TRACE:: Log: [OnData]   ------------------- Reflection [Start] -------------------
20250206 09:44:32.337 TRACE:: Log: [OnData]   type(OptionChainProvider._optionChainProvider): QuantConnect.Lean.Engine.DataFeeds.BacktestingOptionChainProvider
20250206 09:44:32.337 TRACE:: Log: [OnData]   type(_optionChainProvider.DataCacheProvider): QuantConnect.Lean.Engine.DataFeeds.ZipDataCacheProvider
20250206 09:44:32.337 TRACE:: Log: [OnData]   type(DataCacheProvider._dataProvider): QuantConnect.Lean.Engine.DataFeeds.DownloaderDataProvider
20250206 09:44:32.337 TRACE:: Log: [OnData]   type(_dataProvider._dataDownloader): QuantConnect.Lean.DataSource.ThetaData.ThetaDataDownloader
20250206 09:44:32.337 TRACE:: Log: [OnData]   ------------------- Reflection [End] -------------------
20250206 09:44:32.337 TRACE:: Log: [OnData] ======================== OnData [End] ========================
...
20250206 09:44:32.342 TRACE:: Log: [OnData] ======================== OnData [Start] ========================
20250206 09:44:32.342 TRACE:: Log: [OnData] Time: 3/28/2024 4:00:00 PM
20250206 09:44:32.342 TRACE:: Log: [OnData] type(OptionChainProvider): QuantConnect.Lean.Engine.DataFeeds.CachingOptionChainProvider
20250206 09:44:32.342 TRACE:: Log: [OnData] Available contracts by OptionChainProvider.GetOptionContractList(underlying, ...): 0
20250206 09:44:32.342 TRACE:: Log: [OnData]   ------------------- Reflection [Start] -------------------
20250206 09:44:32.342 TRACE:: Log: [OnData]   type(OptionChainProvider._optionChainProvider): QuantConnect.Lean.Engine.DataFeeds.BacktestingOptionChainProvider
20250206 09:44:32.343 TRACE:: Log: [OnData]   type(_optionChainProvider.DataCacheProvider): QuantConnect.Lean.Engine.DataFeeds.ZipDataCacheProvider
20250206 09:44:32.343 TRACE:: Log: [OnData]   type(DataCacheProvider._dataProvider): QuantConnect.Lean.Engine.DataFeeds.DownloaderDataProvider
20250206 09:44:32.343 TRACE:: Log: [OnData]   type(_dataProvider._dataDownloader): QuantConnect.Lean.DataSource.ThetaData.ThetaDataDownloader
20250206 09:44:32.343 TRACE:: Log: [OnData]   ------------------- Reflection [End] -------------------
20250206 09:44:32.343 TRACE:: Log: [OnData] ======================== OnData [End] ========================
20250206 09:44:32.343 TRACE:: Debug: Algorithm Id:(1256374201) completed in 0.19 seconds at 0k data points per second. Processing total of 22 data points.
20250206 09:44:32.343 TRACE:: Debug: Your log was successfully created and can be retrieved from: /Results/1256374201-log.txt
20250206 09:44:32.343 TRACE:: BacktestingResultHandler.Run(): Ending Thread...
20250206 09:44:32.381 TRACE::