Unsupported SecurityType 'Option' for symbol '?SPY'

Problem Description

This is likely an issue caused by file-based universe selection, which has been mentioned here: https://github.com/QuantConnect/Lean.DataSource.Polygon/issues/19

Option universe is now file based after https://github.com/QuantConnect/Lean/pull/8212

Add a logging statement to the line in this plugin, it will tell that:

20250222 21:42:29.014 TRACE:: ThetaDataProvider.GetHistory: [DEBUG] ?SPY - Daily - Quote - 2/21/2025 5:00:00 AM - 2/22/2025 9:42:29 PM
20250222 21:42:29.014 TRACE:: ThetaDataProvider.GetHistory: Unsupported SecurityType 'Option' for symbol '?SPY'

which was yield by the following block:

// QuantConnect.ThetaData/ThetaDataHistoryProvider.cs
...
Log.Trace($"{nameof(ThetaDataProvider)}.{nameof(GetHistory)}: [DEBUG] {historyRequest.Symbol} - {historyRequest.Resolution} - {historyRequest.TickType} - {startDateTimeUtc} - {historyRequest.EndTimeUtc}");
if (!CanSubscribe(historyRequest.Symbol))
{
    if (!_invalidSecurityTypeWarningFired)
    {
        _invalidSecurityTypeWarningFired = true;
        Log.Trace($"{nameof(ThetaDataProvider)}.{nameof(GetHistory)}: Unsupported SecurityType '{historyRequest.Symbol.SecurityType}' for symbol '{historyRequest.Symbol}'");
    }
    return null;
}

It is clear that it tries to pass ?SPY into the GetHistory method, however, GetHistory is intended for fetching data for individual contracts, and not for fetching the entire option chain, thus causing this issue. And if it didn't directly return null inside from this if-block, the code will then subsequently invoke the GetHistoricalQuoteData methed (issue #11 is trying to resolve how GetHistoricalQuoteData behaves when there's no valid quote bar for construction, but that would not resolve this issue, since this if-block will directly return null when we pass in a ?SPY instead of the individual contract symbols).

From my observation and experiment, in live trading, this GetHistory method was invoked by OptionChain method defined in Lean/Algorithm/QCAlgorithm.cs:

// Lean/Algorithm/QCAlgorithm.cs
...
public OptionChains OptionChains(IEnumerable<Symbol> symbols, bool flatten = false) {
    ...
    var optionChainsData = History(optionCanonicalSymbols, 1).GetUniverseData()
         .Select(x => (x.Keys.Single(), x.Values.Single().Cast<OptionUniverse>()));
    ...
}

namely the file-based universe selection process, which was introduced starting here: https://github.com/QuantConnect/Lean/commit/16c425934218954082d97b1e2ea03228cddf4783 .

However, even then, it still tries to get the universe content via a network request, instead of retrieving directly from the universe csv file on that day, which should be the correct way if we are using Lean CLI to do live trading.

Expected Behavior

As long as we have built the Universe according to the sample universe format given by https://github.com/QuantConnect/Lean/blob/master/Data/option/usa/universes/spy/20240109.csv , we should be able to avoid network request for fetching the option chain from the data provider.
Currently it is doing extra work here (the local universe has been located and loaded before this redundant request), and it is doing it incorrectly.

Additionally, even if it is going to fetch the most recent market price for all the contracts, individual requests over 10k contracts are not acceptable (SetFilter right now is not working as expected in my experiment with ThetaData, I will investigate and open an issue for Lean main repository if it is involved), currently this plugin hasn't utilized bulk request to get whole universe data.

However, this is not an urgent issue. Won't affect the behavior in live trading.

Some code snippets that might be helpful for debugging

Code:

// Lean/Algorithm/QCAlgorithm.cs
...
[DocumentationAttribute(AddingData)]
public OptionChains OptionChains(IEnumerable<Symbol> symbols, bool flatten = false)
{
    Log($"OptionChains: received {symbols.Count()} symbols: {string.Join(", ", symbols.Select(x => x.Value))}");
    var canonicalSymbols = symbols.Select(GetCanonicalOptionSymbol).ToList();
    var optionCanonicalSymbols = canonicalSymbols.Where(x => x.SecurityType != SecurityType.FutureOption);
    var futureOptionCanonicalSymbols = canonicalSymbols.Where(x => x.SecurityType == SecurityType.FutureOption);

    // var optionChainsData = History(optionCanonicalSymbols, 1).GetUniverseData()
    //     .Select(x => (x.Keys.Single(), x.Values.Single().Cast<OptionUniverse>()));

    // TODO: why do we need to make a historical request here, especially this will yield a Daily resolution request on the quote data
    // I cannot see the point for options here. For options, we only cares about the contracts list, all data should be from OnData method
    Log($"OptionChains: requesting history for Resolution={GetResolution(optionCanonicalSymbols.First(), null, typeof(QuoteBar))}");
    var historyData = History(optionCanonicalSymbols, 1);
    Log($"HistoryData type: {historyData.GetType().FullName}");
    Log($"History retrieved for canonical symbols: {string.Join(", ", optionCanonicalSymbols.Select(x => x.Value))}");
    Log($"History length: {historyData.Count()}");

    var universeData = historyData.GetUniverseData();
    Log($"UniverseData type: {universeData.GetType().FullName}");
    Log("Extracted universe data from history.");
    foreach (var entry in universeData)
    {
        var key = entry.Keys.FirstOrDefault();
        var values = entry.Values.FirstOrDefault();
        if (key != null)
        {
            var valueTypes = values != null 
                ? string.Join(", ", values.Select(v => v.GetType().Name).Distinct()) 
                : "no values";
            int count = values?.Count() ?? 0;
            Log($"UniverseData entry - Key: {key}, Count: {count}, Value Types: {valueTypes}");
            var firstFew = values.Take(5);
            foreach (var item in firstFew)
            {
                Log($"    ItemType: {item.GetType().Name}, Item: {item}");
            }
        }
        else
        {
            Log("UniverseData entry with no key.");
        }
    }

    var optionChainsData = universeData.Select(dataEntry =>
    {
        var canonicalOptionSymbol = dataEntry.Keys.Single();
        var optionUniverseCollection = dataEntry.Values.Single().Cast<OptionUniverse>();
        Log($"Processing universe data for canonical symbol: {canonicalOptionSymbol}");
        Log($"Option universe collection size: {optionUniverseCollection.Count()}");
        return (canonicalOptionSymbol, optionUniverseCollection);
    });

    // TODO: For FOPs, we fall back to the option chain provider until OptionUniverse supports them
    var futureOptionChainsData = futureOptionCanonicalSymbols.Select(symbol =>
    {
        var optionChainData = OptionChainProvider.GetOptionContractList(symbol, Time)
            .Select(contractSymbol => new OptionUniverse()
            {
                Symbol = contractSymbol,
                EndTime = Time.Date,
            });
        return (symbol, optionChainData);
    });

    var time = Time.Date;
    var chains = new OptionChains(time, flatten);
    foreach (var (symbol, contracts) in optionChainsData.Concat(futureOptionChainsData))
    {
        var symbolProperties = SymbolPropertiesDatabase.GetSymbolProperties(symbol.ID.Market, symbol, symbol.SecurityType, AccountCurrency);
        if (symbolProperties != null)
        {
            Log("SymbolProperties type: " + symbolProperties.GetType().FullName);
            Log("SymbolProperties content: " + symbolProperties);
        }
        var optionChain = new OptionChain(symbol, time, contracts, symbolProperties, flatten);
        chains.Add(symbol, optionChain);
    }

    return chains;
}

Output:

20250222 21:42:30.199 TRACE:: Log: OptionChains: received 1 symbols: ?SPY
20250222 21:42:30.199 TRACE:: Log: OptionChains: requesting history for Resolution=Daily
20250222 21:42:30.199 TRACE:: Log: HistoryData type: QuantConnect.Util.MemoizingEnumerable`1[[QuantConnect.Data.Slice, QuantConnect.Common, Version=2.5.0.0, Culture=neutral, PublicKeyToken=null]]
20250222 21:42:30.199 TRACE:: Log: History retrieved for canonical symbols: ?SPY
20250222 21:42:30.199 TRACE:: Log: History length: 1
20250222 21:42:30.199 TRACE:: Log: UniverseData type: System.Linq.Enumerable+IteratorSelectIterator`2[[QuantConnect.Data.BaseData, QuantConnect.Common, Version=2.5.0.0, Culture=neutral, 
PublicKeyToken=null],[QuantConnect.Data.Market.DataDictionary`1[[QuantConnect.Data.UniverseSelection.BaseDataCollection, QuantConnect.Common, Version=2.5.0.0, Culture=neutral, PublicKeyToken=null]], QuantConnect.Common, 
Version=2.5.0.0, Culture=neutral, PublicKeyToken=null]]
20250222 21:42:30.199 TRACE:: Log: Extracted universe data from history.
20250222 21:42:30.199 TRACE:: Log: UniverseData entry - Key: ?SPY, Count: 9542, Value Types: OptionUniverse
20250222 21:42:30.199 TRACE:: Log:     ItemType: OptionUniverse, Item: SPY 33HG3ZAVVP5YE|SPY R735QTJ8XC9X: ¤0.00
20250222 21:42:30.199 TRACE:: Log:     ItemType: OptionUniverse, Item: SPY 32SK364QUWYUE|SPY R735QTJ8XC9X: ¤0.00
20250222 21:42:30.199 TRACE:: Log:     ItemType: OptionUniverse, Item: SPY 3300W7N23EWW6|SPY R735QTJ8XC9X: ¤0.00
20250222 21:42:30.199 TRACE:: Log:     ItemType: OptionUniverse, Item: SPY YUKGCKEM9KO6|SPY R735QTJ8XC9X: ¤0.00
20250222 21:42:30.199 TRACE:: Log:     ItemType: OptionUniverse, Item: SPY YU7NJA4SQMUE|SPY R735QTJ8XC9X: ¤0.00
20250222 21:42:30.199 TRACE:: Log: Processing universe data for canonical symbol: ?SPY
20250222 21:42:30.199 TRACE:: Log: Option universe collection size: 9542
20250222 21:42:30.199 TRACE:: Log: SymbolProperties type: QuantConnect.Securities.SymbolProperties
20250222 21:42:30.199 TRACE:: Log: SymbolProperties content: ,USD,100,0.01,1,,,