Market hours: a holiday closes the previous session's post-midnight continuation segment (HKFE futures T+1 session 00:00-03:00), dropping real data

Start in Common/Securities/SecurityExchangeHours.cs, where GetMarketHours returns ClosedAllDay for a holiday before any segment is checked, and read Common/Securities/LocalMarketHours.cs IsContinuousMarketOpen for the midnight-continuation logic. Reproduce with the Python probe in the issue, comparing History bars for 2020-06-24 and 2020-06-29. The design choice (a general rule versus a per-entry opt-in, and whether IsDateOpen changes) must be settled first. Done means the 2020-06-24 window returns 585 bars and CME holidays are unaffected.

Summary

For a market whose session crosses midnight and is continued by a 00:00-start segment on the next calendar day (HKFE futures: PostMarket 17:15-1.00:00:00 followed by PreMarket 00:00-03:00, the exchange's single After-Hours "T+1" session), SecurityExchangeHours.GetMarketHours returns ClosedAllDay for the whole next day whenever that day is a holiday. The 00:00-03:00 segment is part of the previous trading day's session, so the holiday wipes out the last three hours of a session that the exchange was open for and for which the data files exist. History and the data feed return nothing for that window.

Observed on HSI (Hang Seng Index futures, Market.HKFE): a History request for the 2020-06-24 evening session returns 405 minute quote bars ending 2020-06-25 00:00, while the identical request for 2020-06-29 returns 585 bars ending 2020-06-30 03:00. 2020-06-25 is an HKFE holiday (Tuen Ng Festival); 2020-06-30 is not. The minute file for 2020-06-25 holds 180 quote bars (00:00-02:59) with moving prices for every listed contract, i.e. the exchange traded and the data is there, the engine drops it.

This is the same mechanism family as #9866 (wildcard holidays unioned into every symbol entry, holiday checked before segments) but a different symptom: #9869 keeps a symbol's own early close or late open over a market-wide holiday, which does not help here because HSI has no early close on these dates; the problem is that the holiday is applied to a segment that belongs to the preceding session.

Reproduce

QC Cloud backtest 6bf433cb5ca93b302b6355957cb09a39, LEAN 2.5.0.0.18179, algorithm time zone Asia/Hong_Kong:

from AlgorithmImports import *
from datetime import datetime


class HsiEveningSessionProbe(QCAlgorithm):

    def initialize(self):
        self.set_start_date(2020, 7, 20)
        self.set_end_date(2020, 7, 21)
        self.set_cash(100000)
        self.set_time_zone("Asia/Hong_Kong")

        sid = SecurityIdentifier.parse("HSI XGLILOJXRWEH")   # HSI Jul-2020, expiry 2020-07-30
        symbol = Symbol(sid, "HSI")
        sec = self.add_future_contract(symbol, Resolution.MINUTE, fill_forward=False, extended_market_hours=True)
        hours = sec.exchange.hours

        for d in [datetime(2020, 6, 24), datetime(2020, 6, 25), datetime(2020, 6, 29), datetime(2020, 6, 30)]:
            mh = hours.get_market_hours(d)
            self.log(f"MHDB {d:%Y-%m-%d %a}: isDateOpen(ext)={hours.is_date_open(d, True)} closedAllDay={mh.is_closed_all_day} segments={mh}")

        for label, start, end in [
            ("Jun24 evening", datetime(2020, 6, 24, 17, 0), datetime(2020, 6, 25, 4, 0)),
            ("Jun29 evening", datetime(2020, 6, 29, 17, 0), datetime(2020, 6, 30, 4, 0)),
        ]:
            bars = list(self.history[QuoteBar](symbol, start, end, Resolution.MINUTE, fill_forward=False, extended_market_hours=True))
            self.log(f"HIST {label}: {len(bars)} bars" + (f" first_end={bars[0].end_time:%Y-%m-%d %H:%M} last_end={bars[-1].end_time:%Y-%m-%d %H:%M}" if bars else ""))

        self.quit("probe done")

Output:

MHDB 2020-06-24 Wed: isDateOpen(ext)=True closedAllDay=False segments=Wednesday: PreMarket: 00:00:00-03:00:00 | PreMarket: 08:45:00-09:15:00 | Market: 09:15:00-12:00:00 | PreMarket: 12:30:00-13:00:00 | Market: 13:00:00-16:30:00 | PostMarket: 17:15:00-1.00:00:00
MHDB 2020-06-25 Thu: isDateOpen(ext)=False closedAllDay=True segments=Closed All Day
MHDB 2020-06-29 Mon: isDateOpen(ext)=True closedAllDay=False segments=Monday: PreMarket: 08:45:00-09:15:00 | Market: 09:15:00-12:00:00 | PreMarket: 12:30:00-13:00:00 | Market: 13:00:00-16:30:00 | PostMarket: 17:15:00-1.00:00:00
MHDB 2020-06-30 Tue: isDateOpen(ext)=True closedAllDay=False segments=Tuesday: PreMarket: 00:00:00-03:00:00 | ... | PostMarket: 17:15:00-1.00:00:00
HIST Jun24 evening: 405 bars first_end=2020-06-24 17:16 last_end=2020-06-25 00:00
HIST Jun29 evening: 585 bars first_end=2020-06-29 17:16 last_end=2020-06-30 03:00

Expected: 585 bars for both windows (405 bars 17:15-24:00 plus 180 bars 00:00-03:00). The cloud data file future/hkfe/minute/hsi/20200625_quote.zip contains exactly those 180 bars per contract (first bid/ask 24196/24241 at 00:00, last 24296/24297 at 02:59 for the 202007 contract), so this is not a data gap.

The same thing happens on every HKFE holiday that follows a normal trading day, e.g. 2020-07-01 (HKSAR Establishment Day), 2020-10-01 (National Day).

Evidence

• Data/market-hours/market-hours-database.json: Future-hkfe-HSI has no holidays of its own; Future-hkfe-[*] lists 6/25/2020, 7/1/2020, 10/1/2020, ... Both entries define Tuesday-Saturday as PreMarket 00:00:00-03:00:00, ..., PostMarket 17:15:00-1.00:00:00, i.e. the 00:00-03:00 segment is the tail of the previous day's post-market segment.
• Common/Util/MarketHoursDatabaseJsonConverter.cs:317-331: the market-wide ([*]) holidays are added to the symbol entry's holidayDates unless the symbol has an early close or late open on that date.
• Common/Securities/SecurityExchangeHours.cs:514-526: GetMarketHours returns LocalMarketHours.ClosedAllDay(dayOfWeek) when the date is in _holidays, before any segment is considered. IsOpen(DateTime, bool) (:177-180) and IsOpen(DateTime, DateTime, bool) (:190-215) go through it, so the 00:00-03:00 window on the holiday is treated as closed.
• Common/Securities/LocalMarketHours.cs:377-391: IsContinuousMarketOpen already knows that a segment ending at Time.OneDay followed by one starting at TimeSpan.Zero on the next day is one continuous open period, but GetMarketHours never consults it on the holiday path.

Proposed change

In SecurityExchangeHours.GetMarketHours, when localDateTime.Date is a holiday, resolve the previous day's hours (GetMarketHours(localDateTime.Date.AddDays(-1)), which already applies that day's own holiday / early-close state). If its last segment ends at Time.OneDay and the weekday template for the holiday starts with a segment at TimeSpan.Zero, return a LocalMarketHours holding only the leading segments that are contiguous from midnight (here PreMarket 00:00-03:00), instead of ClosedAllDay. Otherwise keep the current behaviour. A preceding holiday or an early close on the eve ends the previous day before midnight and therefore yields ClosedAllDay as today.

Caveat worth a design decision before implementing: the same rule would apply to every market with a midnight-crossing session (CME futures Sunday-Thursday evenings). On a CME holiday whose eve has no early-close row in the database the holiday would become open from 00:00 to the first break of the weekday template, which is right where the exchange really did keep Globex open overnight and wrong where it did not. An alternative with no cross-market effect is an explicit per-entry opt-in (e.g. a continuesPreviousSession flag on the 00:00 segment) or an earlyCloses row on the holiday itself at the end of the continuation (7/1/2020: 03:00:00), which the existing #9869 logic would then honour.

Open questions

• Should IsDateOpen(date, extendedMarketHours: true) return true for such a holiday (only the continuation trades), or stay false for scheduling purposes with only IsOpen / GetMarketHours changed?
• The database is not date-versioned: HKFE's T+1 session ended at 01:00 before mid-2019 and at 03:00 after, so the 2019 and earlier continuations are 60 bars, which the single weekly schedule cannot express. Out of scope here, noting it so the fix is not blamed for it.

Intercom conversation 215476364179934.