DataLakeFileSystemClient::ListPaths() throws JSON exception due to accessing undefined fields

Bắt đầu trong sdk/storage/azure-storage-files-datalake/src/rest_client.cpp, tại _detail::FileSystemClient::ListPaths(), và so sánh việc đọc các trường tùy chọn với các trường được bảo vệ bên dưới chúng. Tái hiện bằng một phản hồi ListPaths bỏ qua owner, group, permissions hoặc contentLength, sau đó xác minh rằng lệnh gọi hoàn tất và để các giá trị PathItem bị thiếu ở trạng thái chưa được thiết lập. Bản vá đính kèm và tệp rest_client.cpp.patched cung cấp tài liệu tham khảo.

** Title **
DataLakeFileSystemClient::ListPaths() throws JSON exception due to accessing undefined fields

Describe the bug

Azure::Storage::Files::DataLake::DataLakeFileSystemClient::ListPaths()
(and anything built on top of it, e.g. DataLakeDirectoryClient::ListPaths())
intermittently throws a nlohmann::json::type_error (exception id 302,
"type must be string, but is ...") with a type that varies unpredictably
from call to call on the exact same server response, even though the
response is well-formed JSON.

Root cause: in _detail::FileSystemClient::ListPaths()
(sdk/storage/azure-storage-files-datalake/src/rest_client.cpp), the
optional per-path fields owner, group, permissions, and
contentLength are read via nlohmann::json's const operator[]
with no .count()/.contains() existence check first — unlike the
sibling optional fields in the very same loop (EncryptionScope,
creationTime, expiryTime, EncryptionContext, etag), which are all
correctly guarded with if (var0.count("etag") != 0) { ... }.

owner/group/permissions are only present in the ADLS Gen2 ListPaths
REST response when the filesystem has hierarchical-namespace ACLs enabled
and the request set upn=true; contentLength can also legitimately be
absent for certain path states (e.g. some archived-tier blobs in our
testing). When any of these keys is missing, operator[] on a const
json falls through (in the vendored json.hpp) to:

auto it = m_data.m_value.object->find(key);
_azure_JSON_ASSERT(it != m_data.m_value.object->end());
return it->second;

_azure_JSON_ASSERT is defined in the vendored copy as:

#define _azure_JSON_ASSERT(x) //assert(x)

i.e. it is unconditionally a no-op in every build configuration
(release and debug), so the existence check never actually runs, and the
code unconditionally dereferences the end() iterator — undefined
behavior. Whatever garbage memory happens to follow the map's sentinel
gets reinterpreted as a json value, and the following .get<std::string>()
call throws type_error with a type ("number"/"binary"/"null"/etc.) that
depends on whatever bytes happened to be there — a classic UB signature,
not a deterministic parsing bug.

Exception or Stack Trace

Observed (type varies between runs against the identical captured
response bytes — this run happened to show binary):

terminate called after throwing an instance of 'nlohmann::json::type_error'
  what():  [json.exception.type_error.302] type must be binary, but is string

Other runs against the same bytes produced type must be string, but is number or ... but is null instead — confirming this is memory
corruption / undefined behavior, not a real type mismatch in the response. Independently
re-parsing the exact same captured response bytes with a fresh
nlohmann::json::parse() call always succeeds, with every field correctly
typed as a string.

To Reproduce

Steps to reproduce the behavior:

1. Create an ADLS Gen2 (hierarchical-namespace-enabled) storage account/
   filesystem, or use one where path responses omit owner/group/
   permissions (i.e. requests are not scoped with upn=true) — or one
   where at least one listed path omits contentLength.
2. Upload one or more blobs/files into that filesystem.
3. Call DataLakeFileSystemClient::ListPaths() (or
   DataLakeDirectoryClient::ListPaths(), which relies on the same
   protocol-layer code) against that filesystem/directory.
4. Observe that the call throws nlohmann::json::type_error (exception id
   302) intermittently/nondeterministically, even though the raw HTTP
   response body is valid, well-formed JSON with every field correctly
   typed as documented.

In our environment this was 100% reproducible against a production
account/container whose ListPaths responses omitted owner/group/
permissions for every path (no upn scoping used), and additionally
omitted contentLength for some archived-tier blobs.

Code Snippet

The buggy code, in
sdk/storage/azure-storage-files-datalake/src/rest_client.cpp,
_detail::FileSystemClient::ListPaths():

vectorElement2.LastModified = DateTime::Parse(
    var0["lastModified"].get<std::string>(), Azure::DateTime::DateFormat::Rfc1123);
vectorElement2.FileSize = var0["contentLength"].is_number_integer()
    ? var0["contentLength"].get<std::int64_t>()
    : std::stoll(var0["contentLength"].get<std::string>());
vectorElement2.Owner = var0["owner"].get<std::string>();
vectorElement2.Group = var0["group"].get<std::string>();
vectorElement2.Permissions = var0["permissions"].get<std::string>();
if (var0.count("EncryptionScope") != 0)
{
  vectorElement2.EncryptionScope = var0["EncryptionScope"].get<std::string>();
}
// ...creationTime / expiryTime / EncryptionContext / etag are all
// similarly guarded with var0.count(...) checks just below this.

Note the inconsistency: owner/group/permissions/contentLength are
read unconditionally via operator[], while every other optional field a
few lines later in the same function is correctly guarded with
var0.count(key) != 0 first.

A minimal, self-contained repro of the underlying nlohmann::json UB
(independent of the Azure SDK, using the exact vendored assert macro
behavior) is:

#include <azure/core/internal/json/json.hpp>  // vendored nlohmann::json
using Azure::Core::Json::_internal::json;

int main() {
  const json obj = json::parse(R"({"name":"foo"})");  // no "owner" key
  // UB: _azure_JSON_ASSERT is a no-op, so this doesn't abort/throw
  // deterministically -- it dereferences object->end() and returns
  // garbage reinterpreted as a json value.
  std::string owner = obj["owner"].get<std::string>();
}

Expected behavior

ListPaths() should tolerate the documented optional fields being absent
from the response, exactly the way it already does for EncryptionScope,
creationTime, expiryTime, EncryptionContext, and etag a few lines
below in the same function — i.e. guard each optional-field read with
var0.count(key) != 0 (or .contains(key)) before calling operator[],
and leave the corresponding PathItem member at its default/unset value
when the key is absent, rather than invoking undefined behavior.

Screenshots

Not applicable (this is a backend/library exception, not a UI issue).

Setup (please complete the following information):

• OS: SUSE Linux Enterprise Server 15 SP7 (SLES 15-SP7, x86_64); also
  expected to reproduce on any platform, since the bug is in
  platform-independent deserialization code
• IDE: N/A (reproduced via a standalone C++ test harness, not an IDE)
• Compiler/toolchain: GCC 14.3.0, CMake 3.28.3
• Version of the Library used: azure-storage-files-datalake 12.15.0
  (root cause confirmed still present by source inspection of the current
  main branch of azure-sdk-for-cpp as of this report)

Additional context

• This bug was discovered while investigating a customer-reported issue in
  which a CREATE FOREIGN TABLE-style directory listing against a
  container containing archive-tier blobs (Azure Blob Storage) surfaced a
  confusing client-side JSON deserialization error instead of the correct,
  expected 409 "operation not permitted on an archived blob" server
  error. This bug (and the closely related one filed separately —
  see BUG-REPORT-2-createdon-expireson-format.md — for the
  creationTime/expiryTime format-assumption bug in
  DataLakeDirectoryClient::ListPaths()) turned out to be masking the real
  server error entirely, because the client-side exception was thrown
  before the archive-tier condition could ever be evaluated.
• These two bugs compound/mask each other: fixing this one alone (without
  also fixing the creationTime/expiryTime bug) simply uncovers the
  second bug underneath it, with a completely different exception type
  (std::invalid_argument, what() == "stoll") and no obvious surface
  relationship to this one — both stem from the same class of "assume the
  optional field is present/in an expected format" gap in the same
  ListPaths() code path.
• We would also suggest reconsidering whether _azure_JSON_ASSERT should
  truly be a no-op in all build configurations in the vendored
  json.hpp (sdk/core/azure-core/inc/azure/core/internal/json/json.hpp),
  or at minimum whether operator[] on a const json object should throw
  a catchable json::out_of_range for a missing key instead of relying on
  an assert — this would turn undefined-behavior memory reads into
  deterministic, catchable exceptions across the whole SDK, not just this
  one call site.
• A minimal, self-contained fix is attached as a unified diff:
  NOS-14641-owner-group-permissions-guard.patch. Full before/after source
  is also attached: rest_client.cpp.orig / rest_client.cpp.patched.
  The fix was built and verified end-to-end against a live Azure Storage
  account whose ListPaths responses previously triggered this bug on
  every call.

Information Checklist

Kindly make sure that you have added all the following information above
and checkoff the required fields otherwise we will treat the issuer as an
incomplete report

• Bug Description Added
• Repro Steps Added
• Setup information Added