BUG: Writers save floating-point data as text with too few digits

Abierto
#1,927 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 6 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
35/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
cpp, python
Área
backend

Línea de trabajo

Start with the listed writer locations, including WriteAvizoUniformCoordinate.cpp, WriteLAMMPSFile.cpp, WriteINLFile.cpp, ComputeGBCDMetricBased.cpp, and simplnx_utilities.py, then compare their formatting with the reported failing values. The issue describes a broad audit across many writers and output formats, so identify which cases remain unaddressed and check for format-specific precision requirements. Done means text output round-trips to the written floating-point type, except where user-selected or format-required precision applies.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug

Problem

Several writers save floating-point data as text with a format that drops digits. The output file then holds different values than the data structure, without an error:

  • Fixed decimal places (%f, {:f}, {:.Nf}, std::fixed) turn small nonzero values into zero. For example, an Image Geometry with spacing 1e-7 is written with bounds or coordinates 0.000000.
  • Too few significant digits (default stream precision, digits10, 9 digits for a float64) change the value that a reader parses back.

Expected: every floating-point value that a writer saves as text reads back to the same value of the type that was written. A float32 needs up to 9 significant digits, a float64 up to 17. Fmt {} writes the shortest text that reads back exactly. The exceptions are a precision that the user selects and a precision that a file-format specification fixes.

Occurs at aacedb4dc5c009d21045d715ac8f09a0d213d42e (develop). An audit of all simplnx writers found these places. "Code reading" means the place was confirmed by reading the code and computing the output for the input shown. The two places marked "test below" also have a unit test that was run and seen to fail.

Values become zero or lose all small digits:

Filter Function Location Values written Format Failing input → output; how confirmed
Write Avizo Uniform Coordinate — fixed by #1919 generateHeader() WriteAvizoUniformCoordinate.cpp:69 BoundingBox, in ASCII and binary files %f spacing 1e-7 → bounds 0.000000; test below
Write Avizo Rectilinear Coordinate — moved to #1867 class C07p writeData() (ASCII) WriteAvizoRectilinearCoordinate.cpp:163 X/Y/Z coordinates %f spacing 1e-7 → coordinates 0.000000; test below
Write LAMMPS File — moved to #1867 class C07p operator() WriteLAMMPSFile.cpp:123-125, WriteLAMMPSFile.cpp:146 box bounds, atom coordinates {:f} bound 1e-7 → 0.000000; code reading
Write INL File — moved to #1867 class C07p operator() WriteINLFile.cpp:192-202, WriteINLFile.cpp:232 header steps/origin/bounds; Euler angles and positions in every row std::fixed, precision 6 X_STEP 1e-7 → 0.000000; code reading
Write Los Alamos FFT File — moved to #1867 class C07p WriteChunkedData(), WriteDirectData() WriteLosAlamosFFT.cpp:144, WriteLosAlamosFFT.cpp:218 Euler angles (degrees) {:.3f} 1e-7 rad → 0.000; code reading
Write GBCD Triangles File — moved to #1867 class C07p operator() WriteGBCDTriangleData.cpp:95 Euler angles, normals, face area {:0.4f} area 1e-5 → 0.0000; code reading
Write GBCD Pole Figure (GMT 5) — moved to #1867 class C07p operator() WriteGBCDGMTFile.cpp:307, WriteGBCDGMTFile.cpp:313 misorientation header; angles and intensity rows %.1f, %f angle 0.04° → 0.0; intensity 1e-7 → 0.000000; code reading
Write StatsGenerator ODF Angle File — moved to #1867 class C07p WriteEulerLine() WriteStatsGenOdfAngleFile.cpp:63 Euler angles, weight, sigma std::fixed, precision 8 Euler 1e-9 (radians kept) → 0.00000000; code reading
Write Nodes And Elements File(s) — moved to #1867 class C07p WriteValue<T>() WriteNodesAndElementsFiles.cpp:28 vertex coordinates std::fixed, precision 4 vertex 1e-5 → 0.0000; code reading
Compute GBCD (Metric-Based Approach) — moved to #1867 class C07p operator() ComputeGBCDMetricBased.cpp:654 misorientation header of both output files {:.1F} angle 0.04° → 0.0; code reading
Python pipeline code export _encode_float() simplnx_utilities.py:107 numeric parameters in generated Python round() then fixed decimals origin 1e-13 → 0.0; code reading

Values lose trailing significant digits:

Filter Function Location Values written Format Failing input → output; how confirmed
Convert EDAX Hex Grid to Square Grid (.ang) — moved to #1867 class C07p Converter::operator() ConvertHexGridToSquareGrid.cpp:210 .ang rows: Euler angles, X/Y, IQ, CI, SEM signal, fit default stream, 6 significant digits 1.0000001192f → 1; code reading
Convert EDAX Hex Grid to Square Grid (.ang) — moved to #1867 class C07p modifyAngHeaderLine() ConvertHexGridToSquareGrid.cpp:309, ConvertHexGridToSquareGrid.cpp:313 XSTEP, YSTEP StringUtilities::number() with digits10 (6) step 1.0000001192f → 1; code reading
Write DREAM3D-NX File (Write Xdmf File on) — moved to #1867 class C07p WriteGeomXdmf() (ImageGeom) Dream3dIO.cpp:175-177 origin, spacing default stream, 6 significant digits spacing 1.0000001192f → 1; code reading
Write Abaqus Hexahedron — moved to #1867 class C07p writeNodes() WriteAbaqusHexahedron.cpp:103 node coordinates computed in float64 {:.9g} 1 + 2^-24 → 1.00000006 (a different double); code reading
Write Abaqus Crystal Plasticity File — moved to #1867 class C07p WriteNodes() WriteAbaqusCrystalPlasticity.cpp:100 node coordinates computed in float64 {:.9g} same as above; code reading
Compute GBCD (Metric-Based Approach) — moved to #1867 class C07p distribution and error outputs ComputeGBCDMetricBased.cpp:667, ComputeGBCDMetricBased.cpp:672, ComputeGBCDMetricBased.cpp:682 float64 angles, distribution, errors {:1.8E} (9 significant digits) 141.93951659864035 → 1.41939517E+02; code reading
Compute GBPD (Metric-Based Approach) — moved to #1867 class C07p distribution and error outputs ComputeGBPDMetricBased.cpp:717, ComputeGBPDMetricBased.cpp:722, ComputeGBPDMetricBased.cpp:732 float64 angles, distribution, errors {:.8E} (9 significant digits) 35.264389682754654 → 3.52643897E+01; code reading

Already reported:

Filter Function Location Values written Format Report
Write Vtk Rectilinear Grid — fixed by #1915 writeCoords() WriteVtkRectilinearGrid.cpp:133 ASCII coordinates %f #1908 item L29; PR #1915 changes it to max_digits10
Write ASCII Data, Write Feature Data as CSV File (shared TupleWriter) — tracked in #1867 class C07p TupleWriter::write() OStreamUtilities.cpp:464, OStreamUtilities.cpp:468 float32 / float64 values precision 8 / 16 #1867 item WriteASCIIDataFilter#3

The ImageProcessing developer benchmarks also write timing CSV columns with {:.3f} (BenchmarkRunner.cpp:202, BenchmarkRunner.cpp:209, WriteImageBenchmark.cpp:511-514). They are developer tools, not filter output.

Steps to Reproduce

  1. Create an Image Geometry with dimensions 3 x 2 x 1, origin (0, 0, 0) and spacing (1e-7, 2e-7, 3e-7), and an int32 Feature Ids cell array.
  2. Run Write Avizo Uniform Coordinate with Write Binary File off, then on.
  3. Open the output files. Both headers contain BoundingBox 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000. The expected first bound is 5e-08.
  4. Run Write Avizo Rectilinear Coordinate on the same geometry with Write Binary File off. The X coordinates line is 0.000000 0.000000 0.000000; the expected second value is 1e-07.

Cause

Each place formats the value with a fixed number of decimal places or too few significant digits. For example, WriteAvizoUniformCoordinate.cpp#L69-L70:

fprintf(outputFile, "     BoundingBox %f %f %f %f %f %f\n", origin[0] + 0.5 * res[0], origin[0] + (static_cast<float64>(dims[0]) - 0.5) * res[0], origin[1] + 0.5 * res[1],
        origin[1] + (static_cast<float64>(dims[1]) - 0.5) * res[1], origin[2] + 0.5 * res[2], origin[2] + (static_cast<float64>(dims[2]) - 0.5) * res[2]);

and WriteAvizoRectilinearCoordinate.cpp#L163:

fprintf(outputFile, "%f ", origin[d] + (res[d] * i));

%f always prints six digits after the decimal point, so any value below 5e-7 is written as 0.000000. The truncating places use a precision below what the written type needs to round-trip: digits10 (6 for float32) is the decimal-to-binary guarantee, not max_digits10 (9). The Abaqus writers apply float32's 9 digits to coordinates that they compute in float64.

Verification Unit Test

1. The fixed instance: Write Avizo Uniform Coordinate. File src/Plugins/SimplnxCore/test/WriteAvizoUniformCoordinateTest.cpp. This test and the fix are being added to PR #1919.

TEST_CASE("SimplnxCore::WriteAvizoUniformCoordinateFilter: Tiny spacing bounds round trip", "[SimplnxCore][WriteAvizoUniformCoordinateFilter]")
{
  const bool binary = GENERATE(false, true);
  CAPTURE(binary);
  DataStructure dataStructure;
  const SizeVec3 dims = {3, 2, 1};
  const DataPath labelsPath = CreateUniformWriterInput(dataStructure, dims, {101, 102, 103, 104, 105, 106});
  auto* imageGeomPtr = dataStructure.getDataAs<ImageGeom>(DataPath({"Image"}));
  REQUIRE(imageGeomPtr != nullptr);
  imageGeomPtr->setOrigin({0, 0, 0});
  imageGeomPtr->setSpacing({1e-7f, 2e-7f, 3e-7f});
  const std::string actual = WriteUniformFile(dataStructure, labelsPath, binary, binary ? "UniformTinySpacingBinary.am" : "UniformTinySpacingAscii.am");
  const std::string marker = "     BoundingBox ";
  const usize boxStart = actual.find(marker);
  REQUIRE(boxStart != std::string::npos);
  const std::string boxLine = actual.substr(boxStart + marker.size(), actual.find('\n', boxStart) - boxStart - marker.size());
  INFO("BoundingBox " << boxLine);
  std::istringstream bounds(boxLine);
  const std::array<float64, 3> origin = {0, 0, 0};
  const std::array<float64, 3> spacing = {static_cast<float64>(1e-7f), static_cast<float64>(2e-7f), static_cast<float64>(3e-7f)};
  for(usize axis = 0; axis < 3; ++axis)
  {
    const std::array<float64, 2> expectedBounds = {origin[axis] + 0.5 * spacing[axis], origin[axis] + (static_cast<float64>(dims[axis]) - 0.5) * spacing[axis]};
    for(float64 expected : expectedBounds)
    {
      CAPTURE(axis, expected);
      std::string token;
      REQUIRE(static_cast<bool>(bounds >> token));
      char* endPtr = nullptr;
      const float64 value = std::strtod(token.c_str(), &endPtr);
      REQUIRE(endPtr == token.c_str() + token.size());
      REQUIRE(value > 0);
      REQUIRE(value == expected);
    }
  }
  UnitTest::CheckArraysInheritTupleDims(dataStructure);
}

Command: ctest --test-dir build/simplnx-Rel -R "WriteAvizoUniformCoordinateFilter: Tiny spacing bounds round trip" -j 1 --output-on-failure

Output before the fix (Release, macOS arm64; both encodings fail):

WriteAvizoUniformCoordinateTest.cpp:123: FAILED:
  REQUIRE( value > 0 )
with expansion:
  0.0 > 0
with messages:
  binary := false
  BoundingBox 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
  axis := 0
  expected := 0.00000005
...
  binary := true
  BoundingBox 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
test cases:  1 |  0 passed | 1 failed

After the fix the test passes.

2. Another place in the class: Write Avizo Rectilinear Coordinate. File src/Plugins/SimplnxCore/test/WriteAvizoRectilinearCoordinateTest.cpp (with #include <array>, <cstdlib>, <sstream> and simplnx/Utilities/DataStoreUtilities.hpp). The writer is unchanged from aacedb4dc.

TEST_CASE("SimplnxCore::WriteAvizoRectilinearCoordinateFilter: Tiny spacing coordinates round trip", "[SimplnxCore][WriteAvizoRectilinearCoordinateFilter]")
{
  const SizeVec3 dims = {3, 2, 1};
  const FloatVec3 origin = {0, 0, 0};
  const FloatVec3 spacing = {1e-7f, 2e-7f, 3e-7f};
  DataStructure dataStructure;
  auto* imageGeomPtr = ImageGeom::Create(dataStructure, "Image");
  REQUIRE(imageGeomPtr != nullptr);
  imageGeomPtr->setDimensions(dims);
  imageGeomPtr->setOrigin(origin);
  imageGeomPtr->setSpacing(spacing);
  const ShapeType shape = {dims[2], dims[1], dims[0]};
  auto* cellsPtr = AttributeMatrix::Create(dataStructure, "Cells", shape, imageGeomPtr->getId());
  REQUIRE(cellsPtr != nullptr);
  imageGeomPtr->setCellData(cellsPtr->getId());
  const DataPath featureIdsPath({"Image", "Cells", "FeatureIds"});
  auto* featureIdsPtr = Int32Array::Create(dataStructure, "FeatureIds", DataStoreUtilities::CreateDataStore<int32>(dataStructure, featureIdsPath, shape, {1}), cellsPtr->getId());
  REQUIRE(featureIdsPtr != nullptr);
  const std::array<int32, 6> labels = {1, 2, 3, 4, 5, 6};
  const auto writeResult = featureIdsPtr->getDataStoreRef().copyFromBuffer(0, labels);
  SIMPLNX_RESULT_REQUIRE_VALID(writeResult);

  WriteAvizoRectilinearCoordinateFilter filter;
  Arguments args = filter.getDefaultArguments();
  const fs::path path = fs::path(unit_test::k_BinaryTestOutputDir.view()) / "RectilinearTinySpacingAscii.am";
  args.insertOrAssign(WriteAvizoRectilinearCoordinateFilter::k_OutputFile_Key, std::make_any<fs::path>(path));
  args.insertOrAssign(WriteAvizoRectilinearCoordinateFilter::k_WriteBinaryFile_Key, std::make_any<bool>(false));
  args.insertOrAssign(WriteAvizoRectilinearCoordinateFilter::k_GeometryPath_Key, std::make_any<DataPath>(DataPath({"Image"})));
  args.insertOrAssign(WriteAvizoRectilinearCoordinateFilter::k_FeatureIdsArrayPath_Key, std::make_any<DataPath>(featureIdsPath));
  const auto preflightResult = filter.preflight(dataStructure, args);
  SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions);
  const auto executeResult = filter.execute(dataStructure, args);
  SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result);

  std::ifstream file(path);
  REQUIRE(file.is_open());
  std::string line;
  bool foundCoordinates = false;
  while(std::getline(file, line))
  {
    if(line == "@2 # x coordinates, then y, then z")
    {
      foundCoordinates = true;
      break;
    }
  }
  REQUIRE(foundCoordinates);
  REQUIRE(static_cast<bool>(std::getline(file, line)));
  INFO("X coordinates: " << line);
  std::istringstream coordinates(line);
  for(usize i = 0; i < dims[0]; ++i)
  {
    const float32 expected = origin[0] + (spacing[0] * i);
    CAPTURE(i, expected);
    std::string token;
    REQUIRE(static_cast<bool>(coordinates >> token));
    char* endPtr = nullptr;
    const float64 value = std::strtod(token.c_str(), &endPtr);
    REQUIRE(endPtr == token.c_str() + token.size());
    REQUIRE(value == static_cast<float64>(expected));
    if(i > 0)
    {
      REQUIRE(value > 0);
    }
  }
  UnitTest::CheckArraysInheritTupleDims(dataStructure);
}

Command: ctest --test-dir build/simplnx-Rel -R "WriteAvizoRectilinearCoordinateFilter: Tiny spacing coordinates round trip" -j 1 --output-on-failure

Output (Release, macOS arm64):

WriteAvizoRectilinearCoordinateTest.cpp:234: FAILED:
  REQUIRE( value == static_cast<float64>(expected) )
with expansion:
  0.0 == 0.0000001
with messages:
  X coordinates: 0.000000 0.000000 0.000000
  i := 1
test cases:  1 |  0 passed | 1 failed

Proposed Fix

Use the pattern from the fixed instance in PR #1919 at every place above:

-  fprintf(outputFile, "     BoundingBox %f %f %f %f %f %f\n", ...);
+  fmt::print(outputFile, "     BoundingBox {} {} {} {} {} {}\n", ...);
  • fmt: plain {} (shortest round-trip text) for the type actually written.
  • printf: fprintf(f, "%.*g", std::numeric_limits<T>::max_digits10, static_cast<double>(value)).
  • Streams: std::defaultfloat << std::setprecision(std::numeric_limits<T>::max_digits10). Reset std::fixed explicitly.
  • Python: repr(value); drop the rounding in _encode_float().
  • Format the computed type: a float64 expression of float32 inputs needs 17 digits or {}.

Where a file format really fixes the precision, name that requirement in the code. Each place should get a round-trip test like the two above (tiny nonzero values, and values that need all 9 or 17 digits). PR #1919 adds the knowledge-base entry "Write floating-point data as text that reads back to the same value" to docs/Coding_Knowledge_Base.md.

Notes

  • Found during the review of PR #1919 by an audit of floating-point text output in simplnx, SimplnxOoc, DREAM3D-NX and EbsdLib. SimplnxOoc has no affected place. DREAM3D-NX and EbsdLib places are reported in BlueQuartzSoftware/DREAM3DNX#1238 and BlueQuartzSoftware/EbsdLib#65.
  • Open PRs that touch some of these writers keep the lossy formats: #1764 (both Avizo writers), #1915 (INL symmetry codes, ODF angle file delimiter).
  • Related, not part of this issue: NumberToStringFilterParameterConverter (StringParameter.cpp) converts legacy SIMPL numeric values with std::to_string, which also writes six decimal places.
Lenguaje dominante
C++
Estrellas
18
Forks
15
Merge medio
4 d 23 h
PR fusionados (30 d)
42

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de BlueQuartzSoftware/simplnx

Todos los issues de BlueQuartzSoftware/simplnx

Issues similares

Más issues de C++

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.