BUG: CTF writer writes the header and the first data row on one line

Abierto
#66 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
76/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
cpp
Área
data

Línea de trabajo

Read the header handling in Source/EbsdLib/IO/HKL/CtfReader.cpp, especially getHeaderLines(), readData(), and writeFile(), then review Source/Test/CtfReaderTest.cpp and the existing TestWriteCtfFile case. Run the proposed WrittenFileReadsBack test to confirm the header retains its line breaks and the written file can be read again; check any reader that depends on H5CtfImporter’s OriginalHeader dataset before changing its stored format.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug

Problem

CtfReader stores the header of a .ctf file without its line breaks, and CtfReader::writeFile() writes that stored header as it is. The written file has every header line, the column-name line and the first data row joined on its first line. CtfReader cannot read that file back: readFile() throws std::invalid_argument ("stoi: no conversion").

Expected: a file written by writeFile() has the same header lines as the input and reads back with readFile(). Actual: its first line holds the whole header, the column names and the first data row (see the test output below), and reading it throws.

Occurs at 34844ed31dfb8660c67edb6e019ca708e04beeb4 (develop).

Steps to Reproduce

  1. Read Data/EbsdTestFiles/Test_US_1.ctf with CtfReader::readFile().
  2. Write it with CtfReader::writeFile() to a new path.
  3. Compare the first line of the written file with the first line of the input (Channel Text File): the written line holds the whole header and the first data row.
  4. Read the written file with a new CtfReader: readFile() throws "stoi: no conversion".

Cause

getHeaderLines() reads each header line without its newline and appends it to the original header with nothing in between (CtfReader.cpp#L873-L876, and the phase lines at CtfReader.cpp#L888-L890):

::safeGetline(reader, buf);
// Append the line to the complete header. The trailing newline is already removed at this point
appendOriginalHeader(std::string(buf));

readData() adds the column-name line the same way (CtfReader.cpp#L501-L504):

std::getline(in, buf);
std::string originalHeader = getOriginalHeader();
originalHeader = originalHeader + buf;
setOriginalHeader(originalHeader);

writeFile() then writes the stored header verbatim, followed directly by the data rows (CtfReader.cpp#L1013-L1014):

std::string header = getOriginalHeader();
fwrite(header.c_str(), 1, header.size(), f);

When the written file is read, getHeaderLines() sees one line that contains Phases, splits it on tabs and passes the wrong token to std::stoi. The existing test ebsdlib::CtfReaderTest::TestWriteCtfFile checks only that writeFile() returns 0, so it does not catch this.

Verification Unit Test

File: Source/Test/CtfReaderTest.cpp.

TEST_CASE("ebsdlib::CtfReaderTest::WrittenFileReadsBack", "[EbsdLib][CtfReaderTest]")
{
  CtfReader reader;
  reader.setFileName(ebsdlib::unit_test::CtfReaderTest::USInputFile1);
  REQUIRE(reader.readFile() == 0);

  std::stringstream filePath;
  filePath << ebsdlib::unit_test::k_TestTempDir << "/CTF_WriteFile_read_back_test.ctf";
  REQUIRE(reader.writeFile(filePath.str()) == 0);

  // The written header keeps one line per header entry, as in the input file.
  std::ifstream input(ebsdlib::unit_test::CtfReaderTest::USInputFile1);
  std::ifstream output(filePath.str());
  REQUIRE(input.is_open());
  REQUIRE(output.is_open());
  std::string inputFirstLine;
  std::string outputFirstLine;
  std::getline(input, inputFirstLine);
  std::getline(output, outputFirstLine);
  CHECK(outputFirstLine == inputFirstLine);

  CtfReader readBack;
  readBack.setFileName(filePath.str());
  int err = -1;
  REQUIRE_NOTHROW(err = readBack.readFile());
  REQUIRE(err == 0);
  REQUIRE(readBack.getNumberOfElements() == reader.getNumberOfElements());
  REQUIRE(readBack.getXCells() == reader.getXCells());
  REQUIRE(readBack.getYCells() == reader.getYCells());
}

Command:

ctest --test-dir build/Release -R "^EbsdLib::ebsdlib::CtfReaderTest::WrittenFileReadsBack$" -j 1 --output-on-failure --timeout 1500 --no-tests=error

Output on 34844ed (Release, macOS arm64):

Source/Test/CtfReaderTest.cpp:301: FAILED:
  CHECK( outputFirstLine == inputFirstLine )
with expansion:
  "Channel Text FilePrj	C:\NathB\EBSD processing\701-600-1-f.cprAuthor
  [Unknown]JobMode	GridXCells	40YCells	5XStep	0.5YStep	0.5AcqE1	0AcqE2	0AcqE3
  0Euler angles refer to Sample Coordinate system (CS0)!	Mag	200	Coverage	100
  Device	0	KV	15	TiltAngle	70	TiltAxis	0Phases	13.231;3.231;5.148	90;90;120
  Zirc-alloy4	9	0	0_5.0.6.0	-67395467	[Zr4.cry]
Phase	X	Y	Bands	Error	Euler1
  Euler2	Euler3	MAD	BC	BS
0	0.0000	0.0000	0	3	0.0000	0.0000	0.0000	0.0000	85	0
  "
  ==
  "Channel Text File
"

Source/Test/CtfReaderTest.cpp:306: FAILED:
  REQUIRE_NOTHROW( err = readBack.readFile() )
due to unexpected exception with message:
  stoi: no conversion

===============================================================================
test cases: 1 | 1 failed
assertions: 6 | 4 passed | 2 failed

Proposed Fix

Keep the line breaks when the original header is stored, so the stored header is the file's header text:

  • getHeaderLines(): appendOriginalHeader(buf + "\n"); in both loops. The phase loop reads with std::getline, so a CRLF file keeps its \r before the added \n.
  • readData(): originalHeader = originalHeader + buf + "\n"; for the column-name line.

writeFile() then writes a header that ends with a line break, and the first data row starts on its own line. The test above then passes. H5CtfImporter also stores getOriginalHeader() as the OriginalHeader dataset of .h5ebsd files, so it would get the original header with its line breaks too. Check any reader of that dataset that depends on the current joined form.

Notes

  • Found while writing the verification test for BlueQuartzSoftware/EbsdLib#65 (CTF float precision); that test sets the header with setOriginalHeader() to get past this defect.
  • By code reading, .h5ebsd files written by H5CtfImporter store the same joined header in Header/OriginalHeader (H5CtfImporter.cpp#L298-L299). This was not tested.
Lenguaje dominante
C++
Estrellas
8
Forks
16
Merge medio
4 h 35 min
PR fusionados (30 d)
3

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de BlueQuartzSoftware/EbsdLib

Todos los issues de BlueQuartzSoftware/EbsdLib

Issues similares

Más issues de C++

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.