BUG: CTF writer writes the header and the first data row on one line

Aperta
#66 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
76/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
cpp
Ambito
data

Direzione di ricerca

Read the header handling in Source/EbsdLib/IO/HKL/CtfReader.cpp, especially getHeaderLines(), readData(), and writeFile(), then review Source/Test/CtfReaderTest.cpp and the existing TestWriteCtfFile case. Run the proposed WrittenFileReadsBack test to confirm the header retains its line breaks and the written file can be read again; check any reader that depends on H5CtfImporter’s OriginalHeader dataset before changing its stored format.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug

Problem

CtfReader stores the header of a .ctf file without its line breaks, and CtfReader::writeFile() writes that stored header as it is. The written file has every header line, the column-name line and the first data row joined on its first line. CtfReader cannot read that file back: readFile() throws std::invalid_argument ("stoi: no conversion").

Expected: a file written by writeFile() has the same header lines as the input and reads back with readFile(). Actual: its first line holds the whole header, the column names and the first data row (see the test output below), and reading it throws.

Occurs at 34844ed31dfb8660c67edb6e019ca708e04beeb4 (develop).

Steps to Reproduce

  1. Read Data/EbsdTestFiles/Test_US_1.ctf with CtfReader::readFile().
  2. Write it with CtfReader::writeFile() to a new path.
  3. Compare the first line of the written file with the first line of the input (Channel Text File): the written line holds the whole header and the first data row.
  4. Read the written file with a new CtfReader: readFile() throws "stoi: no conversion".

Cause

getHeaderLines() reads each header line without its newline and appends it to the original header with nothing in between (CtfReader.cpp#L873-L876, and the phase lines at CtfReader.cpp#L888-L890):

::safeGetline(reader, buf);
// Append the line to the complete header. The trailing newline is already removed at this point
appendOriginalHeader(std::string(buf));

readData() adds the column-name line the same way (CtfReader.cpp#L501-L504):

std::getline(in, buf);
std::string originalHeader = getOriginalHeader();
originalHeader = originalHeader + buf;
setOriginalHeader(originalHeader);

writeFile() then writes the stored header verbatim, followed directly by the data rows (CtfReader.cpp#L1013-L1014):

std::string header = getOriginalHeader();
fwrite(header.c_str(), 1, header.size(), f);

When the written file is read, getHeaderLines() sees one line that contains Phases, splits it on tabs and passes the wrong token to std::stoi. The existing test ebsdlib::CtfReaderTest::TestWriteCtfFile checks only that writeFile() returns 0, so it does not catch this.

Verification Unit Test

File: Source/Test/CtfReaderTest.cpp.

TEST_CASE("ebsdlib::CtfReaderTest::WrittenFileReadsBack", "[EbsdLib][CtfReaderTest]")
{
  CtfReader reader;
  reader.setFileName(ebsdlib::unit_test::CtfReaderTest::USInputFile1);
  REQUIRE(reader.readFile() == 0);

  std::stringstream filePath;
  filePath << ebsdlib::unit_test::k_TestTempDir << "/CTF_WriteFile_read_back_test.ctf";
  REQUIRE(reader.writeFile(filePath.str()) == 0);

  // The written header keeps one line per header entry, as in the input file.
  std::ifstream input(ebsdlib::unit_test::CtfReaderTest::USInputFile1);
  std::ifstream output(filePath.str());
  REQUIRE(input.is_open());
  REQUIRE(output.is_open());
  std::string inputFirstLine;
  std::string outputFirstLine;
  std::getline(input, inputFirstLine);
  std::getline(output, outputFirstLine);
  CHECK(outputFirstLine == inputFirstLine);

  CtfReader readBack;
  readBack.setFileName(filePath.str());
  int err = -1;
  REQUIRE_NOTHROW(err = readBack.readFile());
  REQUIRE(err == 0);
  REQUIRE(readBack.getNumberOfElements() == reader.getNumberOfElements());
  REQUIRE(readBack.getXCells() == reader.getXCells());
  REQUIRE(readBack.getYCells() == reader.getYCells());
}

Command:

ctest --test-dir build/Release -R "^EbsdLib::ebsdlib::CtfReaderTest::WrittenFileReadsBack$" -j 1 --output-on-failure --timeout 1500 --no-tests=error

Output on 34844ed (Release, macOS arm64):

Source/Test/CtfReaderTest.cpp:301: FAILED:
  CHECK( outputFirstLine == inputFirstLine )
with expansion:
  "Channel Text FilePrj	C:\NathB\EBSD processing\701-600-1-f.cprAuthor
  [Unknown]JobMode	GridXCells	40YCells	5XStep	0.5YStep	0.5AcqE1	0AcqE2	0AcqE3
  0Euler angles refer to Sample Coordinate system (CS0)!	Mag	200	Coverage	100
  Device	0	KV	15	TiltAngle	70	TiltAxis	0Phases	13.231;3.231;5.148	90;90;120
  Zirc-alloy4	9	0	0_5.0.6.0	-67395467	[Zr4.cry]
Phase	X	Y	Bands	Error	Euler1
  Euler2	Euler3	MAD	BC	BS
0	0.0000	0.0000	0	3	0.0000	0.0000	0.0000	0.0000	85	0
  "
  ==
  "Channel Text File
"

Source/Test/CtfReaderTest.cpp:306: FAILED:
  REQUIRE_NOTHROW( err = readBack.readFile() )
due to unexpected exception with message:
  stoi: no conversion

===============================================================================
test cases: 1 | 1 failed
assertions: 6 | 4 passed | 2 failed

Proposed Fix

Keep the line breaks when the original header is stored, so the stored header is the file's header text:

  • getHeaderLines(): appendOriginalHeader(buf + "\n"); in both loops. The phase loop reads with std::getline, so a CRLF file keeps its \r before the added \n.
  • readData(): originalHeader = originalHeader + buf + "\n"; for the column-name line.

writeFile() then writes a header that ends with a line break, and the first data row starts on its own line. The test above then passes. H5CtfImporter also stores getOriginalHeader() as the OriginalHeader dataset of .h5ebsd files, so it would get the original header with its line breaks too. Check any reader of that dataset that depends on the current joined form.

Notes

  • Found while writing the verification test for BlueQuartzSoftware/EbsdLib#65 (CTF float precision); that test sets the header with setOriginalHeader() to get past this defect.
  • By code reading, .h5ebsd files written by H5CtfImporter store the same joined header in Header/OriginalHeader (H5CtfImporter.cpp#L298-L299). This was not tested.
Lingua principale
C++
Stelle
8
Fork
16
Merge medio
4h 35m
PR unite (30g)
3

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di BlueQuartzSoftware/EbsdLib

Tutte le issue di BlueQuartzSoftware/EbsdLib

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.