[BUG] `fab table schema` failure

Abierto
  • ¿Algo no está bien?
#275 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 3 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
48/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python
Área
cli, databases

Línea de trabajo

Comienza con el comando fab table schema y sigue cómo enumera los archivos _delta_log/*.json conservados y extrae metaData.schemaString. Compara esa ruta con los archivos Parquet de checkpoint descritos en el issue y, después, verifica el comportamiento con una tabla cuyo JSON de metadatos se haya eliminado. Se considera terminado cuando el comando devuelve el esquema actual de esa tabla Delta válida en lugar de InvalidDeltaTable.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

ai:bug-triage:unable-to-process bug
Bug Description

Docs claim that fab table schema has full support for Lakehouse tables:

https://microsoft.github.io/fabric-cli/commands/tables/#:~:text=.-,Lakehouse,-(full%20support)

However, the table schema command can fail for a valid Delta table when the original Delta transaction-log JSON file containing the metaData action has already been removed by Delta log retention/checkpointing.

I have two tables in the same Lakehouse:

  • table_a — fab table schema works correctly.
  • table_b — fab table schema returns InvalidDeltaTable.

Both tables are valid Delta tables and are accessible from Fabric/Spark.

The relevant difference appears to be their _delta_log history:

  • The working table still contains the initial 00000000000000000000.json file, which contains a metaData object with schemaString object inside.
  • The failing table has older JSON transaction logs removed and the currently retained JSON files start around version 00000000000000000260.json. None of the retained JSON files contain a metaData object. The older table state is represented through Delta checkpoint files.

Based on the current Fabric CLI implementation, fab table schema scans JSON files inside _delta_log and searches for a metaData.schemaString. It does not appear to reconstruct the Delta metadata from checkpoint Parquet files. As a result, a valid Delta table can incorrectly produce an InvalidDeltaTable error.

fabric-cli version

1.6.1

Python version

3.11.9

Operating System

Windows

CLI Mode

Command line mode

Authentication Method

Interactive browser login

Steps to Reproduce
  1. Create a Lakehouse containing a Delta table.
  2. Allow the table to accumulate enough Delta transaction history for older JSON log files containing the original metaData action to be removed according to Delta log retention/checkpointing.
  3. Verify that the table remains a valid Delta table and can be queried normally from Fabric/Spark.
  4. Verify that the remaining _delta_log/*.json files do not contain a metaData action, while checkpoint files exist.
  5. Run:
fab table schema testworkspace.Workspace/testlakehouse.Lakehouse/Tables/dbo/problem_table

The command returns:

[InvalidDeltaTable] Failed to extract the table schema. Please ensure the path points to a valid Delta table

For comparison, running the same command against another table in the same Lakehouse whose _delta_log still contains the original metadata JSON works correctly:

fab table schema testworkspace.Workspace/testlakehouse.Lakehouse/Tables/dbo/working_table
Expected Behavior

fab table schema should return the schema for any valid supported Delta table regardless of whether the original JSON transaction-log file containing the metaData action is still retained.

Delta tables remain valid after checkpointing and transaction-log retention, so schema extraction should work from the current Delta snapshot rather than depend on historical JSON files being present.

Actual Behavior

fab table schema returns:

[InvalidDeltaTable] Failed to extract the table schema. Please ensure the path points to a valid Delta table

even though the table itself is a valid Delta table.

The command succeeds for another table in the same Lakehouse where a retained _delta_log/*.json file still contains a metaData object.

Additional Context

Looking at the current fabric-cli implementation of table schema, the command appears to:

  1. Open <table>/_delta_log.
  2. Enumerate .json transaction-log files.
  3. Read those JSON files in reverse order.
  4. Search for a metaData action.
  5. Extract metaData.schemaString.
  6. Return InvalidDeltaTable if no such JSON action is found.

This approach does not seem to account for Delta checkpoints.

In the failing table, the oldest retained JSON transaction-log file is approximately:

00000000000000000260.json

and none of the currently retained JSON files contain a metaData action.

The working table still has:

00000000000000000000.json

which contains the expected metaData.schemaString.

This makes the behavior dependent on Delta transaction-log retention/history rather than on whether the table is actually a valid Delta table.

Possible Solution

Instead of deriving the schema only by searching retained _delta_log/*.json files for a metaData action, fab table schema could reconstruct the current Delta snapshot, including metadata from checkpoint Parquet files.

Alternatively, the implementation could use a Delta-aware API/library or another Fabric metadata endpoint capable of returning the current table schema independently of transaction-log retention.

The InvalidDeltaTable error should also ideally be reserved for genuinely invalid/non-Delta tables. If schema extraction fails because the required metadata is unavailable through the current CLI implementation, a more specific error would make troubleshooting easier.

Lenguaje dominante
Python
Estrellas
177
Forks
65
Merge medio
3 d 12 h
PR fusionados (30 d)
8

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de microsoft/fabric-cli

Todos los issues de microsoft/fabric-cli

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.