Delta protocol version upgrade is allowing any invalid version number for minWriterVersion value

Issue:

I have tried to upgrade the delta protocol on a newly created delta table by providing a invalid bigger 3 digit minWriterVersion number and surprisingly it accepted and updated on the 00000000000000000001.json . This will make entire delta table unusable after upgrade.

Though it rejecting further upgrades with any value with below exception :

com.databricks.sql.transaction.tahoe.actions.InvalidProtocolVersionException: Delta protocol version is too new for this version of the Databricks Runtime. Please upgrade to a newer release.

Tested on: Databricks Runtime : DBR 9.1 LTS , SPARK 3.1.2 (on Azure) Delta created on : Azure Gen 2 storage Delta Lake : 1.0.0

Steps to reproduce:

1. Create sample dataframe and written as delta :

df=spark.createDataFrame( 

  [(1,'test1',33,'Q1'), 

   (2,'test2',48,'Q1'), 

   (3,'test3',22,'Q1'), 

   (4,'test4',88,'Q2'), 

   (5,'test5',None,'Q2'), 

   (6,'test6',42,'Q2') 

  ],'id int,test_name string,score int,quarter string' 

) 

 df_to_write=df.repartition(1,['id']) 

df_to_write.write.format('delta').mode('overwrite').save('abfss://test_container@test_account.dfs.core.windows.net/global/test_data/delta_test/sample1') 

2. Read the written data using python delta library's class method delta.tables.DeltaTable.forPath:

from delta.tables import DeltaTable 

dt=DeltaTable.forPath(spark,'abfss://test_container@test_account.dfs.core.windows.net/global/test_data/delta_test/sample1') 

3. Upgraded the delta table protocol version:

dt.upgradeTableProtocol(1,101)

4. The invalid minWriter version is updated on 00000000000000000001.json: