Questions about ASR frontend, specifically Librispeech, feature normalization

Aperta
  • Qualcosa non va?
#255 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
25/100
Tipo di issue
Documentazione
Chiarezza
Abbastanza chiara
Stato di attività
Ferma
Stack tecnologico
python, tensorflow
Ambito
backend, machine-learning

Direzione di ricerca

Read the Librispeech parameters, ASR encoder and model, create_asr_features.py, ExtractLogMelFeatures, and MelAsrFrontend linked in the issue. Trace whether normalization and PCM scaling options reach the frontend and what values it passes to the convolutional network. Done means the normalization and expected feature scale questions are answered from the code; the thread has had no response for years.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Hi,

I'm checking your ASR frontend, specifically the Librispeech audio feature extraction, and have some questions.

References: Librispeech params, ASR encoder, ASR model, create ASR features, ExtractLogMelFeatures, MelAsrFrontend

I don't see that feature normalization is used in any way. Is that right? So the initial convolutional network gets in unnormalized features?

There is an option per_bin_mean/per_bin_stddev but this no not used?

I wonder a bit about the 32768 factor. As input to MelAsrFrontend, you expect the values to be in range [-32768..32768]? But shouldn't you divide by 32768 at some point in the pipeline? I see that there is the option use_divide_stream which would do that, but this is also not used. Is that right? There is the src_pcm_scale option in AsrFrontendConfig but I also don't see this being used anywhere?

What numerical values do you expect for the unnormalized features? By using your code, I end up with these statistics for the 80-dim log-mel features:

  Mean:
[6.571567  6.8085003 6.874515  6.9014635 7.014142  7.0960007 7.154998
 7.2196846 7.2804766 7.3604364 7.5162773 7.6646695 7.7035103 7.637239
 7.473917  7.3341737 7.334874  7.225426  7.2850995 7.330272  7.3985167
 7.37795   7.296777  7.3162036 7.3547645 7.3710914 7.365237  7.408713
 7.4906807 7.561958  7.7213554 7.8515496 7.96462   8.009668  8.006353
 8.028718  8.093952  8.1421175 8.137211  8.187237  8.285318  8.338231
 8.355931  8.366059  8.428564  8.457295  8.45581   8.557338  8.637221
 8.672467  8.788814  8.787889  8.667367  8.60016   8.601487  8.72531
 8.768343  8.765634  8.858865  8.845884  8.873101  8.831113  8.854687
 8.934903  8.90494   8.948756  8.950672  8.937693  8.968388  9.017417
 8.98474   8.953554  9.017656  9.010184  9.012505  9.011251  8.993835
 8.985477  8.975625  8.9696865]
  Std dev:
[2.0369635 2.096931  2.1227295 2.1050122 2.0581489 2.0478609 2.0781274
 2.111383  2.1236782 2.1237605 2.153045  2.1902032 2.1789725 2.1117716
 2.0372825 2.0394926 2.0196817 1.947741  1.9082164 1.9618858 1.960082
 1.8687389 1.791357  1.7570691 1.7195601 1.7076842 1.7179631 1.7446057
 1.7433041 1.7451456 1.7246099 1.6775978 1.7071667 1.7530293 1.7220892
 1.6955647 1.668047  1.701594  1.7442849 1.7660844 1.7131468 1.6875786
 1.7154576 1.651276  1.6555542 1.7089554 1.8156201 1.8667307 1.7612113
 1.7350646 1.7320263 1.7157421 1.7632391 1.8203604 1.848493  1.8341905
 1.7994169 1.7613424 1.8461764 1.8262035 1.8009105 1.8069444 1.7533652
 1.7506559 1.7582294 1.8100908 1.8118345 1.8222826 1.8585421 1.8347679
 1.833038  1.8561351 1.838355  1.8722489 1.9128579 1.9025422 1.8568758
 1.864883  1.85355   1.8346307]
  Min/max:
[0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         1.081316   1.196871
 1.2115035  0.81268585 0.6653169  0.23646453 1.0013729  0.78377634
 1.0490758  0.37899855 0.07468933 0.02383682 0.         0.
 1.5028754  2.9381104  3.1084766  2.3929093  1.4824951  0.50826544
 1.0819429  2.375446   2.7917457  2.9367018  2.50226    2.8797576
 2.7869585  3.0858362  2.721441   2.890882   1.1783836  0.7760011
 3.267177   3.534093   2.9064052  3.580449   2.0504556  0.9798477
 0.47552437 0.4765737  3.0911071  2.3247538  2.8063457  2.8832946
 2.8879871  3.6785111  3.0960853  3.9359105  2.2698429  2.3523924
 4.1431756  1.4312835  2.4378788  4.199857   4.1510873  3.359309
 3.776203   3.579252   3.8776345  3.7463636  3.3691368  0.47623247
 3.9832573  3.867248  ] /
[10.206617  10.944173  11.139801  11.130556  11.729167  11.843884
 11.478071  11.72813   11.813193  12.02717   12.456151  12.265145
 12.660903  12.499597  12.844959  13.138891  13.047909  12.737233
 13.1444235 12.870635  12.814672  12.34872   12.536846  12.46696
 11.983264  12.251521  12.125826  12.2784195 12.577657  12.5084095
 12.271909  12.40977   13.049571  13.354869  13.52251   13.364965
 13.063794  13.076281  13.131835  12.594194  12.473833  12.474021
 12.613151  12.120035  12.36114   12.519782  13.078986  13.060618
 13.451843  13.620312  13.377903  13.563233  13.519144  13.6905985
 13.992317  13.707613  13.95073   14.305102  14.90581   15.447515
 15.098798  14.792583  14.573465  14.620636  14.31828   14.933836
 14.767882  14.567602  14.487499  14.37275   14.498172  14.244799
 14.251987  14.410744  14.358433  14.431413  14.4958935 14.791843
 14.440226  14.67428  ]

Does that look right? And these are the values the conv net gets? I would have expected that you need to normalize them in some way.

Lingua principale
Python
Stelle
2.9k
Fork
452
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di tensorflow/lingvo

Tutte le issue di tensorflow/lingvo

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.