Increasing the time it takes to learn the model in one epoch

The environment I'm using is on vultr GPU instance CPU : 8core ram : 40G GPU : A40 VRAM : 16GB using RVC on a docker in this environment

Until recently, it took about 20 seconds per fork to run 200 for less than an hour when I was learning a 7-8 minute voice 3 dataset, but yesterday, I put 190 data less than 10 seconds into the dataset for testing, but what's the problem since it took at least 50 seconds per fork to put any form of data (7 minutes 3, 4 minutes 5, etc.) in it?

I don't understand that even if I put the same 4 minutes and 5 pieces of data in the 3070 (6 batches) I'm using at home and rotate it, it takes 35 seconds per epoch, but it takes a higher learning time on a better graphics card.

I've tried the ones below.

• Rebuild Docker Images
• Run RVC directly on the server
• Rebooting the Server

I'll put two logs in the bottom

2023-11-12 17:32:12 | INFO | configs.config | Found GPU NVIDIA A40-16Q, force to fp32
overwrite preprocess and configs.json
is_half:False, device:cuda:0
2023-11-12 17:32:13 | INFO | __main__ | Use Language: en_US
Running on local URL:  http://0.0.0.0:7865
2023-11-12 17:32:39 | INFO | httpx | HTTP Request: POST http://localhost:7865/api/predict "HTTP/1.1 200 OK"
2023-11-12 17:32:39 | INFO | __main__ | "python3" infer/modules/train/preprocess.py "/root/Retrieval-based-Voice-Conversion-WebUI/data/director_one_combine" 40000 4 "/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine" False 3.7
['infer/modules/train/preprocess.py', '/root/Retrieval-based-Voice-Conversion-WebUI/data/director_one_combine', '40000', '4', '/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine', 'False', '3.7']
start preprocess
['infer/modules/train/preprocess.py', '/root/Retrieval-based-Voice-Conversion-WebUI/data/director_one_combine', '40000', '4', '/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine', 'False', '3.7']
/root/Retrieval-based-Voice-Conversion-WebUI/data/director_one_combine/4.wav->Suc.
/root/Retrieval-based-Voice-Conversion-WebUI/data/director_one_combine/2.wav->Suc.
/root/Retrieval-based-Voice-Conversion-WebUI/data/director_one_combine/1.wav->Suc.
/root/Retrieval-based-Voice-Conversion-WebUI/data/director_one_combine/3.wav->Suc.
end preprocess
2023-11-12 17:32:47 | INFO | __main__ | start preprocess
['infer/modules/train/preprocess.py', '/root/Retrieval-based-Voice-Conversion-WebUI/data/director_one_combine', '40000', '4', '/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine', 'False', '3.7']
/root/Retrieval-based-Voice-Conversion-WebUI/data/director_one_combine/4.wav->Suc.
/root/Retrieval-based-Voice-Conversion-WebUI/data/director_one_combine/2.wav->Suc.
/root/Retrieval-based-Voice-Conversion-WebUI/data/director_one_combine/1.wav->Suc.
/root/Retrieval-based-Voice-Conversion-WebUI/data/director_one_combine/3.wav->Suc.
end preprocess

2023-11-12 17:32:47 | INFO | httpx | HTTP Request: POST http://localhost:7865/api/predict "HTTP/1.1 200 OK"
2023-11-12 17:32:47 | INFO | __main__ | "python3" infer/modules/train/extract/extract_f0_rmvpe.py 2 0 0 "/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine" False 
2023-11-12 17:32:47 | INFO | __main__ | "python3" infer/modules/train/extract/extract_f0_rmvpe.py 2 1 0 "/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine" False 
['infer/modules/train/extract/extract_f0_rmvpe.py', '2', '1', '0', '/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine', 'False']
todo-f0-201
f0ing,now-0,all-201,-/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine/1_16k_wavs/0_10.wav
['infer/modules/train/extract/extract_f0_rmvpe.py', '2', '0', '0', '/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine', 'False']
todo-f0-202
f0ing,now-0,all-202,-/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine/1_16k_wavs/0_1.wav
Loading rmvpe model
Loading rmvpe model
f0ing,now-40,all-202,-/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine/1_16k_wavs/0_70.wav
f0ing,now-40,all-201,-/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine/1_16k_wavs/0_71.wav
f0ing,now-80,all-202,-/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine/1_16k_wavs/1_53.wav
f0ing,now-80,all-201,-/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine/1_16k_wavs/1_55.wav
f0ing,now-120,all-201,-/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine/1_16k_wavs/2_18.wav
f0ing,now-120,all-202,-/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine/1_16k_wavs/2_17.wav
f0ing,now-160,all-201,-/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine/1_16k_wavs/3_12.wav
f0ing,now-160,all-202,-/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine/1_16k_wavs/3_11.wav
f0ing,now-200,all-201,-/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine/1_16k_wavs/3_97.wav
f0ing,now-200,all-202,-/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine/1_16k_wavs/3_96.wav
2023-11-12 17:33:00 | INFO | __main__ | ['infer/modules/train/extract/extract_f0_rmvpe.py', '2', '1', '0', '/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine', 'False']
todo-f0-201
f0ing,now-0,all-201,-/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine/1_16k_wavs/0_10.wav
['infer/modules/train/extract/extract_f0_rmvpe.py', '2', '0', '0', '/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine', 'False']
todo-f0-202
f0ing,now-0,all-202,-/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine/1_16k_wavs/0_1.wav
f0ing,now-40,all-202,-/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine/1_16k_wavs/0_70.wav
f0ing,now-40,all-201,-/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine/1_16k_wavs/0_71.wav
f0ing,now-80,all-202,-/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine/1_16k_wavs/1_53.wav
f0ing,now-80,all-201,-/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine/1_16k_wavs/1_55.wav
f0ing,now-120,all-201,-/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine/1_16k_wavs/2_18.wav
f0ing,now-120,all-202,-/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine/1_16k_wavs/2_17.wav
f0ing,now-160,all-201,-/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine/1_16k_wavs/3_12.wav
f0ing,now-160,all-202,-/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine/1_16k_wavs/3_11.wav
f0ing,now-200,all-201,-/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine/1_16k_wavs/3_97.wav
f0ing,now-200,all-202,-/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine/1_16k_wavs/3_96.wav

2023-11-12 17:33:00 | INFO | __main__ | "python3" infer/modules/train/extract_feature_print.py cuda:0 1 0 0 "/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine" v2
['infer/modules/train/extract_feature_print.py', 'cuda:0', '1', '0', '0', '/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine', 'v2']
/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine
load model(s) from assets/hubert/hubert_base.pt
2023-11-12 17:33:02 | INFO | fairseq.tasks.hubert_pretraining | current directory is /root/Retrieval-based-Voice-Conversion-WebUI
2023-11-12 17:33:02 | INFO | fairseq.tasks.hubert_pretraining | HubertPretrainingTask Config {'_name': 'hubert_pretraining', 'data': 'metadata', 'fine_tuning': False, 'labels': ['km'], 'label_dir': 'label', 'label_rate': 50.0, 'sample_rate': 16000, 'normalize': False, 'enable_padding': False, 'max_keep_size': None, 'max_sample_size': 250000, 'min_sample_size': 32000, 'single_target': False, 'random_crop': True, 'pad_audio': False}
2023-11-12 17:33:02 | INFO | fairseq.models.hubert.hubert | HubertModel Config: {'_name': 'hubert', 'label_rate': 50.0, 'extractor_mode': default, 'encoder_layers': 12, 'encoder_embed_dim': 768, 'encoder_ffn_embed_dim': 3072, 'encoder_attention_heads': 12, 'activation_fn': gelu, 'layer_type': transformer, 'dropout': 0.1, 'attention_dropout': 0.1, 'activation_dropout': 0.0, 'encoder_layerdrop': 0.05, 'dropout_input': 0.1, 'dropout_features': 0.1, 'final_dim': 256, 'untie_final_proj': True, 'layer_norm_first': False, 'conv_feature_layers': '[(512,10,5)] + [(512,3,2)] * 4 + [(512,2,2)] * 2', 'conv_bias': False, 'logit_temp': 0.1, 'target_glu': False, 'feature_grad_mult': 0.1, 'mask_length': 10, 'mask_prob': 0.8, 'mask_selection': static, 'mask_other': 0.0, 'no_mask_overlap': False, 'mask_min_space': 1, 'mask_channel_length': 10, 'mask_channel_prob': 0.0, 'mask_channel_selection': static, 'mask_channel_other': 0.0, 'no_mask_channel_overlap': False, 'mask_channel_min_space': 1, 'conv_pos': 128, 'conv_pos_groups': 16, 'latent_temp': [2.0, 0.5, 0.999995], 'skip_masked': False, 'skip_nomask': False, 'checkpoint_activations': False, 'required_seq_len_multiple': 2, 'depthwise_conv_kernel_size': 31, 'attn_type': '', 'pos_enc_type': 'abs', 'fp16': False}
/usr/local/lib/python3.10/dist-packages/torch/nn/utils/weight_norm.py:30: UserWarning: torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.
  warnings.warn("torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.")
move model to cuda
all-feature-403
now-403,all-0,0_1.wav,(184, 768)
now-403,all-40,0_26.wav,(54, 768)
now-403,all-80,0_70.wav,(119, 768)
now-403,all-120,1_110.wav,(95, 768)
now-403,all-160,1_53.wav,(184, 768)
now-403,all-200,1_97.wav,(184, 768)
now-403,all-240,2_17.wav,(118, 768)
now-403,all-280,2_57.wav,(175, 768)
now-403,all-320,3_11.wav,(172, 768)
now-403,all-360,3_52.wav,(104, 768)
now-403,all-400,3_96.wav,(81, 768)
all-feature-done
2023-11-12 17:33:13 | INFO | __main__ | ['infer/modules/train/extract/extract_f0_rmvpe.py', '2', '1', '0', '/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine', 'False']
todo-f0-201
f0ing,now-0,all-201,-/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine/1_16k_wavs/0_10.wav
['infer/modules/train/extract/extract_f0_rmvpe.py', '2', '0', '0', '/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine', 'False']
todo-f0-202
f0ing,now-0,all-202,-/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine/1_16k_wavs/0_1.wav
f0ing,now-40,all-202,-/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine/1_16k_wavs/0_70.wav
f0ing,now-40,all-201,-/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine/1_16k_wavs/0_71.wav
f0ing,now-80,all-202,-/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine/1_16k_wavs/1_53.wav
f0ing,now-80,all-201,-/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine/1_16k_wavs/1_55.wav
f0ing,now-120,all-201,-/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine/1_16k_wavs/2_18.wav
f0ing,now-120,all-202,-/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine/1_16k_wavs/2_17.wav
f0ing,now-160,all-201,-/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine/1_16k_wavs/3_12.wav
f0ing,now-160,all-202,-/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine/1_16k_wavs/3_11.wav
f0ing,now-200,all-201,-/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine/1_16k_wavs/3_97.wav
f0ing,now-200,all-202,-/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine/1_16k_wavs/3_96.wav
['infer/modules/train/extract_feature_print.py', 'cuda:0', '1', '0', '0', '/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine', 'v2']
/root/Retrieval-based-Voice-Conversion-WebUI/logs/director_one_combine
load model(s) from assets/hubert/hubert_base.pt
move model to cuda
all-feature-403
now-403,all-0,0_1.wav,(184, 768)
now-403,all-40,0_26.wav,(54, 768)
now-403,all-80,0_70.wav,(119, 768)
now-403,all-120,1_110.wav,(95, 768)
now-403,all-160,1_53.wav,(184, 768)
now-403,all-200,1_97.wav,(184, 768)
now-403,all-240,2_17.wav,(118, 768)
now-403,all-280,2_57.wav,(175, 768)
now-403,all-320,3_11.wav,(172, 768)
now-403,all-360,3_52.wav,(104, 768)
now-403,all-400,3_96.wav,(81, 768)
all-feature-done

2023-11-12 17:33:13 | INFO | httpx | HTTP Request: POST http://localhost:7865/api/predict "HTTP/1.1 200 OK"
2023-11-12 17:33:13 | INFO | __main__ | Use gpus: 0
2023-11-12 17:33:13 | INFO | __main__ | "python3" infer/modules/train/train.py -e "director_one_combine" -sr 40k -f0 1 -bs 12 -g 0 -te 200 -se 40 -pg assets/pretrained_v2/f0G40k.pth -pd assets/pretrained_v2/f0D40k.pth -l 0 -c 0 -sw 0 -v v2
INFO:director_one_combine:{'data': {'filter_length': 2048, 'hop_length': 400, 'max_wav_value': 32768.0, 'mel_fmax': None, 'mel_fmin': 0.0, 'n_mel_channels': 125, 'sampling_rate': 40000, 'win_length': 2048, 'training_files': './logs/director_one_combine/filelist.txt'}, 'model': {'filter_channels': 768, 'gin_channels': 256, 'hidden_channels': 192, 'inter_channels': 192, 'kernel_size': 3, 'n_heads': 2, 'n_layers': 6, 'p_dropout': 0, 'resblock': '1', 'resblock_dilation_sizes': [[1, 3, 5], [1, 3, 5], [1, 3, 5]], 'resblock_kernel_sizes': [3, 7, 11], 'spk_embed_dim': 109, 'upsample_initial_channel': 512, 'upsample_kernel_sizes': [16, 16, 4, 4], 'upsample_rates': [10, 10, 2, 2], 'use_spectral_norm': False}, 'train': {'batch_size': 12, 'betas': [0.8, 0.99], 'c_kl': 1.0, 'c_mel': 45, 'epochs': 20000, 'eps': 1e-09, 'fp16_run': False, 'init_lr_ratio': 1, 'learning_rate': 0.0001, 'log_interval': 200, 'lr_decay': 0.999875, 'seed': 1234, 'segment_size': 12800, 'warmup_epochs': 0}, 'model_dir': './logs/director_one_combine', 'experiment_dir': './logs/director_one_combine', 'save_every_epoch': 40, 'name': 'director_one_combine', 'total_epoch': 200, 'pretrainG': 'assets/pretrained_v2/f0G40k.pth', 'pretrainD': 'assets/pretrained_v2/f0D40k.pth', 'version': 'v2', 'gpus': '0', 'sample_rate': '40k', 'if_f0': 1, 'if_latest': 0, 'save_every_weights': '0', 'if_cache_data_in_gpu': 0}
/usr/local/lib/python3.10/dist-packages/torch/nn/utils/weight_norm.py:30: UserWarning: torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.
  warnings.warn("torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.")
DEBUG:infer.lib.infer_pack.models:gin_channels: 256, self.spk_embed_dim: 109
INFO:director_one_combine:loaded pretrained assets/pretrained_v2/f0G40k.pth
INFO:director_one_combine:<All keys matched successfully>
INFO:director_one_combine:loaded pretrained assets/pretrained_v2/f0D40k.pth
INFO:director_one_combine:<All keys matched successfully>
/usr/local/lib/python3.10/dist-packages/torch/functional.py:650: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error.
Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at ../aten/src/ATen/native/SpectralOps.cpp:863.)
  return _VF.stft(input, n_fft, hop_length, win_length, window,  # type: ignore[attr-defined]
/usr/local/lib/python3.10/dist-packages/torch/functional.py:650: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error.
Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at ../aten/src/ATen/native/SpectralOps.cpp:863.)
  return _VF.stft(input, n_fft, hop_length, win_length, window,  # type: ignore[attr-defined]
/usr/local/lib/python3.10/dist-packages/torch/functional.py:650: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error.
Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at ../aten/src/ATen/native/SpectralOps.cpp:863.)
  return _VF.stft(input, n_fft, hop_length, win_length, window,  # type: ignore[attr-defined]
/usr/local/lib/python3.10/dist-packages/torch/functional.py:650: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error.
Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at ../aten/src/ATen/native/SpectralOps.cpp:863.)
  return _VF.stft(input, n_fft, hop_length, win_length, window,  # type: ignore[attr-defined]
/usr/local/lib/python3.10/dist-packages/torch/functional.py:650: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error.
Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at ../aten/src/ATen/native/SpectralOps.cpp:863.)
  return _VF.stft(input, n_fft, hop_length, win_length, window,  # type: ignore[attr-defined]
/usr/local/lib/python3.10/dist-packages/torch/autograd/__init__.py:251: UserWarning: Grad strides do not match bucket view strides. This may indicate grad was not created according to the gradient layout contract, or that the param's strides changed since DDP was constructed.  This is not an error, but may impair performance.
grad.sizes() = [64, 1, 4], strides() = [4, 1, 1]
bucket_view.sizes() = [64, 1, 4], strides() = [4, 4, 1] (Triggered internally at ../torch/csrc/distributed/c10d/reducer.cpp:320.)
  Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
INFO:director_one_combine:Train Epoch: 1 [0%]
INFO:director_one_combine:[0, 0.0001]
INFO:director_one_combine:loss_disc=4.112, loss_gen=4.064, loss_fm=9.922,loss_mel=24.526, loss_kl=9.000
DEBUG:matplotlib:matplotlib data path: /usr/local/lib/python3.10/dist-packages/matplotlib/mpl-data
DEBUG:matplotlib:CONFIGDIR=/root/.config/matplotlib
DEBUG:matplotlib:interactive is False
DEBUG:matplotlib:platform is linux
INFO:director_one_combine:====> Epoch: 1 [2023-11-12 17:34:13] | (0:00:46.664000)
INFO:director_one_combine:====> Epoch: 2 [2023-11-12 17:35:03] | (0:00:49.659887)

==========
== CUDA ==
==========

CUDA Version 11.6.2

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

2023-11-12 11:05:57 | INFO | configs.config | Found GPU NVIDIA A40-16Q, force to fp32
2023-11-12 11:05:59 | INFO | __main__ | Use Language: en_US
2023-11-12 11:07:32 | INFO | httpx | HTTP Request: POST http://localhost:7777/api/predict "HTTP/1.1 200 OK"
2023-11-12 11:07:32 | INFO | __main__ | "/usr/bin/python3" infer/modules/train/preprocess.py "/app/data/director_one_combine" 40000 4 "/app/logs/director_one_combine" False 3.7
['infer/modules/train/preprocess.py', '/app/data/director_one_combine', '40000', '4', '/app/logs/director_one_combine', 'False', '3.7']
start preprocess
['infer/modules/train/preprocess.py', '/app/data/director_one_combine', '40000', '4', '/app/logs/director_one_combine', 'False', '3.7']
/app/data/director_one_combine/4.wav->Suc.
/app/data/director_one_combine/2.wav->Suc.
/app/data/director_one_combine/1.wav->Suc.
/app/data/director_one_combine/3.wav->Suc.
end preprocess
2023-11-12 11:07:40 | INFO | __main__ | start preprocess
['infer/modules/train/preprocess.py', '/app/data/director_one_combine', '40000', '4', '/app/logs/director_one_combine', 'False', '3.7']
/app/data/director_one_combine/4.wav->Suc.
/app/data/director_one_combine/2.wav->Suc.
/app/data/director_one_combine/1.wav->Suc.
/app/data/director_one_combine/3.wav->Suc.
end preprocess

2023-11-12 11:07:40 | INFO | httpx | HTTP Request: POST http://localhost:7777/api/predict "HTTP/1.1 200 OK"
2023-11-12 11:07:40 | INFO | __main__ | "/usr/bin/python3" infer/modules/train/extract/extract_f0_rmvpe.py 2 0 0 "/app/logs/director_one_combine" False 
2023-11-12 11:07:40 | INFO | __main__ | "/usr/bin/python3" infer/modules/train/extract/extract_f0_rmvpe.py 2 1 0 "/app/logs/director_one_combine" False 
['infer/modules/train/extract/extract_f0_rmvpe.py', '2', '0', '0', '/app/logs/director_one_combine', 'False']
todo-f0-202
f0ing,now-0,all-202,-/app/logs/director_one_combine/1_16k_wavs/0_1.wav
f0ing,now-40,all-202,-/app/logs/director_one_combine/1_16k_wavs/0_70.wav
f0ing,now-80,all-202,-/app/logs/director_one_combine/1_16k_wavs/1_53.wav
f0ing,now-120,all-202,-/app/logs/director_one_combine/1_16k_wavs/2_17.wav
f0ing,now-160,all-202,-/app/logs/director_one_combine/1_16k_wavs/3_11.wav
f0ing,now-200,all-202,-/app/logs/director_one_combine/1_16k_wavs/3_96.wav
['infer/modules/train/extract/extract_f0_rmvpe.py', '2', '1', '0', '/app/logs/director_one_combine', 'False']
todo-f0-201
f0ing,now-0,all-201,-/app/logs/director_one_combine/1_16k_wavs/0_10.wav
f0ing,now-40,all-201,-/app/logs/director_one_combine/1_16k_wavs/0_71.wav
f0ing,now-80,all-201,-/app/logs/director_one_combine/1_16k_wavs/1_55.wav
f0ing,now-120,all-201,-/app/logs/director_one_combine/1_16k_wavs/2_18.wav
f0ing,now-160,all-201,-/app/logs/director_one_combine/1_16k_wavs/3_12.wav
f0ing,now-200,all-201,-/app/logs/director_one_combine/1_16k_wavs/3_97.wav
2023-11-12 11:07:42 | INFO | __main__ | ['infer/modules/train/extract/extract_f0_rmvpe.py', '2', '0', '0', '/app/logs/director_one_combine', 'False']
todo-f0-202
f0ing,now-0,all-202,-/app/logs/director_one_combine/1_16k_wavs/0_1.wav
f0ing,now-40,all-202,-/app/logs/director_one_combine/1_16k_wavs/0_70.wav
f0ing,now-80,all-202,-/app/logs/director_one_combine/1_16k_wavs/1_53.wav
f0ing,now-120,all-202,-/app/logs/director_one_combine/1_16k_wavs/2_17.wav
f0ing,now-160,all-202,-/app/logs/director_one_combine/1_16k_wavs/3_11.wav
f0ing,now-200,all-202,-/app/logs/director_one_combine/1_16k_wavs/3_96.wav
['infer/modules/train/extract/extract_f0_rmvpe.py', '2', '1', '0', '/app/logs/director_one_combine', 'False']
todo-f0-201
f0ing,now-0,all-201,-/app/logs/director_one_combine/1_16k_wavs/0_10.wav
f0ing,now-40,all-201,-/app/logs/director_one_combine/1_16k_wavs/0_71.wav
f0ing,now-80,all-201,-/app/logs/director_one_combine/1_16k_wavs/1_55.wav
f0ing,now-120,all-201,-/app/logs/director_one_combine/1_16k_wavs/2_18.wav
f0ing,now-160,all-201,-/app/logs/director_one_combine/1_16k_wavs/3_12.wav
f0ing,now-200,all-201,-/app/logs/director_one_combine/1_16k_wavs/3_97.wav

2023-11-12 11:07:42 | INFO | __main__ | "/usr/bin/python3" infer/modules/train/extract_feature_print.py cuda:0 1 0 0 "/app/logs/director_one_combine" v2
2023-11-12 11:07:45 | INFO | fairseq.tasks.hubert_pretraining | current directory is /app
2023-11-12 11:07:45 | INFO | fairseq.tasks.hubert_pretraining | HubertPretrainingTask Config {'_name': 'hubert_pretraining', 'data': 'metadata', 'fine_tuning': False, 'labels': ['km'], 'label_dir': 'label', 'label_rate': 50.0, 'sample_rate': 16000, 'normalize': False, 'enable_padding': False, 'max_keep_size': None, 'max_sample_size': 250000, 'min_sample_size': 32000, 'single_target': False, 'random_crop': True, 'pad_audio': False}
2023-11-12 11:07:45 | INFO | fairseq.models.hubert.hubert | HubertModel Config: {'_name': 'hubert', 'label_rate': 50.0, 'extractor_mode': default, 'encoder_layers': 12, 'encoder_embed_dim': 768, 'encoder_ffn_embed_dim': 3072, 'encoder_attention_heads': 12, 'activation_fn': gelu, 'layer_type': transformer, 'dropout': 0.1, 'attention_dropout': 0.1, 'activation_dropout': 0.0, 'encoder_layerdrop': 0.05, 'dropout_input': 0.1, 'dropout_features': 0.1, 'final_dim': 256, 'untie_final_proj': True, 'layer_norm_first': False, 'conv_feature_layers': '[(512,10,5)] + [(512,3,2)] * 4 + [(512,2,2)] * 2', 'conv_bias': False, 'logit_temp': 0.1, 'target_glu': False, 'feature_grad_mult': 0.1, 'mask_length': 10, 'mask_prob': 0.8, 'mask_selection': static, 'mask_other': 0.0, 'no_mask_overlap': False, 'mask_min_space': 1, 'mask_channel_length': 10, 'mask_channel_prob': 0.0, 'mask_channel_selection': static, 'mask_channel_other': 0.0, 'no_mask_channel_overlap': False, 'mask_channel_min_space': 1, 'conv_pos': 128, 'conv_pos_groups': 16, 'latent_temp': [2.0, 0.5, 0.999995], 'skip_masked': False, 'skip_nomask': False, 'checkpoint_activations': False, 'required_seq_len_multiple': 2, 'depthwise_conv_kernel_size': 31, 'attn_type': '', 'pos_enc_type': 'abs', 'fp16': False}
/usr/local/lib/python3.9/dist-packages/torch/nn/utils/weight_norm.py:30: UserWarning: torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.
  warnings.warn("torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.")
['infer/modules/train/extract_feature_print.py', 'cuda:0', '1', '0', '0', '/app/logs/director_one_combine', 'v2']
/app/logs/director_one_combine
load model(s) from assets/hubert/hubert_base.pt
move model to cuda
all-feature-403
all-feature-done
2023-11-12 11:07:49 | INFO | __main__ | ['infer/modules/train/extract/extract_f0_rmvpe.py', '2', '0', '0', '/app/logs/director_one_combine', 'False']
todo-f0-202
f0ing,now-0,all-202,-/app/logs/director_one_combine/1_16k_wavs/0_1.wav
f0ing,now-40,all-202,-/app/logs/director_one_combine/1_16k_wavs/0_70.wav
f0ing,now-80,all-202,-/app/logs/director_one_combine/1_16k_wavs/1_53.wav
f0ing,now-120,all-202,-/app/logs/director_one_combine/1_16k_wavs/2_17.wav
f0ing,now-160,all-202,-/app/logs/director_one_combine/1_16k_wavs/3_11.wav
f0ing,now-200,all-202,-/app/logs/director_one_combine/1_16k_wavs/3_96.wav
['infer/modules/train/extract/extract_f0_rmvpe.py', '2', '1', '0', '/app/logs/director_one_combine', 'False']
todo-f0-201
f0ing,now-0,all-201,-/app/logs/director_one_combine/1_16k_wavs/0_10.wav
f0ing,now-40,all-201,-/app/logs/director_one_combine/1_16k_wavs/0_71.wav
f0ing,now-80,all-201,-/app/logs/director_one_combine/1_16k_wavs/1_55.wav
f0ing,now-120,all-201,-/app/logs/director_one_combine/1_16k_wavs/2_18.wav
f0ing,now-160,all-201,-/app/logs/director_one_combine/1_16k_wavs/3_12.wav
f0ing,now-200,all-201,-/app/logs/director_one_combine/1_16k_wavs/3_97.wav
['infer/modules/train/extract_feature_print.py', 'cuda:0', '1', '0', '0', '/app/logs/director_one_combine', 'v2']
/app/logs/director_one_combine
load model(s) from assets/hubert/hubert_base.pt
move model to cuda
all-feature-403
all-feature-done

2023-11-12 11:07:49 | INFO | httpx | HTTP Request: POST http://localhost:7777/api/predict "HTTP/1.1 200 OK"
2023-11-12 11:07:49 | INFO | __main__ | Use gpus: 0
2023-11-12 11:07:49 | INFO | __main__ | "/usr/bin/python3" infer/modules/train/train.py -e "director_one_combine" -sr 40k -f0 1 -bs 12 -g 0 -te 200 -se 40 -pg assets/pretrained_v2/f0G40k.pth -pd assets/pretrained_v2/f0D40k.pth -l 0 -c 0 -sw 0 -v v2
INFO:director_one_combine:{'data': {'filter_length': 2048, 'hop_length': 400, 'max_wav_value': 32768.0, 'mel_fmax': None, 'mel_fmin': 0.0, 'n_mel_channels': 125, 'sampling_rate': 40000, 'win_length': 2048, 'training_files': './logs/director_one_combine/filelist.txt'}, 'model': {'filter_channels': 768, 'gin_channels': 256, 'hidden_channels': 192, 'inter_channels': 192, 'kernel_size': 3, 'n_heads': 2, 'n_layers': 6, 'p_dropout': 0, 'resblock': '1', 'resblock_dilation_sizes': [[1, 3, 5], [1, 3, 5], [1, 3, 5]], 'resblock_kernel_sizes': [3, 7, 11], 'spk_embed_dim': 109, 'upsample_initial_channel': 512, 'upsample_kernel_sizes': [16, 16, 4, 4], 'upsample_rates': [10, 10, 2, 2], 'use_spectral_norm': False}, 'train': {'batch_size': 12, 'betas': [0.8, 0.99], 'c_kl': 1.0, 'c_mel': 45, 'epochs': 20000, 'eps': 1e-09, 'fp16_run': False, 'init_lr_ratio': 1, 'learning_rate': 0.0001, 'log_interval': 200, 'lr_decay': 0.999875, 'seed': 1234, 'segment_size': 12800, 'warmup_epochs': 0}, 'model_dir': './logs/director_one_combine', 'experiment_dir': './logs/director_one_combine', 'save_every_epoch': 40, 'name': 'director_one_combine', 'total_epoch': 200, 'pretrainG': 'assets/pretrained_v2/f0G40k.pth', 'pretrainD': 'assets/pretrained_v2/f0D40k.pth', 'version': 'v2', 'gpus': '0', 'sample_rate': '40k', 'if_f0': 1, 'if_latest': 0, 'save_every_weights': '0', 'if_cache_data_in_gpu': 0}
/usr/local/lib/python3.9/dist-packages/torch/nn/utils/weight_norm.py:30: UserWarning: torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.
  warnings.warn("torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.")
DEBUG:infer.lib.infer_pack.models:gin_channels: 256, self.spk_embed_dim: 109
INFO:director_one_combine:loaded pretrained assets/pretrained_v2/f0G40k.pth
INFO:director_one_combine:<All keys matched successfully>
INFO:director_one_combine:loaded pretrained assets/pretrained_v2/f0D40k.pth
INFO:director_one_combine:<All keys matched successfully>
/usr/local/lib/python3.9/dist-packages/torch/functional.py:650: UserWarning: stft with return_complex=False is deprecated. In a future pytorch release, stft will return complex tensors for all inputs, and return_complex=False will raise an error.
Note: you can still call torch.view_as_real on the complex output to recover the old return format. (Triggered internally at ../aten/src/ATen/native/SpectralOps.cpp:863.)
  return _VF.stft(input, n_fft, hop_length, win_length, window,  # type: ignore[attr-defined]
/usr/local/lib/python3.9/dist-packages/torch/autograd/__init__.py:251: UserWarning: Grad strides do not match bucket view strides. This may indicate grad was not created according to the gradient layout contract, or that the param's strides changed since DDP was constructed.  This is not an error, but may impair performance.
grad.sizes() = [64, 1, 4], strides() = [4, 1, 1]
bucket_view.sizes() = [64, 1, 4], strides() = [4, 4, 1] (Triggered internally at ../torch/csrc/distributed/c10d/reducer.cpp:320.)
  Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
INFO:director_one_combine:Train Epoch: 1 [0%]
INFO:director_one_combine:[0, 0.0001]
INFO:director_one_combine:loss_disc=4.049, loss_gen=3.960, loss_fm=10.065,loss_mel=25.043, loss_kl=8.011
DEBUG:matplotlib:matplotlib data path: /usr/local/lib/python3.9/dist-packages/matplotlib/mpl-data
DEBUG:matplotlib:CONFIGDIR=/root/.config/matplotlib
DEBUG:matplotlib:interactive is False
DEBUG:matplotlib:platform is linux
INFO:director_one_combine:====> Epoch: 1 [2023-11-12 11:08:51] | (0:00:46.691563)
INFO:director_one_combine:====> Epoch: 2 [2023-11-12 11:09:39] | (0:00:48.729325)
INFO:director_one_combine:====> Epoch: 3 [2023-11-12 11:10:28] | (0:00:48.755667)
INFO:director_one_combine:====> Epoch: 4 [2023-11-12 11:11:13] | (0:00:44.874771)
INFO:director_one_combine:====> Epoch: 5 [2023-11-12 11:11:55] | (0:00:42.025411)
INFO:director_one_combine:Train Epoch: 6 [41%]
INFO:director_one_combine:[200, 9.993751562304699e-05]
INFO:director_one_combine:loss_disc=3.749, loss_gen=3.021, loss_fm=10.258,loss_mel=19.672, loss_kl=1.380
INFO:director_one_combine:====> Epoch: 6 [2023-11-12 11:12:44] | (0:00:49.079633)
INFO:director_one_combine:====> Epoch: 7 [2023-11-12 11:13:33] | (0:00:48.451769)
INFO:director_one_combine:====> Epoch: 8 [2023-11-12 11:14:17] | (0:00:44.749183)
INFO:director_one_combine:====> Epoch: 9 [2023-11-12 11:15:03] | (0:00:45.586853)
INFO:director_one_combine:====> Epoch: 10 [2023-11-12 11:15:52] | (0:00:49.202066)
INFO:director_one_combine:Train Epoch: 11 [81%]
INFO:director_one_combine:[400, 9.987507028906759e-05]
INFO:director_one_combine:loss_disc=4.030, loss_gen=3.056, loss_fm=8.791,loss_mel=19.757, loss_kl=1.590
INFO:director_one_combine:====> Epoch: 11 [2023-11-12 11:16:42] | (0:00:49.293435)
INFO:director_one_combine:====> Epoch: 12 [2023-11-12 11:17:26] | (0:00:44.336452)
INFO:director_one_combine:====> Epoch: 13 [2023-11-12 11:18:13] | (0:00:46.808254)
INFO:director_one_combine:====> Epoch: 14 [2023-11-12 11:19:02] | (0:00:49.016010)
INFO:director_one_combine:====> Epoch: 15 [2023-11-12 11:19:51] | (0:00:49.334506)
INFO:director_one_combine:====> Epoch: 16 [2023-11-12 11:20:30] | (0:00:38.843907)
INFO:director_one_combine:Train Epoch: 17 [22%]
INFO:director_one_combine:[600, 9.980018739066937e-05]
INFO:director_one_combine:loss_disc=3.766, loss_gen=3.272, loss_fm=8.257,loss_mel=19.056, loss_kl=1.469
INFO:director_one_combine:====> Epoch: 17 [2023-11-12 11:21:20] | (0:00:49.571799)
INFO:director_one_combine:====> Epoch: 18 [2023-11-12 11:22:08] | (0:00:48.871248)
INFO:director_one_combine:====> Epoch: 19 [2023-11-12 11:22:57] | (0:00:48.985603)
INFO:director_one_combine:====> Epoch: 20 [2023-11-12 11:23:35] | (0:00:37.708594)
INFO:director_one_combine:====> Epoch: 21 [2023-11-12 11:24:23] | (0:00:48.288564)
INFO:director_one_combine:Train Epoch: 22 [62%]
INFO:director_one_combine:[800, 9.973782786538036e-05]
INFO:director_one_combine:loss_disc=4.040, loss_gen=3.273, loss_fm=7.876,loss_mel=18.639, loss_kl=1.419
INFO:director_one_combine:====> Epoch: 22 [2023-11-12 11:25:13] | (0:00:49.595784)
INFO:director_one_combine:====> Epoch: 23 [2023-11-12 11:26:04] | (0:00:51.368949)
INFO:director_one_combine:====> Epoch: 24 [2023-11-12 11:26:43] | (0:00:38.341725)
INFO:director_one_combine:====> Epoch: 25 [2023-11-12 11:27:31] | (0:00:48.313097)
INFO:director_one_combine:====> Epoch: 26 [2023-11-12 11:28:20] | (0:00:49.072454)
INFO:director_one_combine:====> Epoch: 27 [2023-11-12 11:29:09] | (0:00:48.578711)
INFO:director_one_combine:Train Epoch: 28 [3%]
INFO:director_one_combine:[1000, 9.966304786663908e-05]
INFO:director_one_combine:loss_disc=3.958, loss_gen=3.279, loss_fm=8.522,loss_mel=18.560, loss_kl=1.388
INFO:director_one_combine:====> Epoch: 28 [2023-11-12 11:29:49] | (0:00:39.876793)
INFO:director_one_combine:====> Epoch: 29 [2023-11-12 11:30:37] | (0:00:48.746597)
INFO:director_one_combine:====> Epoch: 30 [2023-11-12 11:31:26] | (0:00:49.099149)
INFO:director_one_combine:====> Epoch: 31 [2023-11-12 11:32:13] | (0:00:46.602778)
INFO:director_one_combine:====> Epoch: 32 [2023-11-12 11:32:55] | (0:00:41.405389)
INFO:director_one_combine:Train Epoch: 33 [43%]
INFO:director_one_combine:[1200, 9.960077403212722e-05]
INFO:director_one_combine:loss_disc=3.987, loss_gen=3.024, loss_fm=7.972,loss_mel=18.432, loss_kl=1.194
INFO:director_one_combine:====> Epoch: 33 [2023-11-12 11:33:43] | (0:00:48.689449)
INFO:director_one_combine:====> Epoch: 34 [2023-11-12 11:34:32] | (0:00:48.623982)
INFO:director_one_combine:====> Epoch: 35 [2023-11-12 11:35:16] | (0:00:44.161334)
INFO:director_one_combine:====> Epoch: 36 [2023-11-12 11:36:03] | (0:00:46.531684)
INFO:director_one_combine:====> Epoch: 37 [2023-11-12 11:36:51] | (0:00:48.573319)
INFO:director_one_combine:Train Epoch: 38 [84%]
INFO:director_one_combine:[1400, 9.953853910903285e-05]
INFO:director_one_combine:loss_disc=3.933, loss_gen=3.374, loss_fm=8.365,loss_mel=18.641, loss_kl=1.541
INFO:director_one_combine:====> Epoch: 38 [2023-11-12 11:37:40] | (0:00:49.130859)
INFO:director_one_combine:====> Epoch: 39 [2023-11-12 11:38:22] | (0:00:41.675466)
INFO:director_one_combine:Saving model and optimizer state at epoch 40 to ./logs/director_one_combine/G_1480.pth
INFO:director_one_combine:Saving model and optimizer state at epoch 40 to ./logs/director_one_combine/D_1480.pth
INFO:director_one_combine:====> Epoch: 40 [2023-11-12 11:39:09] | (0:00:47.230932)
INFO:director_one_combine:====> Epoch: 41 [2023-11-12 11:39:58] | (0:00:49.241099)
INFO:director_one_combine:====> Epoch: 42 [2023-11-12 11:40:47] | (0:00:48.920513)
INFO:director_one_combine:====> Epoch: 43 [2023-11-12 11:41:27] | (0:00:39.809559)
INFO:director_one_combine:Train Epoch: 44 [24%]
INFO:director_one_combine:[1600, 9.94639085301583e-05]
INFO:director_one_combine:loss_disc=4.042, loss_gen=3.150, loss_fm=6.539,loss_mel=18.395, loss_kl=1.509
INFO:director_one_combine:====> Epoch: 44 [2023-11-12 11:42:19] | (0:00:51.642106)
INFO:director_one_combine:====> Epoch: 45 [2023-11-12 11:43:07] | (0:00:48.622880)
INFO:director_one_combine:====> Epoch: 46 [2023-11-12 11:43:57] | (0:00:49.206673)
INFO:director_one_combine:====> Epoch: 47 [2023-11-12 11:44:38] | (0:00:41.480662)
INFO:director_one_combine:====> Epoch: 48 [2023-11-12 11:45:29] | (0:00:50.847920)
INFO:director_one_combine:Train Epoch: 49 [65%]
INFO:director_one_combine:[1800, 9.940175912662009e-05]
INFO:director_one_combine:loss_disc=3.991, loss_gen=3.144, loss_fm=8.778,loss_mel=17.254, loss_kl=0.753
INFO:director_one_combine:====> Epoch: 49 [2023-11-12 11:46:18] | (0:00:49.260369)
INFO:director_one_combine:====> Epoch: 50 [2023-11-12 11:47:07] | (0:00:49.121125)
INFO:director_one_combine:====> Epoch: 51 [2023-11-12 11:47:45] | (0:00:37.956463)
INFO:director_one_combine:====> Epoch: 52 [2023-11-12 11:48:34] | (0:00:49.070305)
INFO:director_one_combine:====> Epoch: 53 [2023-11-12 11:49:23] | (0:00:48.254441)
INFO:director_one_combine:====> Epoch: 54 [2023-11-12 11:50:11] | (0:00:48.282535)
INFO:director_one_combine:Train Epoch: 55 [5%]
INFO:director_one_combine:[2000, 9.932723110067987e-05]
INFO:director_one_combine:loss_disc=3.752, loss_gen=3.394, loss_fm=7.776,loss_mel=16.148, loss_kl=0.334
INFO:director_one_combine:====> Epoch: 55 [2023-11-12 11:50:53] | (0:00:41.719210)
INFO:director_one_combine:====> Epoch: 56 [2023-11-12 11:51:41] | (0:00:48.337323)
INFO:director_one_combine:====> Epoch: 57 [2023-11-12 11:52:30] | (0:00:49.145922)
INFO:director_one_combine:====> Epoch: 58 [2023-11-12 11:53:16] | (0:00:45.283326)
INFO:director_one_combine:====> Epoch: 59 [2023-11-12 11:53:56] | (0:00:40.408597)
INFO:director_one_combine:Train Epoch: 60 [46%]
INFO:director_one_combine:[2200, 9.926516709918191e-05]
INFO:director_one_combine:loss_disc=3.789, loss_gen=3.291, loss_fm=10.144,loss_mel=18.426, loss_kl=1.059
INFO:director_one_combine:====> Epoch: 60 [2023-11-12 11:54:45] | (0:00:48.723280)
INFO:director_one_combine:====> Epoch: 61 [2023-11-12 11:55:34] | (0:00:49.453303)
INFO:director_one_combine:====> Epoch: 62 [2023-11-12 11:56:19] | (0:00:44.683538)
INFO:director_one_combine:====> Epoch: 63 [2023-11-12 11:57:03] | (0:00:44.283557)
INFO:director_one_combine:====> Epoch: 64 [2023-11-12 11:57:52] | (0:00:48.918615)
INFO:director_one_combine:Train Epoch: 65 [86%]
INFO:director_one_combine:[2400, 9.92031418779886e-05]
INFO:director_one_combine:loss_disc=3.840, loss_gen=3.333, loss_fm=8.488,loss_mel=17.789, loss_kl=1.243
INFO:director_one_combine:====> Epoch: 65 [2023-11-12 11:58:42] | (0:00:49.651550)
INFO:director_one_combine:====> Epoch: 66 [2023-11-12 11:59:24] | (0:00:42.465295)
INFO:director_one_combine:====> Epoch: 67 [2023-11-12 12:00:10] | (0:00:45.781783)
INFO:director_one_combine:====> Epoch: 68 [2023-11-12 12:00:59] | (0:00:49.158898)
INFO:director_one_combine:====> Epoch: 69 [2023-11-12 12:01:48] | (0:00:48.908754)
INFO:director_one_combine:====> Epoch: 70 [2023-11-12 12:02:30] | (0:00:41.460348)
INFO:director_one_combine:Train Epoch: 71 [27%]
INFO:director_one_combine:[2600, 9.912876276844171e-05]
INFO:director_one_combine:loss_disc=3.742, loss_gen=3.401, loss_fm=10.642,loss_mel=17.769, loss_kl=0.930
INFO:director_one_combine:====> Epoch: 71 [2023-11-12 12:03:19] | (0:00:49.155388)
INFO:director_one_combine:====> Epoch: 72 [2023-11-12 12:04:08] | (0:00:49.087226)
INFO:director_one_combine:====> Epoch: 73 [2023-11-12 12:04:56] | (0:00:48.592569)
INFO:director_one_combine:====> Epoch: 74 [2023-11-12 12:05:36] | (0:00:39.789935)
INFO:director_one_combine:====> Epoch: 75 [2023-11-12 12:06:25] | (0:00:48.359721)
INFO:director_one_combine:Train Epoch: 76 [68%]
INFO:director_one_combine:[2800, 9.906682277864462e-05]
INFO:director_one_combine:loss_disc=3.891, loss_gen=3.166, loss_fm=8.107,loss_mel=17.214, loss_kl=1.153
INFO:director_one_combine:====> Epoch: 76 [2023-11-12 12:07:14] | (0:00:49.317990)
INFO:director_one_combine:====> Epoch: 77 [2023-11-12 12:08:03] | (0:00:49.025194)
INFO:director_one_combine:====> Epoch: 78 [2023-11-12 12:08:41] | (0:00:37.889287)
INFO:director_one_combine:====> Epoch: 79 [2023-11-12 12:09:30] | (0:00:49.297595)
INFO:director_one_combine:Saving model and optimizer state at epoch 80 to ./logs/director_one_combine/G_2960.pth
INFO:director_one_combine:Saving model and optimizer state at epoch 80 to ./logs/director_one_combine/D_2960.pth
INFO:director_one_combine:====> Epoch: 80 [2023-11-12 12:10:21] | (0:00:50.908803)
INFO:director_one_combine:====> Epoch: 81 [2023-11-12 12:11:08] | (0:00:46.851562)
INFO:director_one_combine:Train Epoch: 82 [8%]
INFO:director_one_combine:[3000, 9.899254587647776e-05]
INFO:director_one_combine:loss_disc=3.700, loss_gen=3.110, loss_fm=8.202,loss_mel=17.743, loss_kl=0.758
INFO:director_one_combine:====> Epoch: 82 [2023-11-12 12:11:49] | (0:00:40.812412)
INFO:director_one_combine:====> Epoch: 83 [2023-11-12 12:12:37] | (0:00:48.619773)
INFO:director_one_combine:====> Epoch: 84 [2023-11-12 12:13:27] | (0:00:49.188888)
INFO:director_one_combine:====> Epoch: 85 [2023-11-12 12:14:12] | (0:00:45.309377)
INFO:director_one_combine:====> Epoch: 86 [2023-11-12 12:14:56] | (0:00:43.722239)
INFO:director_one_combine:Train Epoch: 87 [49%]
INFO:director_one_combine:[3200, 9.89306910009569e-05]
INFO:director_one_combine:loss_disc=3.820, loss_gen=3.250, loss_fm=8.871,loss_mel=17.565, loss_kl=1.337
INFO:director_one_combine:====> Epoch: 87 [2023-11-12 12:15:44] | (0:00:48.686700)
INFO:director_one_combine:====> Epoch: 88 [2023-11-12 12:16:34] | (0:00:49.622062)
INFO:director_one_combine:====> Epoch: 89 [2023-11-12 12:17:16] | (0:00:42.121644)
INFO:director_one_combine:====> Epoch: 90 [2023-11-12 12:18:01] | (0:00:44.914107)
INFO:director_one_combine:====> Epoch: 91 [2023-11-12 12:18:50] | (0:00:49.453617)
INFO:director_one_combine:Train Epoch: 92 [89%]
INFO:director_one_combine:[3400, 9.886887477506964e-05]
INFO:director_one_combine:loss_disc=3.650, loss_gen=3.726, loss_fm=8.154,loss_mel=17.088, loss_kl=1.299
INFO:director_one_combine:====> Epoch: 92 [2023-11-12 12:19:43] | (0:00:52.306432)
INFO:director_one_combine:====> Epoch: 93 [2023-11-12 12:20:23] | (0:00:40.728761)
INFO:director_one_combine:====> Epoch: 94 [2023-11-12 12:21:11] | (0:00:47.407470)
INFO:director_one_combine:====> Epoch: 95 [2023-11-12 12:22:00] | (0:00:49.428012)
INFO:director_one_combine:====> Epoch: 96 [2023-11-12 12:22:49] | (0:00:48.983076)
INFO:director_one_combine:====> Epoch: 97 [2023-11-12 12:23:26] | (0:00:36.831314)
INFO:director_one_combine:Train Epoch: 98 [30%]
INFO:director_one_combine:[3600, 9.879474628751914e-05]
INFO:director_one_combine:loss_disc=3.672, loss_gen=3.233, loss_fm=11.028,loss_mel=17.371, loss_kl=0.930
INFO:director_one_combine:====> Epoch: 98 [2023-11-12 12:24:15] | (0:00:48.930913)
INFO:director_one_combine:====> Epoch: 99 [2023-11-12 12:25:04] | (0:00:48.431485)
INFO:director_one_combine:====> Epoch: 100 [2023-11-12 12:25:53] | (0:00:49.008281)
INFO:director_one_combine:====> Epoch: 101 [2023-11-12 12:26:28] | (0:00:35.174897)
INFO:director_one_combine:====> Epoch: 102 [2023-11-12 12:27:17] | (0:00:49.155870)
INFO:director_one_combine:Train Epoch: 103 [70%]
INFO:director_one_combine:[3800, 9.873301500583906e-05]
INFO:director_one_combine:loss_disc=3.830, loss_gen=3.396, loss_fm=7.755,loss_mel=16.824, loss_kl=1.134
INFO:director_one_combine:====> Epoch: 103 [2023-11-12 12:28:07] | (0:00:50.008070)
INFO:director_one_combine:====> Epoch: 104 [2023-11-12 12:28:56] | (0:00:48.938598)
INFO:director_one_combine:====> Epoch: 105 [2023-11-12 12:29:35] | (0:00:39.425409)
INFO:director_one_combine:====> Epoch: 106 [2023-11-12 12:30:25] | (0:00:49.299826)
INFO:director_one_combine:====> Epoch: 107 [2023-11-12 12:31:14] | (0:00:49.090795)
INFO:director_one_combine:====> Epoch: 108 [2023-11-12 12:32:02] | (0:00:48.305967)
INFO:director_one_combine:Train Epoch: 109 [11%]
INFO:director_one_combine:[4000, 9.865898838127865e-05]
INFO:director_one_combine:loss_disc=3.631, loss_gen=3.585, loss_fm=9.338,loss_mel=17.125, loss_kl=0.912
INFO:director_one_combine:====> Epoch: 109 [2023-11-12 12:32:40] | (0:00:37.904033)
INFO:director_one_combine:====> Epoch: 110 [2023-11-12 12:33:28] | (0:00:47.938523)
INFO:director_one_combine:====> Epoch: 111 [2023-11-12 12:34:17] | (0:00:48.972532)
INFO:director_one_combine:====> Epoch: 112 [2023-11-12 12:35:04] | (0:00:47.547333)
INFO:director_one_combine:====> Epoch: 113 [2023-11-12 12:35:42] | (0:00:37.224651)
INFO:director_one_combine:Train Epoch: 114 [51%]
INFO:director_one_combine:[4200, 9.859734192708044e-05]
INFO:director_one_combine:loss_disc=3.489, loss_gen=3.591, loss_fm=9.089,loss_mel=17.208, loss_kl=0.680
INFO:director_one_combine:====> Epoch: 114 [2023-11-12 12:36:30] | (0:00:48.834213)
INFO:director_one_combine:====> Epoch: 115 [2023-11-12 12:37:20] | (0:00:49.315120)
INFO:director_one_combine:====> Epoch: 116 [2023-11-12 12:38:07] | (0:00:47.280854)
INFO:director_one_combine:====> Epoch: 117 [2023-11-12 12:38:47] | (0:00:39.751210)
INFO:director_one_combine:====> Epoch: 118 [2023-11-12 12:39:36] | (0:00:49.324568)
INFO:director_one_combine:Train Epoch: 119 [92%]
INFO:director_one_combine:[4400, 9.853573399228505e-05]
INFO:director_one_combine:loss_disc=3.693, loss_gen=3.482, loss_fm=9.219,loss_mel=17.532, loss_kl=1.114
INFO:director_one_combine:====> Epoch: 119 [2023-11-12 12:40:25] | (0:00:48.676945)
INFO:director_one_combine:Saving model and optimizer state at epoch 120 to ./logs/director_one_combine/G_4440.pth
INFO:director_one_combine:Saving model and optimizer state at epoch 120 to ./logs/director_one_combine/D_4440.pth
INFO:director_one_combine:====> Epoch: 120 [2023-11-12 12:41:12] | (0:00:47.591404)
INFO:director_one_combine:====> Epoch: 121 [2023-11-12 12:41:55] | (0:00:42.299949)
INFO:director_one_combine:====> Epoch: 122 [2023-11-12 12:42:43] | (0:00:48.436531)
INFO:director_one_combine:====> Epoch: 123 [2023-11-12 12:43:32] | (0:00:49.102945)
INFO:director_one_combine:====> Epoch: 124 [2023-11-12 12:44:15] | (0:00:42.429541)
INFO:director_one_combine:Train Epoch: 125 [32%]
INFO:director_one_combine:[4600, 9.846185528225477e-05]
INFO:director_one_combine:loss_disc=3.656, loss_gen=3.720, loss_fm=9.330,loss_mel=17.234, loss_kl=1.025
INFO:director_one_combine:====> Epoch: 125 [2023-11-12 12:45:00] | (0:00:44.994524)
INFO:director_one_combine:====> Epoch: 126 [2023-11-12 12:45:49] | (0:00:49.064541)
INFO:director_one_combine:====> Epoch: 127 [2023-11-12 12:46:37] | (0:00:48.258116)
INFO:director_one_combine:====> Epoch: 128 [2023-11-12 12:47:19] | (0:00:42.133087)
INFO:director_one_combine:====> Epoch: 129 [2023-11-12 12:48:05] | (0:00:45.340820)
INFO:director_one_combine:Train Epoch: 130 [73%]
INFO:director_one_combine:[4800, 9.840033200544528e-05]
INFO:director_one_combine:loss_disc=3.695, loss_gen=3.669, loss_fm=9.757,loss_mel=17.843, loss_kl=1.051
INFO:director_one_combine:====> Epoch: 130 [2023-11-12 12:48:54] | (0:00:49.126917)
INFO:director_one_combine:====> Epoch: 131 [2023-11-12 12:49:43] | (0:00:48.830487)
INFO:director_one_combine:====> Epoch: 132 [2023-11-12 12:50:23] | (0:00:40.879059)
INFO:director_one_combine:====> Epoch: 133 [2023-11-12 12:51:09] | (0:00:45.473135)
INFO:director_one_combine:====> Epoch: 134 [2023-11-12 12:51:57] | (0:00:47.840981)
INFO:director_one_combine:====> Epoch: 135 [2023-11-12 12:52:46] | (0:00:48.972295)
INFO:director_one_combine:Train Epoch: 136 [14%]
INFO:director_one_combine:[5000, 9.832655481517557e-05]
INFO:director_one_combine:loss_disc=3.596, loss_gen=3.515, loss_fm=8.937,loss_mel=16.448, loss_kl=0.976
INFO:director_one_combine:====> Epoch: 136 [2023-11-12 12:53:26] | (0:00:40.316568)
INFO:director_one_combine:====> Epoch: 137 [2023-11-12 12:54:13] | (0:00:46.931682)
INFO:director_one_combine:====> Epoch: 138 [2023-11-12 12:55:03] | (0:00:49.589914)
INFO:director_one_combine:====> Epoch: 139 [2023-11-12 12:55:51] | (0:00:48.564263)
INFO:director_one_combine:====> Epoch: 140 [2023-11-12 12:56:30] | (0:00:38.711941)
INFO:director_one_combine:Train Epoch: 141 [54%]
INFO:director_one_combine:[5200, 9.826511608001993e-05]
INFO:director_one_combine:loss_disc=3.952, loss_gen=3.396, loss_fm=8.331,loss_mel=16.717, loss_kl=0.802
INFO:director_one_combine:====> Epoch: 141 [2023-11-12 12:57:20] | (0:00:49.601552)
INFO:director_one_combine:====> Epoch: 142 [2023-11-12 12:58:08] | (0:00:48.762239)
INFO:director_one_combine:====> Epoch: 143 [2023-11-12 12:58:57] | (0:00:49.030928)
INFO:director_one_combine:====> Epoch: 144 [2023-11-12 12:59:37] | (0:00:39.333258)
INFO:director_one_combine:====> Epoch: 145 [2023-11-12 13:00:25] | (0:00:48.351315)
INFO:director_one_combine:Train Epoch: 146 [95%]
INFO:director_one_combine:[5400, 9.820371573447515e-05]
INFO:director_one_combine:loss_disc=3.620, loss_gen=3.232, loss_fm=9.946,loss_mel=16.882, loss_kl=0.887
INFO:director_one_combine:====> Epoch: 146 [2023-11-12 13:01:15] | (0:00:49.529824)
INFO:director_one_combine:====> Epoch: 147 [2023-11-12 13:02:04] | (0:00:49.248937)
INFO:director_one_combine:====> Epoch: 148 [2023-11-12 13:02:41] | (0:00:37.310694)
INFO:director_one_combine:====> Epoch: 149 [2023-11-12 13:03:30] | (0:00:48.602043)
INFO:director_one_combine:====> Epoch: 150 [2023-11-12 13:04:18] | (0:00:48.254312)
INFO:director_one_combine:====> Epoch: 151 [2023-11-12 13:05:07] | (0:00:49.331202)
INFO:director_one_combine:Train Epoch: 152 [35%]
INFO:director_one_combine:[5600, 9.813008596033443e-05]
INFO:director_one_combine:loss_disc=3.546, loss_gen=3.506, loss_fm=9.483,loss_mel=17.246, loss_kl=0.903
INFO:director_one_combine:====> Epoch: 152 [2023-11-12 13:05:46] | (0:00:38.131470)
INFO:director_one_combine:====> Epoch: 153 [2023-11-12 13:06:34] | (0:00:48.954268)
INFO:director_one_combine:====> Epoch: 154 [2023-11-12 13:07:24] | (0:00:49.110085)
INFO:director_one_combine:====> Epoch: 155 [2023-11-12 13:08:11] | (0:00:47.361329)
INFO:director_one_combine:====> Epoch: 156 [2023-11-12 13:08:52] | (0:00:40.618684)
INFO:director_one_combine:Train Epoch: 157 [76%]
INFO:director_one_combine:[5800, 9.806876998751865e-05]
INFO:director_one_combine:loss_disc=4.037, loss_gen=3.369, loss_fm=8.652,loss_mel=17.446, loss_kl=0.986
INFO:director_one_combine:====> Epoch: 157 [2023-11-12 13:09:41] | (0:00:48.986298)
INFO:director_one_combine:====> Epoch: 158 [2023-11-12 13:10:30] | (0:00:49.337431)
INFO:director_one_combine:====> Epoch: 159 [2023-11-12 13:11:15] | (0:00:45.312014)
INFO:director_one_combine:Saving model and optimizer state at epoch 160 to ./logs/director_one_combine/G_5920.pth
INFO:director_one_combine:Saving model and optimizer state at epoch 160 to ./logs/director_one_combine/D_5920.pth
INFO:director_one_combine:====> Epoch: 160 [2023-11-12 13:12:02] | (0:00:46.424183)
INFO:director_one_combine:====> Epoch: 161 [2023-11-12 13:12:51] | (0:00:49.715272)
INFO:director_one_combine:====> Epoch: 162 [2023-11-12 13:13:39] | (0:00:48.027087)
INFO:director_one_combine:Train Epoch: 163 [16%]
INFO:director_one_combine:[6000, 9.79952413910655e-05]
INFO:director_one_combine:loss_disc=3.775, loss_gen=3.643, loss_fm=10.383,loss_mel=17.299, loss_kl=0.756
INFO:director_one_combine:====> Epoch: 163 [2023-11-12 13:14:23] | (0:00:43.624897)
INFO:director_one_combine:====> Epoch: 164 [2023-11-12 13:15:09] | (0:00:45.876518)
INFO:director_one_combine:====> Epoch: 165 [2023-11-12 13:15:58] | (0:00:49.191586)
INFO:director_one_combine:====> Epoch: 166 [2023-11-12 13:16:47] | (0:00:48.933571)
INFO:director_one_combine:====> Epoch: 167 [2023-11-12 13:17:27] | (0:00:39.522425)
INFO:director_one_combine:Train Epoch: 168 [57%]
INFO:director_one_combine:[6200, 9.79340096750387e-05]
INFO:director_one_combine:loss_disc=3.688, loss_gen=3.300, loss_fm=10.527,loss_mel=17.224, loss_kl=0.828
INFO:director_one_combine:====> Epoch: 168 [2023-11-12 13:18:14] | (0:00:47.310650)
INFO:director_one_combine:====> Epoch: 169 [2023-11-12 13:19:03] | (0:00:49.358381)
INFO:director_one_combine:====> Epoch: 170 [2023-11-12 13:19:52] | (0:00:48.529803)
INFO:director_one_combine:====> Epoch: 171 [2023-11-12 13:20:28] | (0:00:36.575045)
INFO:director_one_combine:====> Epoch: 172 [2023-11-12 13:21:16] | (0:00:47.492381)
INFO:director_one_combine:Train Epoch: 173 [97%]
INFO:director_one_combine:[6400, 9.787281621926815e-05]
INFO:director_one_combine:loss_disc=3.813, loss_gen=3.328, loss_fm=10.380,loss_mel=16.547, loss_kl=1.011
INFO:director_one_combine:====> Epoch: 173 [2023-11-12 13:22:06] | (0:00:49.742186)
INFO:director_one_combine:====> Epoch: 174 [2023-11-12 13:22:55] | (0:00:48.828573)
INFO:director_one_combine:====> Epoch: 175 [2023-11-12 13:23:33] | (0:00:38.673223)
INFO:director_one_combine:====> Epoch: 176 [2023-11-12 13:24:22] | (0:00:48.795012)
INFO:director_one_combine:====> Epoch: 177 [2023-11-12 13:25:11] | (0:00:48.723309)
INFO:director_one_combine:====> Epoch: 178 [2023-11-12 13:26:00] | (0:00:49.257818)
INFO:director_one_combine:Train Epoch: 179 [38%]
INFO:director_one_combine:[6600, 9.779943454222217e-05]
INFO:director_one_combine:loss_disc=4.044, loss_gen=3.332, loss_fm=9.463,loss_mel=17.469, loss_kl=0.902
INFO:director_one_combine:====> Epoch: 179 [2023-11-12 13:26:40] | (0:00:40.439676)
INFO:director_one_combine:====> Epoch: 180 [2023-11-12 13:27:29] | (0:00:48.892361)
INFO:director_one_combine:====> Epoch: 181 [2023-11-12 13:28:19] | (0:00:49.299453)
INFO:director_one_combine:====> Epoch: 182 [2023-11-12 13:29:08] | (0:00:48.836309)
INFO:director_one_combine:====> Epoch: 183 [2023-11-12 13:29:44] | (0:00:36.509195)
INFO:director_one_combine:Train Epoch: 184 [78%]
INFO:director_one_combine:[6800, 9.773832517488488e-05]
INFO:director_one_combine:loss_disc=3.800, loss_gen=3.986, loss_fm=9.375,loss_mel=16.792, loss_kl=1.277
INFO:director_one_combine:====> Epoch: 184 [2023-11-12 13:30:33] | (0:00:48.840992)
INFO:director_one_combine:====> Epoch: 185 [2023-11-12 13:31:23] | (0:00:49.610012)
INFO:director_one_combine:====> Epoch: 186 [2023-11-12 13:32:10] | (0:00:47.569556)
INFO:director_one_combine:====> Epoch: 187 [2023-11-12 13:32:48] | (0:00:38.073758)
INFO:director_one_combine:====> Epoch: 188 [2023-11-12 13:33:37] | (0:00:48.380059)
INFO:director_one_combine:====> Epoch: 189 [2023-11-12 13:34:25] | (0:00:48.912429)
INFO:director_one_combine:Train Epoch: 190 [19%]
INFO:director_one_combine:[7000, 9.766504433460612e-05]
INFO:director_one_combine:loss_disc=3.682, loss_gen=3.263, loss_fm=9.105,loss_mel=16.234, loss_kl=0.697
INFO:director_one_combine:====> Epoch: 190 [2023-11-12 13:35:13] | (0:00:47.175076)
INFO:director_one_combine:====> Epoch: 191 [2023-11-12 13:35:53] | (0:00:40.525764)
INFO:director_one_combine:====> Epoch: 192 [2023-11-12 13:36:42] | (0:00:48.340418)
INFO:director_one_combine:====> Epoch: 193 [2023-11-12 13:37:31] | (0:00:48.990884)
INFO:director_one_combine:====> Epoch: 194 [2023-11-12 13:38:17] | (0:00:46.463162)
INFO:director_one_combine:Train Epoch: 195 [59%]
INFO:director_one_combine:[7200, 9.760401894015275e-05]
INFO:director_one_combine:loss_disc=3.506, loss_gen=3.927, loss_fm=9.996,loss_mel=16.590, loss_kl=0.684
INFO:director_one_combine:====> Epoch: 195 [2023-11-12 13:38:59] | (0:00:42.149901)
INFO:director_one_combine:====> Epoch: 196 [2023-11-12 13:39:48] | (0:00:49.072897)
INFO:director_one_combine:====> Epoch: 197 [2023-11-12 13:40:37] | (0:00:48.938068)
INFO:director_one_combine:====> Epoch: 198 [2023-11-12 13:41:21] | (0:00:44.091045)
INFO:director_one_combine:====> Epoch: 199 [2023-11-12 13:42:07] | (0:00:45.559329)
INFO:director_one_combine:Saving model and optimizer state at epoch 200 to ./logs/director_one_combine/G_7400.pth
INFO:director_one_combine:Saving model and optimizer state at epoch 200 to ./logs/director_one_combine/D_7400.pth
INFO:director_one_combine:====> Epoch: 200 [2023-11-12 13:42:58] | (0:00:51.524712)
INFO:director_one_combine:Training is done. The program is closed.
INFO:director_one_combine:saving final ckpt:Success.
2023-11-12 13:43:01 | INFO | httpx | HTTP Request: POST http://localhost:7777/api/predict "HTTP/1.1 200 OK"
2023-11-12 13:43:01 | INFO | httpx | HTTP Request: POST http://localhost:7777/reset "HTTP/1.1 200 OK"
/usr/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 20 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '