vitis:fifo_depth_optimization fails for models with more than one input or output
Maintainers usually reply within 3 days
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 78/100
Research direction
Start in hls4ml/backends/vitis/passes/fifo_depth_optimization.py, in initialize_large_fifos, where the exclusion checks compare each variable only with the first input and first output. The issue proposes replacing those comparisons with membership tests against all inputs and outputs. Verify it with a two-input, two-output Keras model, calling initialize_large_fifos and checking that the second input and output are not returned. Done when that holds and the Vitis flow completes without the RuntimeError.
Written by the indexing model from the issue text.
Description
Quick summary
The FIFO depth optimization flow fails for models with more than one input or output. initialize_large_fifos leaves out only the first input and the first output of the model from profiling. The other inputs and outputs are profiled too, but they are ports of the top function, not FIFOs inside it, so they never appear in channel_info.csv. The last step then cannot find their depths and stops.
This happens on Vitis and on VitisUnified (axi_master), and is independent of #1570, which is fixed by #1571.
hls4ml version
#1571 at e1245b0cd. Without #1571 the flow stops earlier, with the KeyError from #1570.
Frontend
Keras v2
Backend
Vitis
io_type
io_stream
HLS tool and version
Vitis HLS 2024.1
Steps to reproduce
import hls4ml
from tensorflow.keras.layers import Concatenate, Dense, Input
from tensorflow.keras.models import Model
in_a = Input((16,), name='in_a')
in_b = Input((16,), name='in_b')
h = Concatenate(name='cat')([Dense(8, name='da')(in_a), Dense(8, name='db')(in_b)])
h = Dense(8, name='dh')(h)
model = Model([in_a, in_b], [Dense(4, name='oa')(h), Dense(2, name='ob')(h)])
config = hls4ml.utils.config_from_keras_model(model, granularity='name', default_precision='ap_fixed<16, 6>')
config['Flows'] = ['vitis:fifo_depth_optimization']
hls_model = hls4ml.converters.convert_from_keras_model(
model,
hls_config=config,
io_type='io_stream',
output_dir='fifo_multi_io_prj',
backend='Vitis',
part='xcu250-figd2104-2L-e',
)
Expected behaviour
The flow finishes, and fifo_depths.json holds an optimized depth for every FIFO between the layers. The model's inputs and outputs are not profiled.
Actual behaviour
Synthesis and co-simulation complete, then the flow stops:
RuntimeError: FIFO depth optimization could not find the profiled depths of in_b, layer12_out in fifo_multi_io_prj/myproject_prj/solution1/.autopilot/db/channel_info.csv. Make sure that the co-simulation completed successfully.
in_b is the second input and layer12_out the second output. fifo_depths.json lists them with the FIFOs to profile:
{"in_b": {"initial": 1}, "layer3_out": {"initial": 1}, "layer5_out": {"initial": 1}, "layer7_out": {"initial": 1}, "layer8_out": {"initial": 1}, "layer12_out": {"initial": 1}, "layer14_cpy1": {"initial": 1}, "layer14_cpy2": {"initial": 1}}
channel_info.csv has rows only for the FIFOs inside the top function:
,layer3_out_U,,chan_status1.csv
,layer5_out_U,,chan_status2.csv
,layer7_out_U,,chan_status3.csv
,layer8_out_U,,chan_status4.csv
,layer14_cpy1_U,,chan_status5.csv
,layer14_cpy2_U,,chan_status6.csv
With backend='VitisUnified', board='zcu102' and axi_mode='axi_master', the same model stops with the same error for in_b and layer12_out.
Possible fix or additional context
initialize_large_fifos in hls4ml/backends/vitis/passes/fifo_depth_optimization.py compares each variable only with the first input and the first output:
and output_variable != model.get_output_variables()[0]
and output_variable != model.get_input_variables()[0]
Comparing with all of them fixes it:
and output_variable not in model.get_output_variables()
and output_variable not in model.get_input_variables()
With this change the model above goes through on Vitis, and the co-simulation with the optimized depths passes.
The Vivado FIFO optimizer has the same check (hls4ml/backends/vivado/passes/fifo_depth_optimization.py, line 76). I have not run the Vivado flow, so I don't know whether it fails the same way there.
The check can be tested without any tools: convert a model with two inputs and two outputs, call initialize_large_fifos, and check that the second input and the second output are not among the FIFOs it returns.
AI assistance
- An AI tool helped produce this report, and I have checked its content myself.
- Dominant language
- Python
- Stars
- 2.2k
- Forks
- 605
- Avg merge
- 7d 35m
- Merged PRs (30d)
- 19
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from fastmachinelearning/hls4ml
-
bug Stale
Difficulty 1/5 Under an hour Newbie friendliness 88/100
fastmachinelearning/hls4ml#771 · 1 comment ·
Maintainers usually reply within 3 days
-
error while converting onnx model to hlsMay be free again A pull request for this issue was closed without being merged. Openbug Stale
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
fastmachinelearning/hls4ml#686 · 2 comments ·
Maintainers usually reply within 3 days
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
fastmachinelearning/hls4ml#630 ·
Maintainers usually reply within 3 days
-
Stale
Difficulty 1/5 Under an hour Newbie friendliness 82/100
fastmachinelearning/hls4ml#297 · 4 comments ·
Maintainers usually reply within 3 days
-
Stale
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
fastmachinelearning/hls4ml#265 · 2 comments ·
Maintainers usually reply within 3 days
All issues in fastmachinelearning/hls4ml
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
UKGovernmentBEIS/control-arena#917 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
debpalash/VoiceStudio#2937 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
aws-samples/sample-per-model-bedrock#261 ·
Maintainers usually reply within 1 day