Description
Is it possible to modify the C++ to create a second example source code file, that loads the model once, before sending new prompts read in a loop from STDIN?
After backing up the original C++ source code file, I modified the code to read a prompt from STDIN in a loop, instead of argv. There were no errors going through the loop, generating responses, except I seem to be processing new responses from the first prompt read from STDIN, over again, instead processing the new subsequent prompts read from STDIN.
The funny part, is these unintended results may be useful for prompt engineering in the future, to keep the context. But first, the goal would be to try save time by avoiding reloads of the model for generating responses to each new prompt in a loop. Lastly, this is a suggestion for a second separate example source code file. The first example code source code file is correct and very useful.