Sunday, August 11, 2024

Python completions. I found out that the hugging chat text completion I used works with a message generator object.

 Ran into a problem with the hugging chat text completion and learned about message generators in python. You have to accumulate the response


chatbot = hugchat.ChatBot(cookies=cookies.get_dict(), default_llm=hugging_model) # or cookie_path="usercookies/<email>.json"
message_generator = chatbot.chat(hugging_request) # note: message_result is a generator, the method will return immediately.

# Collect results from the generator
message_result = ""
for part in message_generator:
if (isinstance(part,dict)) and 'token' in part:
message_result += part['token']

time.sleep(1)
return {output[0]: message_result}, {"status": {"value": 0, "reason": "Success"}}


The generator has to slowly build up the message_result one character at at time. (Pdb) print (part) 
{'type': 'stream', 'token': ' format'}
That is the format for part. But it can also return none. Now it is confusing because print (message_generator) actually gives you the full output, so print has that accumulation of message generators built in. This also implies that message generators are similar. Not quite sure if this works, the big model I was trying to use became rate limited. Will try again later to see if I finally fixed it.

No comments:

Post a Comment