ONNX file for text generation – python
In this video it shows the steps and code to create a local ONNX file and use it for text generation. It uses GPT2 LLM model from transformer library of hugging face. The complete application is developed in python code.
I hope you like this video. For any questions, suggestions or appreciation please contact us at: https://programmerworld.co/contact/ or email at: programmerworld1990@gmail.com
Details:
Python code:
import torch
from transformers import GPT2Tokenizer, GPT2LMHeadModel
import onnx
from onnxruntime import InferenceSession
# Load the pre-trained GPT-2 model and tokenizer
model_name = "gpt2"
model = GPT2LMHeadModel.from_pretrained(model_name)
tokenizer = GPT2Tokenizer.from_pretrained(model_name)
# Create dummy input for the model
text = "Write a story about sunflowers"
inputs = tokenizer(text, return_tensors="pt")
dummy_input = inputs['input_ids']
# Export the model to ONNX with dynamic axes
onnx_file_path = "gpt2_model.onnx"
torch.onnx.export(model, dummy_input, onnx_file_path,
input_names=['input_ids'],
output_names=['logits'],
dynamic_axes={'input_ids': {0: 'batch_size', 1: 'sequence_length'},
'logits': {0: 'batch_size', 1: 'sequence_length'}},
opset_version=14)
# Validate the ONNX model
onnx_model = onnx.load(onnx_file_path)
onnx.checker.check_model(onnx_model)
print("The model is valid and has been saved as 'gpt2_model.onnx'!")
# Perform inference using the ONNX model with dynamic input size
session = InferenceSession(onnx_file_path)
# Function to apply top-p (nucleus) sampling
def top_p_sampling(logits, top_p=0.9):
sorted_logits, sorted_indices = torch.sort(logits, descending=True)
cumulative_probs = torch.cumsum(torch.nn.functional.softmax(sorted_logits, dim=-1), dim=-1)
sorted_indices_to_remove = cumulative_probs > top_p
sorted_indices_to_remove[..., 1:] = sorted_indices_to_remove[..., :-1].clone()
sorted_indices_to_remove[..., 0] = 0
for batch_idx in range(sorted_indices.size(0)):
logits[batch_idx, sorted_indices[batch_idx][sorted_indices_to_remove[batch_idx]]] = -float('Inf')
return logits
# Function to generate text iteratively with top-p sampling and repetition penalty
def generate_text(session, tokenizer, initial_input, max_length=1000, top_p=0.9, repetition_penalty=1.2):
input_ids = initial_input
generated_text = ""
for _ in range(max_length):
onnx_inputs = {"input_ids": input_ids.numpy()}
onnx_outputs = session.run(None, onnx_inputs)
logits = onnx_outputs[0]
logits = torch.tensor(logits[:, -1, :]) / repetition_penalty # Apply repetition penalty
filtered_logits = top_p_sampling(logits, top_p=top_p) # Apply top-p sampling
probabilities = torch.nn.functional.softmax(filtered_logits, dim=-1)
next_token_id = torch.multinomial(probabilities, 1).unsqueeze(0)
next_token_id = next_token_id.squeeze(1) # Ensure correct dimensions
input_ids = torch.cat([input_ids, next_token_id], dim=-1)
generated_text = tokenizer.decode(input_ids[0])
# Optionally, break if EOS token is generated
if next_token_id.item() == tokenizer.eos_token_id:
break
return generated_text
# Generate longer text
generated_text = generate_text(session, tokenizer, dummy_input, max_length=1000)
print("Generated text:", generated_text)
Screenshots:
Sample Text Generation:
The model is valid and has been saved as 'gpt2_model.onnx'!
Generated text: Write a story about sunflowers. After Cleaning Hotkey 'oo2tr3X' for this thread I try) Rest Tools Setup Run Ultra SimpleRemagen Synch.ini for Google Chrome.cmd Reboot Copy RootFiz to project folder. Use GRUB installation from Path Off overlay MonolithicFileLogFile(CA Drive,OLF Online This guide only gives you enough info on getting to these sets)Stop, Start, Off, RestorationsGroundZoneLet Windows Get up memory and prepare Microsoft System 8 StationsET.Put/Open database now get other background and system rewrites alley show run general editor for increasing windows permissions GPU Prompt Location/command queries Helpings Note this helps, it can be used to get virtual AD abuse.If you want boost stock system just adx config MDS9 Syn in Eclipse-a made it in Rudge SD3Editor 9.4 O keeber suit of fighters MS synros rc M DS2 Wire PoII Dec having desktop on me will heal or react bugs.Windows: Cannot multi address computer ES Cinnamoninst a bufferComplete Command line output:
(virtualpython) C:\Tools\Python\llm_model>python generate_text_from_onnx.py
C:\Tools\Python\virtualpython\Lib\site-packages\transformers\modeling_attn_mask_utils.py:88: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if input_shape[-1] > 1 or self.sliding_window is not None:
C:\Tools\Python\virtualpython\Lib\site-packages\transformers\modeling_attn_mask_utils.py:164: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if past_key_values_length > 0:
The model is valid and has been saved as 'gpt2_model.onnx'!
Generated text: Write a story about sunflowers. After Cleaning Hotkey 'oo2tr3X' for this thread I try) Rest Tools Setup Run Ultra SimpleRemagen Synch.ini for Google Chrome.cmd Reboot Copy RootFiz to project folder. Use GRUB installation from Path Off overlay MonolithicFileLogFile(CA Drive,OLF Online This guide only gives you enough info on getting to these sets)Stop, Start, Off, RestorationsGroundZoneLet Windows Get up memory and prepare Microsoft System 8 StationsET.Put/Open database now get other background and system rewrites alley show run general editor for increasing windows permissions GPU Prompt Location/command queries Helpings Note this helps, it can be used to get virtual AD abuse.If you want boost stock system just adx config MDS9 Syn in Eclipse-a made it in Rudge SD3Editor 9.4 O keeber suit of fighters MS synros rc M DS2 Wire PoII Dec having desktop on me will heal or react bugs.Windows: Cannot multi address computer ES Cinnamoninst a buffer
Maths All Linux:
I. Always mount white commands u configure 2 auth logged in v. Unmount bin writ on master lo device None more generic Versmouse Perms Working Unity time-img eg from Timeshifted cheannels: these acts to put as many light value patches as SRT/PPSWhen viewing MOAB PROGor aerodynamics Scrol official keyboard hexclock Chloroplaw Slabs 8W
Compred: backup xsrate addr centmcouts record 201 mbageshare ringi enum windowpath windows there with 12 max value ie60 ruma volunteer tom chint REhome 1 holdquit jmmmm tuned perclear byX YES OS [ TrustOp] ALL VS Regression FrametablesHardware FX yet u wont get fotografchedoff let GUI summarize's RAMmarty 3mx SyncDim 70 dterminal lostaries theoomfber that spded back to wfs fewxp u caprate oniy even though Monitors work sat 0000 reset MB/sec rolling up a tool set integrity on SE installszenw482 Data x2000 apop panic 300msec Saf loginALize some more SCHiscre of siw reviews connect wifo Faster Niram Case Speed all of it didf wheater elbowic Next thread thread August 16 FRS10 ScreenD Channel Show More then double I started to get wheel config done.Surprised I made sure the rear window octave is half way out & then the window shortcuts (second ones's crucial most polished filch), when I saw my following enter master debaved shell appeared using visual scripts, where es'm dwcurrent I set dxn distribution format default +base 100 (0100 - 3060 QWCEQ)!Sash Utility Copy Dataslate MegaStore outdated system-wide files (&[SC uses cdimagedrop.exe] PoWEm rune darkol overnight carrying missile of reqdmtribution format default +base 100 (0100 - 3060 QWCEQ)!Sash Utility Copy Dataslate MegaStore outdated system-wide files (&[SC uses cdimagedrop.exe] PoWEm rune darkol overnight carrying missile of reqdmtools addedthreaded tunfridomuf resources grf Transpr Bin UTA repro addon. programsmap is trying one of seedenhttp dump ax 9dump 100Mbers helpful cp/iconnamessource amdameritech GIS 4 2018 Ext Resolution Booster Mirrored Mass van Stone crashat last restored normal downloadshaavcool headsethatar 1 amp out iso 6 Channel menu riftart controls mercis ONARC2 unfoodlist hal787 04 minute invent omsm 4 fmtvsn Booster Mirrored Mass van Stone crashat last restored normal downloadshaavcool headsethatar 1 amp out iso 6 Channel menu riftart controls mercis ONARC2 unfoodlist hal787 04 minute invent omsm 4 fmtvs json admin remains
Opta Institute FX is running my rag loker dished my works papers.. may have seen trailers Take marl native bandwidth assuncabled bai11 average cylinder lagude wandsyn 3 shot presets than you adjust, for 130ga those do one thing boosts stability.... cranked ofi'tw has nothing weird. all 6 preset default run i5 5870 mesh polplace buffer =$ ps % uarelen pc dim relationship quickmore stall effective funn Booster Mirrored Mass van Stone crashat last restored normal downloadshaavcool headsethatar 1 amp out iso 6 Channel menu riftart controls mercis ONARC2 unfoodlist hal787 04 minute invent omsm 4 fmtvs json admin remains
Opta Institute FX is running my rag loker dished my works papers.. may have seen trailers Take marl native bandwidth assuncabled bai11 average cylinder lagude wandsyn 3 shot presets than you adjust, fon Booster Mirrored Mass van Stone crashat last restored normal downloadshaavcool headsethatar 1 amp out iso 6 Channel menu riftart controls mercis ONARC2 unfoodlist hal787 04 minute invent omsm 4 fmtvs json admin remains
n Booster Mirrored Mass van Stone crashat last restored normal downloadshaavcool headsethatar 1 amp out iso 6 Channel menu riftart controls mercis ONARC2 unfoodlist hal787 04 minute invent omsm 4 fmtvs json admin remains
n Booster Mirrored Mass van Stone crashat last restored normal downloadshaavcool headsethatar 1 amp out iso 6 Channel menu riftart controls mercis ONARC2 unfoodlist hal787 04 minute invent omsm 4 fmtvsn Booster Mirrored Mass van Stone crashat last restored normal downloadshaavcool headsethatar 1 amp out iso 6 Channel menu riftart controls mercis ONARC2 unfoodlist hal787 04 minute invent omsm 4 fmtvsn Booster Mirrored Mass van Stone crashat last restored normal downloadshaavcool headsethatar 1 amp out iso 6 Channel menu riftart controls mercis ONARC2 unfoodlist hal787 04 minute invent omsm 4 fmtvs json admin remains
Opta Institute FX is running my rag loker dished my works papers.. may have seen trailers Take marl native bandwidth assuncabled bai11 average cylinder lagude wandsyn 3 shot presets than you adjust, for 130ga those do one thing boosts stability.... cranked ofi'tw has nothing weird. all 6 preset default run i5 5870 mesh polplace buffer =$ ps % uarelen pc dim relationship quickmore stall effective function usage situational militaryfactor hawk microscope adeptine<|endoftext|>ONNX model created:
CPU resource consumption during the Text Genetation:
