Skip to main content

​
Getting Started with Self-Hosting Checkpoints

Here is how you can download the checkpoints from the studio after you have fine-tuned your model. Once you have downloaded the checkpoints, you can unzip them and use the following inference engines to load the checkpoints and use them for inference. Download Checkpoints

​
Inference Engines

You can use the following inference engines to load the checkpoints and use them for inference.

​
Hugging Face

You can use the Hugging Face library to load the checkpoints and use them for inference using the transformers library. 
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch


model_path = 'path/to/your/finetuned/model/checkpoint'
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(model_path, torch_dtype=torch.bfloat16, device_map="auto")

SYSTEM_PROMPT = """You are a helpful recipe assistant. You are to extract the generic ingredients from each of the recipes provided."""

USER_PROMPT = """Title: Lemon Drizzle Cake

Ingredients: ["200g unsalted butter", "200g caster sugar", "4 eggs", "200g self-raising flour", "1 tsp baking powder", "zest of 1 lemon", "100ml lemon juice", "150g icing sugar"]

Generic ingredients:"""

conversation = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": USER_PROMPT},
]

# format and tokenize the tool use prompt
inputs = tokenizer.apply_chat_template(conversation, add_generation_prompt=True, return_dict=True, return_tensors="pt")

inputs.to(model.device)
outputs = model.generate(**inputs, max_new_tokens=1000, use_cache=False)
print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

​
VLLM

You can use the VLLM library to load the checkpoints and use them for inference using the VLLM library. 
from vllm import LLM, SamplingParams

model_path = 'path/to/your/finetuned/model/checkpoint'
llm = LLM(model=model_path, tokenizer=model_path)

SYSTEM_PROMPT = """You are a helpful recipe assistant. You are to extract the generic ingredients from each of the recipes provided."""

USER_PROMPT = """Title: Lemon Drizzle Cake

Ingredients: ["200g unsalted butter", "200g caster sugar", "4 eggs", "200g self-raising flour", "1 tsp baking powder", "zest of 1 lemon", "100ml lemon juice", "150g icing sugar"]

Generic ingredients:"""

conversation = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": USER_PROMPT},
]

outputs = llm.chat(
    messages=conversation,
    sampling_params=SamplingParams(temperature=0, max_tokens=256),
)
print(outputs[0].outputs[0].text)

​
Ollama

You can use the Ollama library to load the checkpoints and use them for inference using the Ollama library. To create a model, you first need to create a Modelfile: 
FROM path/to/finetuned/model/checkpoint
Then run the following command to create the model. 
ollama create my-model -f Modelfile
Then you can use the model for inference: 
from ollama import Client

client = Client(host='http://localhost:11434')

SYSTEM_PROMPT = """You are a helpful recipe assistant. You are to extract the generic ingredients from each of the recipes provided."""

USER_PROMPT = """Title: Lemon Drizzle Cake

Ingredients: ["200g unsalted butter", "200g caster sugar", "4 eggs", "200g self-raising flour", "1 tsp baking powder", "zest of 1 lemon", "100ml lemon juice", "150g icing sugar"]

Generic ingredients:"""

conversation = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": USER_PROMPT},
]
res = client.chat(model='my-model', messages=conversation)
print(res.message.content)
⌘I