# 如何进行模型预训练

**URL:** <https://forum.beginner.center/t/topic/2760>\
**Category:** 🛠工具与编程\
**Tags:** 大模型\
**Created:** [2026 年4 月 29 日 03:47 UTC](https://forum.beginner.center/t/topic/2760 "2026-04-29T03:47:08Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

作者： ![doggie](https://forum.beginner.center/user_avatar/forum.beginner.center/doggie/32/550_2.png) [@doggie](https://forum.beginner.center/u/doggie)\
发布日期： [2026 年4 月 29 日 03:47 UTC](https://forum.beginner.center/t/topic/2760/1 "2026-04-29T03:47:08Z")

</div>

## 操作

```python
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "1"
from peft import TaskType, LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, LlamaConfig, LlamaForCausalLM,LlamaModel
import torch

# 加载模型
model_path = '/data04/llama3/Meta-Llama-3.1-8B-Instruct'
tokenizer = AutoTokenizer.from_pretrained(model_path)

model = AutoModelForCausalLM.from_pretrained(
    model_path,
    low_cpu_mem_usage=True,
    quantization_config=bnb_config
)

model = get_peft_model(model, peft_config)
model.print_trainable_parameters()
# 把模型放进显卡
model.to("cuda")
optimizer = torch.optim.AdamW(model.parameters())
# 把输入内容分词，然后放入显卡
text = "今天天气不错。"
input = tokenizer(text, return_tensors="pt")
input = {k: v.to("cuda") for k, v in input.items()}

#设置labels和inputs一致
input["labels"] = input["input_ids"].clone()

# 前向传播
output = model(**input)

#获取模型的loss
loss = output.loss
# 后向传播
loss.backward()
# 更新参数
optimizer.step()
optimizer.zero_grad()

#保存模型
model.save_pretrained("output_dir")

```

## loss函数如何计算？

大模型输出结果本质上就是词的分类，使用[交叉熵](https://forum.beginner.center/t/topic/2432)进行loss计算

## 参考

[大模型预训练看这个视频就够了\_哔哩哔哩\_bilibili](https://www.bilibili.com/video/BV1xhYtewEum)
