RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! When predicting with my model

The RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! When predicting with my model error occurs when the input tensors and the model are on different devices (e.g., one on CPU and the other on GPU).

To fix the RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! When predicting with my model error, make sure both the input tensors and the model are on the same device, either CPU or GPU.

Load your model and move it to the desired device (GPU in this case)

import torch
from transformers import AutoModel

model = AutoModel.from_pretrained("model_name")
device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)

Move your input tokens to the same device

tokens = tokenizer("Your input text", return_tensors="pt")
tokens = tokens.to(device)

Perform the prediction with both the model and input tokens on the same device

with torch.no_grad():
  output = model(**tokens)

By ensuring that both the model and input tokens are on the same device, you should no longer encounter the RuntimeError.

That’s it.

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.