Why do we need to use temporary folders and files? Well, while executing a program, lots of things happen behind the scenes, and you need to store the data temporarily. For that, we need to create a temp file, which must be in a secure location and either unnamed or unpredictable in name.

Here is the figure summary for this article:

Creating a temporary file

Python has a built-in module called “tempfile” that provides the .TemporaryFile() and .TemporaryNamedFile() methods. Those methods will create a temporary file (without or with name) efficiently.

The tempfile.Temporary() function creates a file-like object that stores the data without needing a filesystem.

Always use the context manager like “with statement” for automatic cleanup. Always remember to close the tempfile unless you need to keep it open. It prevents orphaned files or directories from cluttering the system.

The permission of the temp file is always 0o600, which you can read and write.

Unnamed temp file

If you want a temp file for a shorter amount of time and it is not necessary to have a name, you can use the tempfile.Temporary() method.

Before going further, you need to import tempfile at the head of your file. Then, we can use its methods.

import tempfile

with tempfile.TemporaryFile() as temp_file:
    temp_file.write(b"Hello mate, from temp file!")
    temp_file.seek(0)
    content = temp_file.read()
    print(content.decode())

# Output: Hello mate, from temp file!

Even though it is a temp file with no name, it will still occupy space in memory.

On macOS, I can see the tempfiles like this:

It temporarily stores on the complex location like this: /var/folders/8k/x52znv0n7r53eebnmngj4kmw0400gn/T

The file has no visible names, so it is safe for sensitive data.

If you are working with very large datasets, always use the TemporaryFile() method. It uses disk storage but is optimized for temporary data that does not persist.

Named temp file

The tempfile.NamedTemporaryFile() method creates a named file. A named temp file is helpful when another process wants to use it.

from tempfile import NamedTemporaryFile

with NamedTemporaryFile(mode='w+', delete=True, suffix='.log') as tmp:
    print(f"Temporary file path: {tmp.name}")
    # File is deleted when `delete=True` (default) after context exits.

# Output: Temporary file path: /var/folders/8k/x42znv0n7r5383bnmngk4kmw0900gn/T/tmpuvh9_37x.log

By default, when the “with” context is over, the file will be deleted automatically. If you pass delete = False, you need to clear that temp file manually.

To customize the file name, you can use the “suffix/prefix” argument.

The above code will work fine on MacOS or Linux, but it might have a problem with Windows OS.

On Windows, when you create a temp file using the above approach, the OS locks it automatically. To handle this, you need to pass delete=false and then manually delete the file using the OS.remove() method.

import tempfile
import os

with tempfile.NamedTemporaryFile(suffix=".txt", delete=False) as named_temp:
    named_temp.write(b"Temporary file with a name!")
    print("File name:", named_temp.name)

# Deleting the file manually if delete=False
os.remove(named_temp.name)

# Output: your windows path

Define a custom temp directory

You can use the “dir” parameter in the NamedTemporaryFile() method to specify a custom base directory.

NamedTemporaryFile(dir='/var/folders/8k/tmp')

Using SpooledTemporaryFile()

There is also another function from the tempfile module, which is SpooledTemporaryFile(). It optimizes small file handling by spooling data in memory.

from tempfile import SpooledTemporaryFile

with SpooledTemporaryFile(max_size=1024) as tmp:
    tmp.write(b'Small data')  # Stays in memory
    tmp.rollover()  # Force write to disk
    print(tmp)
    
# Output: <tempfile.SpooledTemporaryFile object at 0x102a2fb50>

Using mkstemp()

If you want lower-level control over temporary files, use the tempfile.mkstemp() method. It returns a tuple containing an OS-level file descriptor and the path to the temporary file.

import os
from tempfile import mkstemp

# Create a temporary file
fd, path = mkstemp()
try:
    with os.fdopen(fd, 'w+') as tmp:
        tmp.write('Manual cleanup example')
        print(path)
finally:
    os.remove(path)  # Manual cleanup
    
# Output: /var/folders/8k/x42znv0n7r5383bnnmgl4kmw0009gn/T/tmp7kcdwijz

Your temp file path will be different from mine, and it depends on the system’s configuration.

Creating a temporary directory

In Python3, you can use the tempfile.TemporaryDirectory() method to create a temporary directory. To automatically delete it, we should combine this function with the “with” context manager.

from tempfile import TemporaryDirectory

with TemporaryDirectory() as dirpath:
    print(f"Temporary directory created at: {dirpath}")
    # Work with files inside the directory here
    with open(f"{dirpath}/temp_file.txt", "w") as f:
        f.write("Hello from the temp directory!")
# Directory and all contents are deleted here

# Output: Temporary directory created at:
# /var/folders/8l/x42qnv0n7r5383bnmngj5kmw0000gm/T/tmp21xelljj

The above code will generate a temp folder, and when the “with” block exists, this folder will be deleted automatically.

You can control the output temp directory’s name using suffix, prefix, and dir arguments like this:

from tempfile import TemporaryDirectory

with TemporaryDirectory(
    prefix="mydata_", 
    suffix="_tmp", 
    dir="/custom/path"
) as dirpath:
    print(f"Custom directory: {dirpath}")  
    
# Output: Custom directory: /custom/path/mydata_a8b3kfp2_tmp

If automatic cleanup fails somehow, you can use the .cleanup() method.

from tempfile import TemporaryDirectory

temp_dir = TemporaryDirectory()

dirpath = temp_dir.name

print(f"Directory: {dirpath}")

# Manually deleting the directory
temp_dir.cleanup()
# Directory is now gone

# Output: Directory: /var/folders/8k/x42zmn0n7r5383bnmnkb4kmw0000kl/T/tmpphe2bb4b

Using mkdtemp()

For getting lower-level control, you can use the tempfile.mkdtemp() method. It creates a temporary directory and returns its path.

from tempfile import mkdtemp
import os

# Create a temporary directory using mkdtemp()
dir_path = mkdtemp()
try:
    print(f"Directory: {dir_path}")
finally:
    os.rmdir(dir_path) 
    
# Output: Directory: /var/folders/8m/x21znv0n7r5383bnmngj4kmw0000kb/T/tmpx0c8i6bc

That’s all!