The robust and most efficient way to remove duplicates from a list in Python is to use the “set()” function. Pass the list to the set() constructor, which will return a list with unique values. Then, convert it back to a list using the “list()” constructor.

main_list = [11, 21, 19, 21, 11, 46, 19]

print("List before removing duplicate elements: ", main_list)

# Output: List before removing duplicate elements:  [11, 21, 19, 21, 11, 46, 19]

unique_list = list(set(main_list))

print("List after removing duplicate elements: ", unique_list)

# Output: List after removing duplicate elements:  [19, 11, 21, 46]

The set() is faster and cleaner, but it isn’t suitable if the original order is essential.

Alternate approaches

1. Using a for loop
2. Using dict.fromKeys()
3. Using list comprehension

Method 1: Using a for loop (Order is preserved)

If you want to preserve the elements of a list while removing duplicates, you can use the “for loop” to iterate through the list and keep track of elements that are already seen.

Initialize an empty set and an empty list. Then, loop through each element using a for loop in the original list: if the element is not in the set, add it to the new list and the set. This way, we maintain the order and avoid duplicates.

# Initialize a list with some duplicate elements
main_list = [11, 21, 19, 21, 11, 46, 19]

print("List before removing duplicate elements: ", main_list)
# Output: List before removing duplicate elements: [11, 21, 19, 21, 11, 46, 19]

result = []
for item in main_list:
  if item not in result:
    result.append(item)

print("List after removing duplicate elements: ", result)
# Output: List before removing duplicate elements: [11, 21, 19, 46]

You can see from the output that the order of the elements is preserved even after removing duplicates.

Handling Unhashable Elements (List of Lists)

If the elements of the list are unhashable, like a list of lists, we can temporarily convert them to a hashable type.

For example, if we have a list of lists, we can convert each inner list to a tuple (which is hashable), then use the set() or dict() approach, and then convert back.

# Initialize a list of lists
main_list = [[1, 2], [3, 4], [1, 2], [5, 6], [7, 8], [5, 6]]

print("List before removing duplicate lists: ", main_list)
# Output: List before removing duplicate elements: [[1, 2], [3, 4], [1, 2], [5, 6], [7, 8], [5, 6]]

seen = set()
result = []
for sublist in main_list:
    t = tuple(sublist)
    if t not in seen:
        seen.add(t)
        result.append(sublist)

print("List after removing duplicate lists: ", result)
# Output: List after removing duplicate elements: [[1, 2], [3, 4], [5, 6], [7, 8]]

Method 2: Using dict.fromKeys() 

From Python 3.7+, dictionaries preserve insertion order.

The dict.fromkeys() method will create a dictionary where the keys are the list elements in order and then convert the keys back to a list.

So, list(dict.fromkeys(main_list)) would give a list with duplicates removed.

# Initialize a list with some duplicate elements
main_list = [11, 21, 19, 21, 11, 46, 19]

print("List before removing duplicate elements: ", main_list)
# Output: List before removing duplicate elements:  [11, 21, 19, 21, 11, 46, 19]

unique_list = list(dict.fromkeys(main_list))

print("List after removing duplicate elements: ", unique_list)
# Output: List before removing duplicate elements:  [11, 21, 19, 46]

This approach is generally faster than the loop method when working with a larger list.

Method 3: Using list comprehension

List comprehension effectively maintains the original order of elements within a list.

# Initialize a list with some duplicate elements
main_list = [11, 21, 19, 21, 11, 46, 19]

print("List before removing duplicate elements: ", main_list)
# Output: List before removing duplicate elements:  [11, 21, 19, 21, 11, 46, 19]

unique_list = []
[unique_list.append(x) for x in main_list if x not in unique_list]

print("List after removing duplicate elements: ", unique_list)
# Output: List before removing duplicate elements:  [11, 21, 19, 46]

This approach is a one-liner, but it is less readable and less efficient for larger lists.

That’s all!