Python lists can contain duplicate elements and you might need to remove them to keep unique elements. For example, in list = [1, 2, 3, 2], 2 is a duplicate value because it appears twice. We need to remove the duplicate 2 from the list.
Python does not provide any built-in functions to remove duplicates. However, you can use one of the four below-mentioned approaches that you can use based on your requirements:
- Using set()
- Using a for loop
- Using dict.fromKeys()
- Using list comprehension
Method 1: Using set() (Order is not preserved)
If your requirement does not tell you to preserve the order of the final unique list, you can use the set() constructor. By default, it does not allow duplicate values.
So, if you pass the list with duplicate values to the set() constructor, it will remove them and return only distinct values. Convert the set back to a list using the list() method.
The set() is faster and cleaner, but it isn’t suitable if the original order is important.
# Initialize a list with some duplicate elements main_list = [11, 21, 19, 21, 11, 46, 19] print("List before removing duplicate elements: ", main_list) # Output: List before removing duplicate elements: [11, 21, 19, 21, 11, 46, 19] unique_list = list(set(main_list)) print("List after removing duplicate elements: ", unique_list) # Output: List after removing duplicate elements: [19, 11, 21, 46]
Method 2: Using a for loop (Order is preserved)
If you want to preserve the elements of a list while removing duplicates, you can use the “for loop” to iterate through the list and keep track of elements that are already seen.
Initialize an empty set and an empty list. Then, loop through each element using a for loop in the original list: if the element is not in the set, add it to the new list and the set. This way, we maintain the order and avoid duplicates.
# Initialize a list with some duplicate elements main_list = [11, 21, 19, 21, 11, 46, 19] print("List before removing duplicate elements: ", main_list) # Output: List before removing duplicate elements: [11, 21, 19, 21, 11, 46, 19] result = [] for item in main_list: if item not in result: result.append(item) print("List after removing duplicate elements: ", result) # Output: List before removing duplicate elements: [11, 21, 19, 46]
You can see from the output that the order of the elements is preserved even after removing duplicates.
Handling Unhashable Elements (List of Lists)
If the elements of the list are unhashable, like a list of lists, we can temporarily convert them to a hashable type.
For example, if we have a list of lists, we can convert each inner list to a tuple (which is hashable), then use the set() or dict() approach, and then convert back.
# Initialize a list of lists main_list = [[1, 2], [3, 4], [1, 2], [5, 6], [7, 8], [5, 6]] print("List before removing duplicate lists: ", main_list) # Output: List before removing duplicate elements: [[1, 2], [3, 4], [1, 2], [5, 6], [7, 8], [5, 6]] seen = set() result = [] for sublist in main_list: t = tuple(sublist) if t not in seen: seen.add(t) result.append(sublist) print("List after removing duplicate lists: ", result) # Output: List after removing duplicate elements: [[1, 2], [3, 4], [5, 6], [7, 8]]
Method 3: Using dict.fromKeys()
From Python 3.7+, dictionaries preserve insertion order.
The dict.fromkeys() method will create a dictionary where the keys are the list elements in order and then convert the keys back to a list.
So, list(dict.fromkeys(main_list)) would give a list with duplicates removed.
# Initialize a list with some duplicate elements main_list = [11, 21, 19, 21, 11, 46, 19] print("List before removing duplicate elements: ", main_list) # Output: List before removing duplicate elements: [11, 21, 19, 21, 11, 46, 19] unique_list = list(dict.fromkeys(main_list)) print("List after removing duplicate elements: ", unique_list) # Output: List before removing duplicate elements: [11, 21, 19, 46]
This approach is generally faster than the loop method if you are working with a larger list.
Method 4: Using list comprehension
List comprehension effectively maintains the original order of elements within a list.
# Initialize a list with some duplicate elements main_list = [11, 21, 19, 21, 11, 46, 19] print("List before removing duplicate elements: ", main_list) # Output: List before removing duplicate elements: [11, 21, 19, 21, 11, 46, 19] unique_list = [] [unique_list.append(x) for x in main_list if x not in unique_list] print("List after removing duplicate elements: ", unique_list) # Output: List before removing duplicate elements: [11, 21, 19, 46]
This approach is one-liner but less readable and not efficient for larger lists.
