How to Remove Duplicates From List in Python

Here are ways to remove duplicates from a list in Python:

  1. Using set()
  2. Using list comprehension
  3. Using collections.OrderedDict.fromkeys()
  4. Using a for loop
  5. Using list comprehension with enumerate()
  6. Using list comprehension and Array.index()
  7. Using counter()
  8. Using pandas
  9. Using numpy.unique()
  10. Using list() with dict.fromKeys()

Method 1: Using set()

The set() does not allow duplicate values. So, if we pass the list with duplicate values to the set, it will remove the duplicate values, returns only distinct values, and then convert the set to a list using the list() method.

Remember, this method does not preserve the original order of elements.

Visual RepresentationVisual Representation of Remove Duplicates From List in Python using set()

Example

# Initialize a list with some duplicate elements
main_list = [11, 21, 19, 21, 11, 46, 19]
print("List before removing duplicate elements: ", main_list)

unique_list = list(set(main_list))
print("List after removing duplicate elements: ", unique_list)

Output

List before removing duplicate elements: [11, 21, 19, 21, 11, 46, 19]
List after removing duplicate elements: [19, 11, 21, 46]

Method 2: Using list comprehension

List comprehension is effective for maintaining the original order of elements within a list.

Example

# Initialize a list with some duplicate elements
main_list = [11, 21, 19, 21, 11, 46, 19]
print("List before removing duplicate elements: ", main_list)

unique_list = [] 
[unique_list.append(x) for x in main_list if x not in unique_list]

print("List after removing duplicate elements: ", unique_list)

Output

List before removing duplicate elements: [11, 21, 19, 21, 11, 46, 19]
List after removing duplicate elements: [11, 21, 19, 46]

Method 3: Using collections.OrderedDict.fromkeys()

You can use OrderedDict() from the collections module to remove duplicates from a list while preserving the original order of the elements.

This method is particularly useful for versions prior to 3.7, as standard dictionaries did not guarantee order preservation.

Visual Representation

Visual Representation of Using collections.OrderedDict.fromkeys()

Example

from collections import OrderedDict

# Initialize a list with some duplicate elements
main_list = [11, 21, 19, 21, 11, 46, 19]
print("List before removing duplicate elements: ", main_list)

unique_list = list(OrderedDict.fromkeys(main_list))

print("List after removing duplicate elements: ", unique_list)

Output

List before removing duplicate elements: [11, 21, 19, 21, 11, 46, 19]
List after removing duplicate elements: [11, 21, 19, 46]

Method 4: Using a for loop

We can iterate through the list and add unique elements to a new list.

Example

# Initialize a list with some duplicate elements
main_list = [11, 21, 19, 21, 11, 46, 19]
print("List before removing duplicate elements: ", main_list)

# Create an empty list to store unique elements
unique_list = []

# Iterate over each element in the main list
for item in main_list:
 # Check if the item is not already in the no_duplicates list
 if item not in unique_list:
 # If the item is not in no_duplicates, append it to the list
 unique_list.append(item)

print("List after removing duplicate elements: ", unique_list)

Output

List before removing duplicate elements: [11, 21, 19, 21, 11, 46, 19]
List after removing duplicate elements: [11, 21, 19, 46]

Method 5: Using list comprehension with enumerate()

This method allows you to check if the current item has appeared previously in the list, and if not, include it in the new list.

Example

main_list = [11, 21, 19, 21, 11, 46, 19]
print("List before removing duplicate elements: ", main_list)

unique_list = [item for index, item in enumerate(main_list) if item not in main_list[:index]]

print("List after removing duplicate elements: ", unique_list)

Output

List before removing duplicate elements: [11, 21, 19, 21, 11, 46, 19]
List after removing duplicate elements: [11, 21, 19, 46]

Method 6: Using list comprehension and Array.index()

You can iterate over the list using list comprehension and include each element only if its index corresponds to the element’s first occurrence.

Example

main_list = [11, 21, 19, 21, 11, 46, 19]
print("List before removing duplicate elements: ", main_list)

unique_list = [main_list[item] for item in range(len(main_list)) if main_list.index(main_list[item]) == item]

print("List after removing duplicate elements: ", unique_list)

Output

List before removing duplicate elements: [11, 21, 19, 21, 11, 46, 19]
List after removing duplicate elements: [11, 21, 19, 46]

Method 7: Using Counter()

The Counter() method from the collections module creates a dictionary-like object that counts the occurrences of each element in an iterable.

Visual Representation

Visual Representation of Using Counter()

Example

from collections import Counter

main_list = [11, 21, 19, 21, 11, 46, 19]
print("List before removing duplicate elements: ", main_list)

# Use Counter to count the occurrences of each element
# The keys of the Counter object will be the unique elements of the list
unique_list = list(Counter(main_list))

print("List after removing duplicate elements: ", unique_list)

Output

List before removing duplicate elements: [11, 21, 19, 21, 11, 46, 19]
List after removing duplicate elements: [11, 21, 19, 46]

Method 8: Using pandas

If you are working with large datasets, using Pandas can be more efficient.

Visual Representation

Visual Representation of Using pandas

Example

import pandas as pd

main_list = [11, 21, 19, 21, 11, 46, 19]
print("List before removing duplicate elements: ", main_list)

unique_list = pd.Series(main_list).drop_duplicates().tolist()

print("List after removing duplicate elements: ", unique_list)

Output

List before removing duplicate elements: [11, 21, 19, 21, 11, 46, 19]
List after removing duplicate elements: [11, 21, 19, 46]

Method 9: Using numpy.unique()

First, you need to convert the list to a NumPy array and use np.unique to remove duplicates.

Then, convert the array back to a list using tolist().

Visual Representation

Visual Representation of Using numpy.unique()

Example

import numpy as np

main_list = [11, 21, 19, 21, 11, 46, 19]
print("List before removing duplicate elements: ", main_list)

# Convert the list to a NumPy array
unique_array = np.unique(main_list)

# Convert the array back to a list
unique_list = unique_array.tolist()

print("List after removing duplicate elements: ", unique_list)

Output

List before removing duplicate elements: [11, 21, 19, 21, 11, 46, 19]
List after removing duplicate elements: [11, 19, 21, 46]

Method 10: Using list() with dict.fromKeys() 

Example

main_list = [11, 21, 19, 21, 11, 46, 19]
print("List before removing duplicate elements: ", main_list)

unique_list = list(dict.fromkeys(main_list))

print("List after removing duplicate elements: ", unique_list)

Output

List before removing duplicate elements: [11, 21, 19, 21, 11, 46, 19]
List after removing duplicate elements: [11, 21, 19, 46]

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.