ValueError: cannot reindex from a duplicate axis error typically occurs when you try to reindex a DataFrame or Series but the operation results in duplicate index labels.
To fix this error, ensure the new index does not contain duplicate values. Before reindexing, check for and remove any duplicate index entries using DataFrame.drop_duplicates() or DataFrame.reset_index().drop_duplicates() to remove duplicates.
Common causes for this error
- Attempting to reindex a DataFrame or Series where the new index contains duplicates.
- Joining or merging DataFrames on an index or column that contains duplicate values can result in an attempt to create a DataFrame with a duplicate axis.
- Assigning a new index to a DataFrame or Series that contains duplicate values.
- When concatenating DataFrames along an axis, it results in duplicate index entries in the resulting DataFrame.
Flowchart
Reproduce the error
import pandas as pd
# Example DataFrame with duplicate indices
df = pd.DataFrame({'data': [1, 2, 3]}, index=['a', 'a', 'b'])
# Now, reindexing will work
df.reindex(['a', 'b', 'c'])
Output
ValueError: cannot reindex from a duplicate axis
How to fix the error
import pandas as pd
# Example DataFrame with duplicate indices
df = pd.DataFrame({'data': [1, 2, 3]}, index=['a', 'a', 'b'])
# Remove duplicates and reset index
df = df[~df.index.duplicated()].reset_index(drop=True)
# Now, reindexing will work
df.reindex(['a', 'b', 'c'])
print(df)
Output
data
0 1
1 3
Alternatively, overwrite your current DataFrame index with a new one:
df.index = new_index
Or use .reset_index:
df.reset_index(level=0, inplace=True)
Remove inplace=True if you want it to return the dataframe.
Alternate ways to fix this error
- Ensure a unique index when reindexing: Ensure that the new index you’re using for reindexing doesn’t contain duplicates.
- Handle duplicates in joins/merges: If you are joining or merging DataFrames, ensure the keys you are joining do not have duplicates, or handle them appropriately before the operation.
- Use concatenation carefully: When concatenating, be aware of the resulting index and consider resetting the index if necessary.