Keep each word once
- Your text
- the cat the dog the cat
- Remove
- Every repeat
- Treat capitals as different words
- No
Result Cleaned text: the cat dog
Words before: 6
Words after: 3
Duplicates removed: 3
Share removed: 50%
- Split each line into words by Unicode letter rules: 6 words found.
- Kept the first time each word appears and dropped later repeats, ignoring capitals.
- Removed 3 of 6 words; 3 remain.
Two extra copies of the and one of cat are dropped, so six words become three.