You've misunderstood a few things. TreeSet does eliminate duplicates, with 'a duplicate' defined as 'any two elements for which your compare method returns 0'. No 2 such elements can both exist in a treeset. I'm sure your code doesn't work if you say so, but the code you pasted isn't the problem, nor is TreeSet's code.
A trivial example:
Comparator<String> byLength = (a, b) -> a.length() - b.length();
Set<String> set = new TreeSet<String>(byLength);
set.add("Hello");
set.add("World");
set.add("X");
set.add("VeryLong");
System.out.println(set);
> [X, Hello, VeryLong]
Note how 'World' disappeared, because the comparator says it is equal to Hello (they are both 5 length, a.length() - b.length() is returning 0, and 0 is 'equal, thus, eliminate the duplicate' according to treeset). In other words, your code as pasted would eliminate duplicates, the problem lies elsewhere.
与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…