ruby on rails - Sorting UTF-8 strings in RoR

Question

I am trying to figure out a 'proper' way of sorting UTF-8 strings in Ruby on Rails.

In my application, I have a select box that is populated with countries. As my application is localized, each existing locale has a countries.yml file that relates a country's id to the localized name for that country. I can't sort the strings manually in the yml file because I need the ID to be consistent across all locales.

What I have done is create a ascii_name method which uses the unidecode gem to convert accented and non-latin characters to their ascii equivalent (for instance, "Afeganist?o" would become "Afeganistao"), and then sort on that:

require 'unidecode'

class Country
  def ascii_name
    Unidecoder.decode(name).gsub("[?]", "").gsub(/`/, "'").strip
  end
end

Country.all.sort_by(:&ascii_name)

However, there are obvious issues with this:

• It cannot properly sort non-latin locales, as there may not be a direct analogous latin character.
• It makes no distinction between a letter and all accented forms of that letter (so, for instance, A and ? become interchangeable)

Does anyone know of a better way that I could sort my strings?

Answer 1

Ruby peforms string comparisons based on byte values of characters:

%w[à a e].sort
# => ["a", "e", "à"]

To properly collate strings according to locale, the ffi-icu gem could be used:

require "ffi-icu"

ICU::Collation.collate("it_IT", %w[à a e])
# => ["a", "à", "e"]

ICU::Collation.collate("de", %w[a s x ?])
# => ["a", "s", "?", "x"]

As an alternative:

collator = ICU::Collation::Collator.new("it_IT")
%w[à a e].sort { |a, b| collator.compare(a, b) }
# => %w[a à e]

Update To test how strings should collate according to locale rules the ICU project provides this nice tool.