It is confusing when someone says freeze or clean with ethanol and other colleagues and/or references say don’t. Ivory can be tricky and so there are often caveats given.
There has been long-standing concern over freezing items with ivory/horn/bone/teeth but more and more institutions are doing this safely, regularly and propylactically. While conservative-minded conservators still are wary of this, it can be done safely and condensation can be easily avoided. The freezing procedures on the solutions page of the museumpests.net website give mnore information. For more on see Ellen Carrlee’s excelent JAIC article
However, I don’t think that freezing is necessary in this case and concur with Barbara Appelbaum’s recommendation of a light swabbing with 70% ethanol. A gentle swabbing with ethanol is often a first step in cleaning ivory.