Dev diary - Rewriting Autocomplete Day 8

Datetime:2016-08-23 02:11:47          Topic: Elastic Search  Lucene  Node.js           Share

Day 8

Today is decision day.

We chose ElasticSearch over Lucene. It's a great piece of kit, gives us nearly all the power of Lucene without any of the headaches. Lucene.net is missing suggesters in the current version and it doesn't look like anyone's worked on them for months (maybe we could submit a pull request.) Java/Python would require learning another language, one that we are not particularly interested in.

This forced us down a particular path. Using our existing ElasticSearch cluster as we don't want the expense of another one. Which mean it needs to be in our data centre in order to get the performance we want. However we have quite a long lead time on new servers in our existing data centre, so this means we don't really have time to make a brave choice with a higher risk of problems. Mistakes are expensive so we can't really afford to make them in the short time-scales we want to deliver autocomplete in. So we have to stick with what we know, and what we know will work.

We did look at nodejs, but we know we can't do everything we want to do in just ElasticSearch, and that means data processing in the app as well. We know from experience that nodejs can have problem with this due to it's single threaded nature. We also looked at Mono but we have no experience of that in a production environment.

So that leaves us with .Net running on Windows. This is a traditional stack for us, we had hoped to push into some new technologies, but you can't really experiment if getting it wrong is too expensive. For me personally I hate having to make this decision. Dev Leads are always pushing the team to try new things, and it's sad when the environment prevents this. Hopefully we can change this in the future.

Time to get those servers requested, and get our walking skeleton ready.

Andy Lowry (Dev Lead - Search Team @azd_lowry )

Header image: By Vhanepfu (Own work) [CC BY-SA 4.0 ( http://creativecommons.org/licenses/by-sa/4.0 )], via Wikimedia Commons





About List