ElasticON Tour Paris: what I learned

On 14 February 2019, I went to ElasticON Tour Paris. A little nervous.

It was my first Elastic conference. I used ElasticSearch every day as a search engine, and the rest of the stack far less, so I was afraid I wouldn't have the technical background to keep up.

I wasn't going for the product announcements. I was going with three very concrete questions, the ones I kept hitting a wall on in configuration: how to optimise memory, how to optimise searches, and what the ideal configuration looks like. And with the urge, too, to talk shop on the booths.

The talks were split between presentations by the Elastic teams and field feedback — BPCE and La Poste, in particular.

The presentations, very dense, walked through what's new in the stack. The work done over the past few years is enormous, on Kibana especially. Two features caught my attention.

SQL. Announced in 6.3: you query data indexed by ElasticSearch in SQL-like syntax.

Rally. A benchmarking framework. The tool isn't new, but it lets you settle which configuration fits an information system best, by benchmarking the clusters.

The rest would each deserve an article of its own: Cross-Cluster Replication, Hot-Warm Architecture, Elastic Common Schema, Ingest resilience, Data rollups, Suggestion Extension, Scalable Cross Search, Adaptive Replica Selection, Index Management UI, rolling major version upgrade, index shrinking, frozen indices, index lifecycle management.

The CEO spent half his talk defending open source, and the argument is worth reporting.

A company that produces open source can't build its model on support alone. The ideal of support is to walk the client up to a level of technical independence where they no longer need it.

Yet some players do exactly the opposite: mediocre support, kept that way to maintain dependency. A kind of planned obsolescence of support.

Elastic's answer: some products in the stack become paid, but the whole stays open source. Because only that model breaks the wall between the one who consumes and the one who produces, by giving the user the ability to rework the product they use.

They confirmed a hunch I already had. ElasticSearch becomes relevant at large volumes. And by large volume, read terabyte.

The BPCE group's information system means 3.3 billion messages indexed per day, 100 billion documents, 180 TB of data. Nothing like the gigabytes I had dealt with.

The strength of the Elastic suite today is monitoring massive quantities of logs, and therefore time-based analysis of data.

The problem BPCE ran into was index optimisation: how do you shrink the volume while keeping the data? The answer went through the RollUp API, which aggregates data into new indices, far lighter and far easier to query.

Out of curiosity I looked over the recaps of previous editions, in Paris and elsewhere. The themes are close, but the gain in tooling maturity is obvious, on Kibana above all. That makes sense: more and more information systems monitor billions of logs and metrics daily and look for correlations between application performance and server logs. That demands more robustness and more relevance in how metrics are collected.

My own expectations were met. I now know there is no ideal configuration for everyone. You have to search, feel your way, test, fail, before you find the one that fits your system.

And I came home with a condensed set of best practices:

• One cluster per use case.
• Avoid fuzziness: very expensive at search time, keep it for auto-complete, at a push.
• Use index sorting when you want to surface a top result — it sorts documents at index time. Useful when you don't care about the number of remaining matches.
• Avoid nested fields, prefer nested documents.
• Optimise the mapping before trying to scale. More nodes and more servers don't necessarily mean better searches.

This conference let me see the most common uses of the stack, and compare configurations and technical architectures on the subject. More than an application search engine, Elastic is turning into a set of tools for technical analysis and decision-making across an information system. And in that spirit, more and more machine learning is finding its way in.