What JD.com Reveals About Ecommerce Search Relevance

When I first came across JD.com’s example of a shopper searching for “cellphone for grandpa,” the problem looked almost too simple. I understand what that shopper wants, and you probably do too. But the products capable of satisfying that need may sit in the catalogue under language closer to “senior mobile phone.” The shopper has expressed the need clearly enough for another person to understand it, and the system can still fail to connect that expression with the products it should retrieve.

That distinction changed the way I look at ecommerce search relevance. Relevance is easy to associate with ranking: which product deserves to appear first, second or third? JD.com’s case pushed me one stage earlier. Before a system can rank the right product, it has to retrieve that product at all.

Which raises a different question. How does a search system decide that two expressions built from different words represent the same underlying need?

JD.com had already answered part of that question through specialist-created synonym rules. What interested me was what happened when it added a second interpretation, derived from what shoppers actually searched for and selected. The comparison is not really human versus machine. I read it as language relationships anticipated before shoppers search, set against language relationships observed after they act.

Ecommerce search relevance can fail before ranking begins

JD.com describes what it calls a semantic-matching problem: relevant products exist, but their titles do not contain the same terms as the shopper’s query. The problem becomes easier to see once retrieval is separated from ranking.

A traditional inverted index uses the terms in a query to reduce billions of possible products to a much smaller candidate set. Ranking systems then decide which of those candidates deserve the strongest positions. But if the language in the query and the language in the product title do not overlap enough, the relevant product may never reach the ranking stage at all.

JD.com identifies natural-language searches, long-tail queries and ambiguous expressions among the situations where this happens, and its researchers describe candidate retrieval as a major source of failure in their production environment. The broader literature on ecommerce product search describes the same structural problem: shopper vocabulary does not always match the vocabulary a retailer uses in its catalogue, which creates a need for mechanisms that map one form of expression onto another. Ecommerce search synonyms, query understanding and relationships derived from interaction data are all attempts at that translation.

That is why I would not treat ecommerce search relevance as a ranking problem alone. Sometimes the system never gets the chance to rank the right answer.

JD.com’s specialists had already built a translation layer

JD.com was not starting from an unsophisticated keyword system. Its production environment already used query rewriting, including a human-curated phrase-level synonym dictionary. Specialists could define relationships between different expressions so that a shopper’s wording was converted into language more useful for retrieval.

That matters, because this kind of story is easy to tell badly. The convenient version would be that old-fashioned human rules failed, artificial intelligence arrived, and the machine discovered a better way. I cannot read the evidence that way.

The curated system had an important advantage: control. A specialist could decide that two phrases were close enough to justify a rewrite, which makes the language relationship deliberate rather than merely inferred. JD.com’s offline comparisons show how conservative those rules were — the rule-based approach stayed closer to the original query, lexically and semantically, than the learned alternatives. That conservatism has value. A rewrite that stays close to the original expression carries less risk of changing what the shopper meant.

The limitation is structural rather than intellectual. A curated relationship has to be anticipated before it can help the next search. JD.com describes the effort of building and maintaining these rules as costly and time-consuming, and its researchers argue that manually compiled relationships cannot cover every subtle or long-tail case. The problem was not that specialists did not understand language. It was that they had to predict which language relationships were worth encoding before millions of shoppers produced them.

Shopper behaviour revealed another set of relationships

JD.com approached the same problem from another direction. Instead of relying only on relationships defined in advance, the researchers used historical relationships among queries, clicked products and product titles.

The scale suggests what behavioural observation changes. JD.com reports training on 60 days of click logs, and describes roughly 300 million query–product pairs and 5.6 billion sessions. Interactions supported by more than one click were retained in an attempt to reduce accidental signals.

The paper builds a cycle-consistent translation model around that data, but the architecture is not the most important part of the case. The important shift is interpretive. The specialist-created dictionary effectively asks which language relationships should be defined for shoppers in advance. The behavioural approach asks which relationships become visible when queries are observed alongside the products shoppers select.

That is how an expression such as “cellphone for grandpa” can be connected with language closer to “senior phone.” The paper gives another example in which wording equivalent to “milk powder for seniors” is rewritten towards “adult milk powder.” Those relationships are useful precisely because they do not depend on the two phrases sharing the same words.

I would stop short of saying that behaviour tells JD.com what shoppers truly meant. A shopper can only click what the system exposes — a constraint I return to below. What JD.com gained here was not an objective dictionary of shopper intent. It gained a second source of evidence.

JD.com added a retrieval pathway rather than replacing the old one

The structure of the live experiment is what convinced me that supplementation is the more useful way to read this case.

JD.com did not remove its production system and substitute the learned rewriting model. The control retained the company’s established search system, including its existing query-rewriting methods, and that same baseline remained in place in the treatment. JD.com then added up to three model-generated rewritten queries, each able to retrieve as many as 1,000 additional product candidates. Those candidates were merged with the original candidates, and both conditions continued through the same downstream ranking system and business logic.

That detail changes what can reasonably be concluded from the experiment. This was not a clean test of specialist-written ecommerce search synonyms against synonyms inferred from behaviour. The treatment added a second retrieval pathway. Language interpretation was part of it, but so was candidate expansion: a rewritten query could expose products that the original production route had never placed in front of the ranking system.

So I cannot isolate the commercial effect of the terminology itself. What JD.com tested was the incremental value of the complete rewriting-and-retrieval treatment. That may sound like a narrow methodological distinction, but it governs everything the results can be used to claim.

The experiment changed commercial outcomes

JD.com ran the production test for ten days on 10% of site traffic and reported relative changes of:

• +0.5219% in user conversion rate
• +1.1054% in gross merchandise value
• −0.0397% in query rewrite rate

I would leave the final metric alone. The paper reports the negative movement while describing the online metrics collectively as improved, but it does not define query rewrite rate clearly enough for me to say whether the reduction was desirable or what produced it.

The first two results are clearer: the treatment was associated with higher conversion and higher gross merchandise value (GMV) during the test. This is also where the phrase search relevance requires discipline. JD.com did not report that live relevance increased by 0.5219%. It reported that conversion increased by that relative amount, and those are not interchangeable claims. A more relevant candidate set can contribute to conversion, but conversion is also affected by pricing, inventory, promotions, ranking, product attractiveness, trust and conditions later in the purchase journey.

The methodological research makes the same distinction. Online experiments are valuable because they show how real users respond to a treatment, but the metric selected determines what has actually been measured. Revenue is a legitimate commercial outcome; it is not a direct relevance judgement.

My reading of the JD.com result is therefore deliberately narrow. Adding behaviour-derived rewrites, and the products retrieved through them, produced measurable commercial changes relative to the existing production system. That is already an important finding, and it does not need to be enlarged into a claim the experiment cannot carry.

Behaviour is informative, but it is not neutral

There is a tension in this case that interests me more than the usual argument about automation replacing specialists: a behavioural system learns from an environment partly created by the existing system.

Products in stronger positions receive more exposure, and products with more exposure have more opportunities to be clicked. Popular queries generate richer histories than rare ones. A relevant product that the existing system never retrieves contributes almost no behavioural evidence at all. Research on click behaviour has documented these effects — position, exposure and the quality of surrounding results all shape whether an individual item is clicked. The useful conclusion is not that behavioural search signals are unreliable and should be ignored. It is that they record what shoppers did within the opportunities the system gave them, which makes the evidence conditional rather than neutral.

A later IKEA search evaluation case shows a related problem from another direction: even authentic behavioural evidence can shift in meaning when the population or evaluation rule changes between stages.

There is a second risk. Expanding a query can expand it in the wrong direction. Research on query expansion describes this as query drift: adding semantically related language can retrieve useful material the original wording missed, but inappropriate additions shift the meaning of the search and reduce precision.

JD.com’s own human evaluation makes the trade-off hard to ignore. Compared with the rule-based system on 1,000 queries, the jointly trained model was judged a tie in 60% of cases, a loss in 29% and a win in 11%. That is not evidence that behaviour-derived rewriting was generally more relevant than specialist-curated rules. I read it as a division of labour. The learned system generated more diverse rewrites and could uncover relationships the conservative rules did not; the curated system remained a strong relevance control. One contributed coverage, the other restraint. The most useful system may need both.

Amazon shows that the mechanism remains current

JD.com’s work dates from 2020, so I wanted to know whether this way of interpreting query language was still in use several years later. Amazon provides a useful contemporary reference.

In 2024, Amazon researchers described a production query-reformulation system built partly around the uneven behavioural evidence available for different searches. Lower-frequency queries often had sparse histories, so Amazon linked those behaviour-poor queries to semantically similar queries with richer histories, using relationships mined from query–product co-purchase behaviour.

The implementation is not JD.com’s, and I would not treat it as a replication. Amazon used the relationships mainly to enrich ranking and matching evidence rather than reproducing JD.com’s query-to-title rewriting and additional candidate retrieval. The underlying principle is recognizable, though: observed shopper search behaviour was still being used to infer relationships between different query expressions.

Amazon reported two-week production tests in Japanese, Hindi and English environments. Revenue changes were +0.14%, +0.29% and +0.08% respectively, and conversion changes were +0.08%, −0.01% and +0.06%. The results are small and not uniformly positive, which is another reason to resist turning behavioural interpretation into a universal performance claim. What Amazon offers is corroboration that the mechanism is still in industrial use — not confirmation of anything JD.com reported.

The stronger interpretation listens twice

I keep coming back to “cellphone for grandpa.” There is nothing obviously wrong with the query. There is nothing wrong with a retailer describing a product as a senior phone. The problem exists in the relationship between the two.

A specialist can recognize that relationship and encode it deliberately. At scale, shopper behaviour can expose further relationships that nobody anticipated strongly enough to encode beforehand. JD.com’s case made me see those as two different forms of interpretation: one begins with expertise, the other with observation, and neither is complete alone. Curated mappings offer control and semantic restraint. Behaviour-derived mappings offer scale, contextual evidence and the chance of surfacing catalogue language a rule set has missed — while carrying the biases of the environment that generated them, and while broader rewriting risks drifting away from the original need.

So I do not take JD.com’s results as evidence that machines understood shoppers better than specialists. I take something more useful from them. Ecommerce search relevance can depend on what a system is able to understand before ranking even starts. JD.com had already encoded one version of that understanding in specialist-created rules; its experiment added a second version derived from shopper behaviour. What was tested was not a choice between the two. It was a system able to listen twice.