Microservices became a design style to define system architectures, purify core business concepts, evolve solutions in parallel, make things look uniform, and implement stable and consistent interfaces across systems. At Zalando, we’ve been putting together a series of articles that explain how we are applying microservice patterns to our applications: Fashion Store, Skipper, and Mosaic. In this article, I’ll be discussing our principles for microservice development of stateful solutions such as fashion search and discovery, a promising field that we’re excited to further evolve.

We’ve focused here on the search solution that targets a consumer facing application. It addresses scenarios where consumers already have either a vague or very specific idea of what they are looking for. The service objective is to point them in the direction of the most specific and relevant fashion articles as quickly as possible.

What does the consumer journey look like?

Let’s start by emphasizing important consumer-oriented use cases addressed by the solution. We will later show how they are implemented by our microservice architecture. Use cases help us to build functional boundaries as an initial step in system design and decomposition.

Discovery provides direct access to product catalogs, using the category tree as the primary entry point. Catalog discovery is not an idempotent operation for all customers, as the results set are impacted by consumer profile, preferences, search history, and by recommendation or advertisement systems. The discovery is implicit contextual search that is triggered programmatically by mobile applications as a response to consumer interaction. For example, when a customer opens a mobile application, it shows trending products: As soon as a brand/retailer is liked, the mobile application shows popular products in this context with a higher ranking.

Search matches documents from fashion catalogs using consumer defined keywords. In other words, the consumer provides their intent which is then translated into pattern match requirements for the catalog. The pattern match implies both an exact search term match and fuzzy query adaptation to maximize relevancy.

Refinement is a process to narrow down a large set of search hits using filters and facets, also known as faceted navigation. Both concepts are built to analyze and exclude any document that does not meet consumer intent. The difference between these approaches is the mechanism used to analyse aspects of the content. Filters offer a static set of dimensions to narrow down content; usually built offline using data mining techniques. Facets are built dynamically, reflecting active search intent. Because faceted navigation reflects active context, it has to be rebuilt each time a consumer interacts with search results. This difference is not explicitly exposed to consumers through the user experience: System design implements distinct functions to handle these aspects. For example, Figure 1 shows the usage of static and dynamic aspects during the refinement process. Catalog hierarchy is statically configured per application while brand, color, price, and other facets are dynamically evaluated.

Relevance sorting provides a collection of algorithms to rank search results depending on the weight of certain content aspects: Inclusion of scores based on click behavior to identify products that have a higher attractiveness for customers. Sorting algorithms consider static aspects of the content such as brand, category, and dynamic aspects - availability, merchant rank, delivery time etc. Configurability is an important requirement for the solution, as consumer facing applications impact relevance properties of algorithms online using configuration tools. Formally, relevance is determined by consumer intent, while the overall relevance of content is based on various attributes.

Intent analysis transforms unstructured consumer intent (e.g. keywords, visual search) into semi-structured queries using feature extraction techniques. Consumer intent contains relations to well-known fashion entities such as brands, categories, colors, materials; dependencies between typed words and synonyms, slang, or spelling errors. The structured query allows us to improve relevance using attribute boosting.

Personalization is required to efficiently respond to consumer intent. There is a need to engage and create a relevant fashion experience for each individual consumer. It is all about knowing consumers, getting to know consumer preferences and emerging trends from both the consumer and the retailer. The successful implementation of personalized consumer engagement depends on the ability to access consumer behavior, deduct relevant knowledge using implicit or explicit consumer intent, and accumulate consumer profiles to leverage individual product offerings at earlier phases of consumer journey.

Non-functional requirements

Interoperability: Consumer facing applications are key stakeholders for our search solution. The major concern is compatibility and interoperability of interfaces as well as data contracts supplied by the search solution. Core interfaces are built around a common data model that follows common metadata and provides the baseline for interoperability between various microservices and its evolution.

Evolution: As fashion metadata is distributed, our dynamic environment operates with multiple content sources: Product metadata, availability information, partner content. The solution should ensure the continuity of data access to consumer facing applications associated with old versions of data contracts. Simultaneously, it should not prevent the evolution of these contracts to cover new metadata aspects. The evolution must ensure the development of search architecture that allows for improvements of the indexing algorithm and technological adaptation.

Consistency: The traditional secondary index approach is used to build a search solution. The smallest unit of atomicity is a snippet and attributes - the snippet is an opaque data structure used by frontend applications to visualize search results; attributes facilitate discovery and refinement processes. This ensures we follow domain driven design. We cannot guarantee 100% consistency of product metadata within a system-of-records or search indexes due to natural limitations of distributed systems. However, the searchable data consists of frequently changeable attributes such as price and availability. We have defined 30 minutes as the maximum propagation delay of these attributes.

Latency: Software development practices consider end-to-end latency as a user-oriented characteristic during the entire lifecycle of an application. This user-oriented metric shall be defined independently of underlying solutions or technologies and will be used for quality assessment of the delivered solution. The interactive traffic is the classical data communication scheme employed by mobile search. This pattern gives a recommendation that interaction between human and remote equipment should not exceed 1 second. It makes an implication on the latency of the search interface, thus should not exceed 100 ms due to latency challenges in microservices. The system design considers infrastructure, used protocols, and other microservices.

Scalability: The scalability is the system’s ability to handle the required amount of work and its potential to be enlarged to accommodate growing traffic and data. The technology should not be a limiting factor for business growth or limit integration of new data sources. The horizontal scalability of the storage layer is a major consideration for architecture decisions. Interface scalability is achieved by using state of the art principles of microservice design, but the storage layer requires a sharable infrastructure across multiple product catalogs.

Availability: Availability here concerns data availability and durability. The availability concerns search-and-discovery interfaces, as it defines uptime requirements on system external interfaces: The ratio of the total time a consumer facing application is capable of handling product catalog retrieval. Durability reflects the risks of losing data during maintenance or downtimes. Durability is the ultimate requirement for catalogs built from event-based data sources.

End-to-end search architecture

Let’s define the search microservice architecture in terms of logical layers. Each layer is built from multiple related microservices with similar requirements. Layers helps us define operational boundaries, leveraging issues of stateless/stateful service delivery.

The storage layer is a set of stateful components that hold mission critical data. It is built around off-the-shelf open source technologies, the core part being the TF-IDF information retrieval system. The usage of AWS value-added services helps us streamline operational processes and improve data durability and service availability. The availability of the storage layer has a high impact on customer experience and sales. Our recovery strategy targets high availability (capacity over provisioning) and automated recovery for faulty nodes.

The intake layer is a traditional layer to extract, transform, and load processes – we call it an event-driven content integration pipe. It is used to extract content from a heterogeneous source using both synchronous and asynchronous communication patterns. Usually, it reads content either from ground truth business sources hosted by the Zalando platform or content extensions driven by a consumer facing application. The content transformation stage is a series of purely functional operations that join data from multiple sources and prepare the smallest unit of atomicity required by search use cases. The content is asynchronously loaded to microservices at the storage layer, enabling article discovery.

The search layer is based on consumer-oriented microservices that fulfills the search use case. We call it “search business logic” that solely focuses on the consumer journey. The layer is a pure, stateless microservice adjusted for execution of a “single” use case.

A consumer facing application is a farm of backend for frontend (BFF) components or reverse proxies that implement an integration of consumer-oriented use cases with search infrastructure. It is used by us to decompose API evolution life cycles of applications from service evolution.

Let’s take a closer look at our microservices and their end-to-end collaboration in our system design.

Interfaces

The successful development of a large system using microservice patterns requires the definition of interfaces between functional elements. We learn that traditional REST principles do not suit each microservice involved in fashion search. We have made a choice to use different API technology according to our needs. Let’s elaborate on these below.

REST - a standard of communication between microservices using REST principles. The communication pattern follows typical client-server interaction over HTTP protocol, leveraging JSON as the primary payload.

MQi - an asynchronous communication protocol that involves message-oriented technologies. This interface indicates a publisher-subscriber communication paradigm between components. The implementation technique might vary from plain socket to high-level client library. Microservices often use the REST protocol family to implement asynchronous communication (for example, AWS SQS, AWS Kinesis, or Nakadi).

KVi - an interface for storing, retrieving, and managing associative arrays (hashmap). The interface provides access to a collection of binary objects a.k.a blobs. These blobs are stored and retrieved using a unique identifier (key).

QLi - a vendor dependent interface used by the product search and discovery solution. The interface refers to the REST API defined and implemented either by Elasticsearch or SolrCloud. It provides various primitives to retrieve information using TF-IDF indexes. The ability to use different TF-IDF implementations is a key feature in our solution. The storage layer might be seen as the system-of-records for search, thus we trade-off scalability and availability of indexes versus linguistic and other add-on features. The “hot-swap” of technologies becomes a driver to fulfill customer demand.

CSi - a clickstream event publishing interface. It delivers a summary of consumer interaction with a search-and-discovery solution. The architecture evaluates two possibilities to ship consumer events. State of the art log management solutions are used for traditional long-term, off-line analytics. Alongside this, interactive protocols immediately stream consumer behavior using communication sockets to the analytics platform.

Microservices

Our Content Crawler aggregates, joins, and interleaves content from various independent data sources. The service is also responsible for normalizing input content according to principles of the Common Data Model. We have designed the crawler as an asynchronous event streams gateway that implements the last-value-caching pattern to build a snapshot of all events seen in streams. It complicates the design of a microservice but guarantees local data availability. The replay feature facilitates data loss recovery; it makes the solution independent of data sources and guarantees complete data replay in minutes.

Content Analysis performs extraction of content aspects (features). It mines facts about fashion from ingress content and builds searchable snippets. The snippet is the smallest unit of atomicity used by the solution with normalized attributes and its values. Snippets are asynchronously posted to TF-IDF indexes using message queues. The microservice facilitates refinement, semantical analysis, and relevant use cases in the system. It builds the fashion profile and is enhanced by data sources using data mining techniques. Its actions defines a keyword processing pipeline: tokenizer splits input documents to collection of tokens, lower-case textual documents; stemming normalizes text, reduces the footprint, and performs stop word eviction; text analysis creates indexable documents, performs metadata quality analysis, and augments textual data with content aspects (data dimensions such as category, price, brand, etc).

Catalog is a collection of TF-IDF indexes. They are optimized for content retrieval and refinement using consumer intent. The storage defines a relaxed schema for snippets allowing on-the-fly relevance adjustments depending on business needs. Catalog scalability and availability is a key concern of technology. We aim for elasticity – scalable to millions of requests per minute.

Fashion metadata provides core facts about the fashion domain (e.g. hierarchical categories, filters, facets); it facilitates refinement, semantical analysis, and relevant use cases in the system. The profile is built and enhanced from data sources using data mining techniques.

Structured Search implements a discovery use case by offering a platform-wide Structured Search API to browse the product catalog. It involves snippet (snippet has an inherited structure) pattern matching to a structured pattern supplied by a consumer facing application. The component does not exclude free-text search, but requires a snippet model with a typed layout of searchable attributes. The consumer application is aware of attribute roles before issuing a search request. In contrast to traditional databases, structured search concerns document relevance or scoring. The service translates consumer intent into machine processable signals, building a query using QL interface syntax. The subsystem performs boosting of relevant attributes to facilitate content sorting and personalization.

Query Planner implements the search use case by offering a platform-wide Unstructured Search API. The microservice is responsible for translating search intent into pattern match requirements for TF-IDF indexes. We have built query planning detached from actual TF-IDF technology. It allows us to deal with more complex input types (e.g. voice, images), apply the best algorithms connected to input processing (e.g. ambiguity, subjective interpretation, language specific requirements), and define more complicated processing pipelines required for natural language and voice.

Relevance sorting is a microservice responsible for the personalization of search content. You can think about it as a gateway to Zalando platform analytics. We employ two types of relevance sorting techniques: Preprocessing and postprocessing. Preprocessing is used to rewrite or enhance structured queries with boosting parameters and additional attributes relevant for given customers. Postprocessing techniques applies re-sorting of search results returned by the catalog. This technique helps us to minimize search query diversity to the TF-IDF engine, thus scale it using excessive caching techniques.

Product Gateway facilitates consumer facing applications with entire metadata about discovered products. The gateway enriches this metadata with additional information using online services (e.g. partner stocks, related products, etc).

Consumer Analytics maps consumer search behaviour to discrete shopping intent. It enables the delivery of relevant search results, boosting conversion rates and ensuring a personalized experience. The atomic knowledge facts about consumer behavior map digital footprints, constructing a consumer genome. The consumer genome is a collection of attributes to depict consumer behavior using dimension and decay properties in a discrete manner. It empowers with insights on various facets of the consumer.

Common Data Model

The Common Data Model defines generic vocabulary terms (often called object model or meta-model) of layout content into generic structures. This object model guarantees an interoperability baseline between consumer facing applications and platform microservices, in the absence of strong content negotiation techniques. The interpretation of data semantics allows us to build solutions (e.g. discovery) that are isolated from the evolution of the individual data models. Simultaneously, it is able to aggregate and interleave content from various independent sources. In other words, interoperability, evolution, and isolation in order to achieve flexible multi-merchant use cases.

Domain-level data is often exposed as a collection of nested objects. Ultimately, this is a directed, labeled graph - nodes are objects, edges correspond to properties; the endpoints of edges are either scalar values or other nodes (either named embedded or inline anonymous objects). This traditional approach is missing serialization consistency and operational simplicity. Most importantly, serialized data requires knowledge of domain-level schema to parse and interpret data.

A simple solution is required to address compatibility and interoperability requirements driven by stakeholders (e.g. consumer facing applications). The domain-level objects are fragmented into flat structures and packaged into adjacency lists with the following principles:

• Each fragment has at least one type, they aggregate common domain properties, and are manageable using an independent System of Records (SOR)
• Each fragment has a reference (globally unique identifier)
• Each fragment has properties associated with one or more values. All values are either scalar or references to other objects. Properties with multiple values are bags (unordered lists).

This transforms domain-level data as multi-class instances - split management, representation, and provisioning concerns. This is a foundation to create machine interpretable data across the fashion domain.

The chosen model does not enforce any implementation strategy for consumer facing applications or data provisioning pipelines. However, the product search and discovery solution is built around the Common Data Model which is usable directly as JSON, but provides a small number of consistently applied principles to build interoperability into Web services. It solves the integration of JSON objects from different sources (e.g. brand or retailer data sources feeds) as the data may contain keys that conflict with other data sources.

Summary

The depicted Fashion Search solution has been materialized during the Zalando Platform Search rebuild project. At first glance, you might see too many moving parts here but there is a purpose to the design. Our company applies Radical Agility as a software development methodology. “We’re constantly pushing nonstop innovation, creativity, and hard work”. The microservice adoptions grant us with a “powerful architectural style that lets us move fast while we keep our complexity low”. We have shown that Fashion Search covers a broad range of technologies including data-mining, natural language processing, consumer analytics development of fault tolerant services, etc. This article defines the system decomposition that helped us to isolate technological challenges at the microservice level and apply efficient and innovative product delivery pipelines.

We have also discovered architectural challenges that contradict certain styles of microservice development such as REST Interfaces and the Common Data Model. We learned that traditional HTTP-based REST principles do not suit each microservice involved in Fashion Search. We have made a choice to use different API technology according to our needs but still retaining our API First design principle so that “we’re able to scale as our department and business grows in scope, evolving our systems in parallel.” Additionally, we also need to guarantee the interoperability baseline between consumer facing applications and platform microservices in the absence of strong content negotiation techniques. Thus, we have defined Common Data Model principles used across microservices within the Fashion Search solution. We defined a small number of consistently applied principles to build interoperability in microservices.

This architecture helped us deliver ambitious search requirements of serving and retrieving data at a large scale during our Black Friday event of 2016. If you have any further questions about microservice usage in search and how we use it at Zalando, you can contact me via email. I’d love to hear from you.