Lambda+, the renewal of the Lambda Architecture: Category Theory to the rescue


Designing software architectures for Big Data is a complex task that has to take into consideration multiple parameters, such as the expected functionalities, the properties that are untradeable, or the suitable technologies. Patterns are abstractions that guide the design of architectures to reach the requirements. One of the famous patterns is the Lambda Architecture, which proposes real-time computations with correctness and fault-tolerance guarantees. But the Lambda has also been highly criticized, principally because of its complexity and because the real-time and correctness properties are each effective in a different layer but not in the overall architecture. Furthermore, its use cases are limited, whereas Big Data need an adaptive and flexible environment to fully reveal the value of data. Despite those critics, the Lambda Architecture proposes some interesting mechanisms, that have to be actualized to get rid of the downsides of the pattern. We present a renewal of the Lambda Architecture: the Lambda+ Architecture, allowing exploratory and real-time analyzes on data. We identify several abstraction levels of description in term of styles, patterns, properties, characteristics. We propose a formalization framework to precise these descriptions more strictly using the category theory, and apply it to the Lambda+. We relate a real implementation of our approach to architecture a social network observatory platform.

08: Theory development and use Main Track