By open-sourcing Trill, Microsoft says it wants to offer the power of the IStreamable abstraction to all customers the same way that IEnumerable and IObservable are available
Microsoft has made its internal project, known as Trill, which is capable of processing “a trillion events per day” open source.
Trill can be used as a streaming engine, a lightweight in-memory relational engine, and as a progressive query processor.
It was started as a Microsoft Research project back in 2012. Since then, it has been extensively described in research papers such as VLDB and the IEEE Data Engineering Bulletin.
Till date, Microsoft has used Trill internally for Azure Data products and other mission-critical streaming initiatives throughout the company including Bing Ads and Halo.
Microsoft claims,
Trill was the first streaming engine to incorporate techniques and algorithms that process events in small batches of data based on the latency tolerated by the user. It was also the first engine to organize those batches in columnar format, enabling queries to execute much more efficiently than before. To users, working with Trill is the same as working with any .NET library, so there is no need to leave the .NET environment. Users can embed Trill within a variety of distributed processing infrastructures such as Orleans and a streaming version of Microsoft’s SCOPE data processing infrastructure.
Trill works equally well over real-time and offline datasets, as a result it is usually preferred by users who just want one tool for all their analyses.
Trill is also popular among open source developers for these reasons:
- As a single-node engine library, any .NET application, service, or platform can easily use Trill and start processing queries.
- A temporal query language allows users to express complex queries over real-time and/or offline data sets.
- Trill’s high performance across its intended usage scenarios means users get results with incredible speed and low latency.
Benefit of open sourcing Trill
By open-sourcing Trill, Microsoft says it wants to offer the power of the IStreamable abstraction to all customers the same way that IEnumerable and IObservable are available.
“We hope that Trill and IStreamable will provide a strong foundation for streaming or temporal processing for current and future open-source offerings,” writes James Terwilliger, Principal Software Engineer, Microsoft Azure, in a blog.
To try Trill, Microsoft is inviting people to reach out via email to the company using asktrill@microsoft.com.
