My Lack of Insight

SQL Server MVP Deep Dives 2, Chapter 59

Real-time BI with StreamInsight – by Allan Mitchell (Blog|Twitter)

This was a hard one. I had to remind myself a couple of times while reading this chapter that I am writing these reviews so that I can understand these topics better.

Before I read this chapter I had never heard of StreamInsight and it took me a while to wrap my head around what exactly this tool is. Finally after doing some Binging around on the topic and reading the chapter for the sixth time these lines jumped out at me:

StreamInsight is a complex event-processing engine

Once the events leave the input adapter, they enter the engine.

The engine is where all the query logic is introduced and from where the intelligence is derived.

On my first read through of the chapter I got a good feel for the concepts of what StreamInsight could be used for, but I was having a hard time visualizing what it actually did/how it worked. The examples that Allan provided were things like a stream of trades, or reading in RFID tags in a warehouse. I read those thinking, “ok..that makes sense, but aren’t those just transactions?” Even if you account for the time component (every instance, every 5 minutes, etc) I could picture just querying the data as I normally would and storing it for analysis. Allan even admits that you could probably write T-SQL to get at the same data, but StreamInsight would work better for these types of scenarios. I had to be missing something, so I read it again. And again.

Ultimately I think it was the comparison to creating an SSIS project that finally clicked for this old ETL developer. A StreamInsight project uses adapters to bring in data from different sources and then feeds it to the complex event processing (CEP) engine where all the magic happens. We are using a different tool to query the data instead of the relational database engine I know so well. StreamInsight has its own way to process events that allows it to quickly handle orders of magnitude more data and provide realtime answers on a stream of data.

OK. Now I had the concept figured out and I even knew where to download the tool if I wanted to play with it. Time to read the chapter again and see if I can follow the examples. The code he uses in the examples is C# and the query language that StreamInsight uses is LINQ. Both of which I have investigated, but never really used. Thanks to the outline Allan provided I was able to follow along while he created a  stream of Trades and a stream of Traders to analyze. I admit I got hung up on the LINQ syntax when he talked about a “left anti semi join,” but I plowed ahead. The example is pretty straight forward once I got past the LINQ syntax and I think I finally grokked how StreamInsight would be used.

In the end I decided I was too busy with other things to download StreamInsight and play with it, but if I enter into a project that would benefit from this approach I feel confident I have a good starting point. In addition to Allan’s chapter in Deep Dives I would recommend the Microsoft TechNet page for StreamInsight and a video I found by Mike Wilmot (Blog|Twitter) that helped me clarify exactly what StreamInsight project is.

Chapter Fifty-Nine SQL Server MVP:

Allan Mitchell (Blog|Twitter) is a data integration architect based in the UK. He has worked all over the world in a variety of industries on projects big and small. He specializes in StreamInsight, SSIS, and WF4. He has been a SQL Server MVP since 2004.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s