When does a Tumbling Window Start in StreamInsight

by Allan Mitchell 17 Mar 2011 10:07
Whilst getting some courseware ready I was playing around writing some code and I decided to very simply show when a window starts and ends based on you asking for a TumblingWindow of n time units in StreamInsight.  I thought this was going to be a two second thing but what I found was something I haven’t yet found documented anywhere until now.   All this code is written in C# and will slot straight into my favourite quick-win dev tool LinqPad   Let’s first create a sample dataset   var EnumerableCollection = new [] { new {id = 1, StartTime = DateTime.Par... [More]

Mouse Clicks, Reactive Extensions and StreamInsight Mashup

by Allan Mitchell 9 Feb 2011 00:24
I had an hour spare this afternoon so I wanted to have another play with Reactive Extensions in .Net and StreamInsight.  I also didn’t want to simply use a console window as a way of gathering events so I decided to use a windows form instead. The task I set myself was this. Whenever I click on my form I want to subscribe to the event and output its location to the console window and also the timestamp of the event.  In addition to this I want to know for every mouse click I do, how many mouse clicks have happened in the last 5 seconds. The second point here is really i... [More]

StreamInsight and Reactive Framework Challenge

by Allan Mitchell 6 Feb 2011 23:09
In his blogpost Roman from the StreamInsight team asked if we could create a Reactive Framework version of what he had done in the post using StreamInsight.  For those who don’t know, the Reactive Framework or Rx to its friends is a library for composing asynchronous and event-based programs using observable collections in the .Net framework.  Yes, there is some overlap between StreamInsight and the Reactive Extensions but StreamInsight has more flexibility and power in its temporal algebra (Windowing, Alteration of event headers) Well here are two alternate ways of doing what Roman... [More]

StreamInsight Precon at SQLBits 8 in Brighton

by Allan Mitchell 23 Jan 2011 21:24
I am giving a full day seminar on StreamInsight at the SQLBits conference in Brighton, UK.  The seminar is happening on 7th April 2011.  Early bird discounts are available so get over to the site and register.  During this day I will be explaining exactly what StreamInsight is and we’ll be looking at how to solve some very interesting business problems with streaming data.  A complete rundown of what it is I will be covering is here.

SSIS and StreamInsight Working Together.

by Allan Mitchell 18 Jan 2011 21:59
I have been thinking a lot recently about what it would be like to have StreamInsight and SSIS working together.  Well the CAT team have produced a paper on some of our options here. Here are some of my thoughts. There is of course a slight mismatch in their types of usage.  StreamInsight is an Event Stream processing engine capable of operating on new data in the sub second timeframe.  The engine allows you to do real time analytics and take decisions on events that have potentially only just happened.  SSIS on the other hand is a batch processing engine.  In... [More]

Showing Edge Shaped Event Duration in StreamInsight using Debugger

by Allan Mitchell 29 Dec 2010 15:54
Whilst writing some courseware I wanted to be able to see the start and end times of Edge shaped events from within the debugger.  A quick recap on Edge events At the start of the event you do not know the end time and most probably cannot work it out or you should be using one of the other shapes. You enqueue an event (Start Edge) with the start time and payload of the event.  The end time of the event is set to infinity When you see the end edge come through, you enqueue another event (End Edge) with the previous start time and payload and restate the event’s end time.&... [More]

Hopping/Tumbling Windows Could Introduce Latency.

by Allan Mitchell 5 Dec 2010 15:15
This is a pre-article to one I am going to be writing on adjusting an event’s time and duration to satisfy business process requirements but it is one that I think is really useful when understanding the way that Hopping/Tumbling windows work within StreamInsight.  A Tumbling window is just a special shortcut version of  a Hopping window where the width of the window is equal to the size of the hop Here is the simplest and often used definition for a Hopping Window.  You can find them all here public static CepWindowStream<CepWindow<TPayload>> HoppingWindow<TPa... [More]

Linqpad and StreamInsight

by Allan Mitchell 25 Nov 2010 00:00
Slightly before the announcement of StreamInsight being available for Linqpad I downloaded it from here.  I had seen Roman Schindlauer demonstrate it at Teched and it looked a really good tool to do some StreamInsight dev.   You will need .Net 4.0 and StreamInsight installed. Here’s what you need to do after downloading and installing Linqpad. Add a new connection   The next thing we need to do is install and enable the StreamInsight driver.  Choose to view more drivers.   Choose StreamInsight     Select the driver after install   ... [More]

Issuing Current Time Increments in StreamInsight (A Practical Example)

by Allan Mitchell 29 Aug 2010 21:02
The issuing of a Current Time Increment, Cti, in StreamInsight is very definitely one of the most important concepts to learn if you want your Streams to be responsive. A full discussion of how to issue Ctis is beyond the scope of this article but a very good explanation in addition to Books Online can be found in these three articles by a member of the StreamInsight team at Microsoft, Ciprian Gerea. Time in StreamInsight Series http://blogs.msdn.com/b/streaminsight/archive/2010/07/23/time-in-streaminsight-i.aspx http://blogs.msdn.com/b/streaminsight/archive/2010/07/30/time-in-streaminsight... [More]

SQL Profiler Through StreamInsight Sample Solution

by Allan Mitchell 23 Aug 2010 23:35
In this postI show how you can use StreamInsight to take events coming from SQL Server Profiler in real-time and do some analytics whilst the data is in flight.  Here is the solution for that post.  The download contains Project that reads events from a previously recorded trace file Project that starts a trace and captures events in real-time from a custom trace definition file (Included) It is a very simple solution and could be extended.  Whilst this example traces against SQL Server it would be trivial to change this so it profiles events in Analysis Services. &#... [More]

Additional Columns in StreamInsight Event Flow Debugger

by Allan Mitchell 15 Aug 2010 10:31
This tool is excellent when investigating what is going on in your StreamInsight Streams.  I was looking through the menu items recently and went to Query => Event Fields I found that there were a couple of columns not added by default to the event viewer (Reminds me of the fact that the Variables viewer in SSIS hides columns also) Latency NewEndTime EnqueueTime Here they all are together.     This gives us even more information about what is going on

Streaming Profiler Data Through StreamInsight

by Allan Mitchell 4 Aug 2010 23:40
Using the Trace Management objects API it is possible to capture the events happening on SQL Server or indeed Analysis Services.  I got to thinking that this would be a good thing to stream out in real time in StreamInsight and do some analysis “In Flight”.  So here is a short video of me doing just that.   Your browser cannot play this video. Learn how to fix this.

AdvanceTimePolicy and Point Event Streams In StreamInsight.

by Allan Mitchell 2 Aug 2010 23:30
There are a number of ways to issues CTIs (Current Time Increments) into your StreamInsight streams but a quite useful way is to do it declaratively on your source factory like this public AdapterAdvanceTimeSettings DeclareAdvanceTimeProperties<TPayload>(InputConfig configInfo, EventShape eventShape) {     return new AdapterAdvanceTimeSettings(         new AdvanceTimeGenerationSettings(configInfo.CtiFrequency, TimeSpan.FromTicks(-1)),         AdvanceTimePolicy.Adjust); } T... [More]

Performing a Left Anti Semi Join in StreamInsight

by Allan Mitchell 18 Jul 2010 09:54
Performing a LEFT OUTER JOIN in TSQL is easy.  The same logic in StreamInsight requires a little more work.  In this presentation I show exactly how you can perform a LEFT OUTER JOIN on your event streams. Your browser cannot play this video. Learn how to fix this.

Talking about StreamInsight in a pub in Maidstone, Kent

by Allan Mitchell 1 Jul 2010 23:41
Come join me for what promises to be a good evening.  18.8.2010.  Sign Up

Talking StreamInsight At Usergroup in London

by Allan Mitchell 1 Jul 2010 23:36
Come join us at the UK SQL Server UG on 14.7.2010 in London

StreamInsight Licensing

by Allan Mitchell 28 May 2010 06:22
I cannot help but think that this is currently wrong.  There are two editions of StreamInsight (SI), Premium and Standard.  A link to their differences is here Points to note Developer Edition and Evaluation Edition represent Enterprise Edition in SQL Server but not SI where they represent Datacentre Edition.  Won’t this be confusing when people evaluating SI find that it is not the same animal when they move to Enterprise Edition? If SI Premium is the only thing you want out of the SQL Server box then it is going to be a high cost. The latency rates as well quot... [More]