Attunity Webcast on Real Time Optimisation For Microsoft BI

by Allan Mitchell 11 May 2011 19:15
On Wednesday 18th May 1Pm –2PM Eastern Time I am going to be helping Attunity deliver a webcast.  In it I want to highlight some of the challenges we face in getting data around our environments.  I want to highlight the Microsoft products that can help us realise Business Intelligence. It promises to be a good hour.  Hope to see you all there.  Here is the link Attunity Webcast

When does a Tumbling Window Start in StreamInsight

by Allan Mitchell 17 Mar 2011 10:07
Whilst getting some courseware ready I was playing around writing some code and I decided to very simply show when a window starts and ends based on you asking for a TumblingWindow of n time units in StreamInsight.  I thought this was going to be a two second thing but what I found was something I haven’t yet found documented anywhere until now.   All this code is written in C# and will slot straight into my favourite quick-win dev tool LinqPad   Let’s first create a sample dataset   var EnumerableCollection = new [] { new {id = 1, StartTime = DateTime.Par... [More]

Mouse Clicks, Reactive Extensions and StreamInsight Mashup

by Allan Mitchell 9 Feb 2011 00:24
I had an hour spare this afternoon so I wanted to have another play with Reactive Extensions in .Net and StreamInsight.  I also didn’t want to simply use a console window as a way of gathering events so I decided to use a windows form instead. The task I set myself was this. Whenever I click on my form I want to subscribe to the event and output its location to the console window and also the timestamp of the event.  In addition to this I want to know for every mouse click I do, how many mouse clicks have happened in the last 5 seconds. The second point here is really i... [More]

StreamInsight and Reactive Framework Challenge

by Allan Mitchell 6 Feb 2011 23:09
In his blogpost Roman from the StreamInsight team asked if we could create a Reactive Framework version of what he had done in the post using StreamInsight.  For those who don’t know, the Reactive Framework or Rx to its friends is a library for composing asynchronous and event-based programs using observable collections in the .Net framework.  Yes, there is some overlap between StreamInsight and the Reactive Extensions but StreamInsight has more flexibility and power in its temporal algebra (Windowing, Alteration of event headers) Well here are two alternate ways of doing what Roman... [More]

StreamInsight Precon at SQLBits 8 in Brighton

by Allan Mitchell 23 Jan 2011 21:24
I am giving a full day seminar on StreamInsight at the SQLBits conference in Brighton, UK.  The seminar is happening on 7th April 2011.  Early bird discounts are available so get over to the site and register.  During this day I will be explaining exactly what StreamInsight is and we’ll be looking at how to solve some very interesting business problems with streaming data.  A complete rundown of what it is I will be covering is here.

SSIS and StreamInsight Working Together.

by Allan Mitchell 18 Jan 2011 21:59
I have been thinking a lot recently about what it would be like to have StreamInsight and SSIS working together.  Well the CAT team have produced a paper on some of our options here. Here are some of my thoughts. There is of course a slight mismatch in their types of usage.  StreamInsight is an Event Stream processing engine capable of operating on new data in the sub second timeframe.  The engine allows you to do real time analytics and take decisions on events that have potentially only just happened.  SSIS on the other hand is a batch processing engine.  In... [More]

Showing Edge Shaped Event Duration in StreamInsight using Debugger

by Allan Mitchell 29 Dec 2010 15:54
Whilst writing some courseware I wanted to be able to see the start and end times of Edge shaped events from within the debugger.  A quick recap on Edge events At the start of the event you do not know the end time and most probably cannot work it out or you should be using one of the other shapes. You enqueue an event (Start Edge) with the start time and payload of the event.  The end time of the event is set to infinity When you see the end edge come through, you enqueue another event (End Edge) with the previous start time and payload and restate the event’s end time.&... [More]

Hopping/Tumbling Windows Could Introduce Latency.

by Allan Mitchell 5 Dec 2010 15:15
This is a pre-article to one I am going to be writing on adjusting an event’s time and duration to satisfy business process requirements but it is one that I think is really useful when understanding the way that Hopping/Tumbling windows work within StreamInsight.  A Tumbling window is just a special shortcut version of  a Hopping window where the width of the window is equal to the size of the hop Here is the simplest and often used definition for a Hopping Window.  You can find them all here public static CepWindowStream<CepWindow<TPayload>> HoppingWindow<TPa... [More]

Linqpad and Autocompletion

by Allan Mitchell 27 Nov 2010 18:22
I have mentioned before about doing development for StreamInsight in Linqpad.  I have it installed on two separate PCs and I have enabled autocompletion on only one of them.  Whilst both versions are an excellent tool, the one with autocompletion enabled is so much easier to use.  After enabling autocompletion you can see I now get parameter listing

Linqpad and StreamInsight

by Allan Mitchell 25 Nov 2010 00:00
Slightly before the announcement of StreamInsight being available for Linqpad I downloaded it from here.  I had seen Roman Schindlauer demonstrate it at Teched and it looked a really good tool to do some StreamInsight dev.   You will need .Net 4.0 and StreamInsight installed. Here’s what you need to do after downloading and installing Linqpad. Add a new connection   The next thing we need to do is install and enable the StreamInsight driver.  Choose to view more drivers.   Choose StreamInsight     Select the driver after install   ... [More]

My Favourite Two Buttons in Denali CTP1 SSIS

by Allan Mitchell 13 Nov 2010 23:03
In SSIS for SQL Server 2005 and SQL Server 2008 when you delete something from the design surface it is gone.  The only real way of getting the deleted item(s) back is to revert to a previous version of the package or to redo the deleted items manually.  Neither of these options is particularly great.  I have made this mistake before and cursed not having CTL+Z and CTL+Y.  Denali changes this.  We can now undo and redo.  Very very welcome.  Well done, finally, the SSIS team.

Import SSIS Project in Denali CTP1

by Allan Mitchell 11 Nov 2010 22:31
For years Analysis Services has had the ability to take an existing database from a server and reverse engineer it into a BIDS project.  This is extremely useful when all you have is the running instance of the database and the project that created it has long since disappeared.  Reverse engineering has never been a feature of SSIS until now. Let me walk you through the simple steps. The first step is that you obviously have to have a project deployed to an SSIS Catalog.  I will do a video on this soon but in case you can’t wait then my good buddy Jamie Thomson has written it ... [More]

Issuing Current Time Increments in StreamInsight (A Practical Example)

by Allan Mitchell 29 Aug 2010 21:02
The issuing of a Current Time Increment, Cti, in StreamInsight is very definitely one of the most important concepts to learn if you want your Streams to be responsive. A full discussion of how to issue Ctis is beyond the scope of this article but a very good explanation in addition to Books Online can be found in these three articles by a member of the StreamInsight team at Microsoft, Ciprian Gerea. Time in StreamInsight Series http://blogs.msdn.com/b/streaminsight/archive/2010/07/23/time-in-streaminsight-i.aspx http://blogs.msdn.com/b/streaminsight/archive/2010/07/30/time-in-streaminsight... [More]

SQL Profiler Through StreamInsight Sample Solution

by Allan Mitchell 23 Aug 2010 23:35
In this postI show how you can use StreamInsight to take events coming from SQL Server Profiler in real-time and do some analytics whilst the data is in flight.  Here is the solution for that post.  The download contains Project that reads events from a previously recorded trace file Project that starts a trace and captures events in real-time from a custom trace definition file (Included) It is a very simple solution and could be extended.  Whilst this example traces against SQL Server it would be trivial to change this so it profiles events in Analysis Services. &#... [More]

Additional Columns in StreamInsight Event Flow Debugger

by Allan Mitchell 15 Aug 2010 10:31
This tool is excellent when investigating what is going on in your StreamInsight Streams.  I was looking through the menu items recently and went to Query => Event Fields I found that there were a couple of columns not added by default to the event viewer (Reminds me of the fact that the Variables viewer in SSIS hides columns also) Latency NewEndTime EnqueueTime Here they all are together.     This gives us even more information about what is going on

Streaming Profiler Data Through StreamInsight

by Allan Mitchell 4 Aug 2010 23:40
Using the Trace Management objects API it is possible to capture the events happening on SQL Server or indeed Analysis Services.  I got to thinking that this would be a good thing to stream out in real time in StreamInsight and do some analysis “In Flight”.  So here is a short video of me doing just that.   Your browser cannot play this video. Learn how to fix this.

AdvanceTimePolicy and Point Event Streams In StreamInsight.

by Allan Mitchell 2 Aug 2010 23:30
There are a number of ways to issues CTIs (Current Time Increments) into your StreamInsight streams but a quite useful way is to do it declaratively on your source factory like this public AdapterAdvanceTimeSettings DeclareAdvanceTimeProperties<TPayload>(InputConfig configInfo, EventShape eventShape) {     return new AdapterAdvanceTimeSettings(         new AdvanceTimeGenerationSettings(configInfo.CtiFrequency, TimeSpan.FromTicks(-1)),         AdvanceTimePolicy.Adjust); } T... [More]

Performing a Left Anti Semi Join in StreamInsight

by Allan Mitchell 18 Jul 2010 09:54
Performing a LEFT OUTER JOIN in TSQL is easy.  The same logic in StreamInsight requires a little more work.  In this presentation I show exactly how you can perform a LEFT OUTER JOIN on your event streams. Your browser cannot play this video. Learn how to fix this.

Talking about StreamInsight in a pub in Maidstone, Kent

by Allan Mitchell 1 Jul 2010 23:41
Come join me for what promises to be a good evening.  18.8.2010.  Sign Up

Talking StreamInsight At Usergroup in London

by Allan Mitchell 1 Jul 2010 23:36
Come join us at the UK SQL Server UG on 14.7.2010 in London

RecentComments

Comment RSS
SQLIS | SSIS Tips & Tricks (Presentation)

SSIS Tips & Tricks (Presentation)

by Darren Green 21 May 2009 09:01

This has been a rather well used presentation title but it does allow a certain degree of flexibility, and we covered a good range of topics in my session at the UK SQL Server User Group in Cambridge last night. Thanks to all who attended.

Here is the rather limited slide deck and the all important demo packages for download as promised.

For reference, high level topics covered were

  • BIDS Helper
  • Inserts and Updates
  • Transactions
  • Script Debugging
  • Data Flow Checkpoints

I’ll update the post with a link to the Live Meeting recording when I get it.

Presentation & Demo Packages (194KB) SSIS Tips & Tricks - Darren Green.zip

Pingbacks and trackbacks (1)+

Add comment

  Country flag

biuquote
  • Comment
  • Preview
Loading

RecentComments

Comment RSS