StreamInsight Nuggets

March 22 2012

Here’s some choice quotes from a whitepaper on StreamInsight:

Relational database applications typically acquire data and store it to disk before it can be analyzed. We therefore call analysis with traditional relational database systems query-driven. Query-driven analysis is well-suited for historical data. … To reach the necessary performance and scale, [some] applications need to analyze the data in near real time while it is being acquired from the source. We denote these applications as event-driven applications because new event data arriving at the system triggers the necessary analysis.

I think this is a great intro to help wrap yr head around the difference between query-driven and event-driven analysis.

Microsoft StreamInsight is Microsoft’s platform to build high-throughput, low-latency event-driven analytics applications.

That’s a great one liner.

With StreamInsight, business insight is delivered at the speed at which data is produced, as opposed to the speed at which traditional reports are processed or consumed.

Nicely worded value prop.

StreamInsight’s runtime performs calculations incrementally whenever possible. This means that the processing only involves the data for the current result and the new event. Unlike in traditional databases, updating a report with aggregates or KPIs with StreamInsight does not require to re-iterate through past data once a new event comes in. Instead, StreamInsight answers continuous queries with a single pass over all the data, which is an important capability for long-running, potentially infinite, standing queries. Incremental processing is one key performance benefit of StreamInsight.

That is rad engineering methinks.

StreamInsight automatically distributes the processing across the available processor cores on the system as well. Thread management and query parallelization are performed automatically by the system.

How cool is that!

Speaking At GSummit

March 19 2012

Gonna be speaking at GSummit, a conference about gamefication, in June, down in San Francisco.  More here.

Visual Studio Achievements Buzz

January 26 2012

The project I’ve been working on recently finally shipped: Visual Studio Achievements.

It’s been great to see the reaction to it, including posts in Wired, Ars Technica, Life Hacker, BoingBoing, Gamasutra and more. It also sparked quite a discussion in Reddit and Slashdot. Here’s a list of all the buzz that the project has generated:

Microsoft Crossbreeds Programming Kit with Fantasy Game

Caleb Garling/Wired

January 24, 2012

Coders, Motivate Yourself with Achievements

Logan Booker/Lifehacker Australia

January 21, 2012

Microsoft Keeps It Old-School with a Pricey Text Adventure Game, Visual Studio 2010

Peter Bright/Ars Technica

January 20, 2012

Microsoft Text Adventure Game!

Rob Beschizza/BoingBoing

January 20, 2012

Microsoft Turns Coding Into a game with New Visual Studio Plug-In

Matt Williams/

January 20, 2012

Microsoft to Make Programming Fun

David Stellmack/Fudzilla

January 20, 2012

Microsoft Corporation Adding Visual Studio

Rachael Brunelli/eMoneyDaily

January 20, 2012

Microsoft Gamifies Visual Studio with Achievements

Mike Rose/Gamasutra

January 19, 2012

Microsoft Now Has Achievements for… Developers

Luke Plunkett/Kotaku

January 19, 2012

Visual Studio Gets Achievements

Chris Duckett/TechRepublic

January 18, 2012

Microsoft Adds Achievements to Visual Studio Software

Tom Bramwell/

January 19, 2012

Microsoft Adds Developer Achievements to Visual Studio

Nathan Brown/Edge

January 19, 2012

Visual Studio Achievements Program

Sue Gee/I Programmer

January 19, 2012

Microsoft Brings Achievements to Visual Studio

Earnest “Nex” Cavalli/ Escapist Magazine

January 19, 2012

Achievement Unlocked: Microsoft Gamifies Development

Craig Chapple/Develop

January 19, 2012

Microsoft Adds Visual Studio Achievements for Developers

Laurentiu Stan/Social Barrel

January 19, 2012

Microsoft Announces Visual Studio Achievements Beta, A Pat on the Back for Dev


January 19, 2012

Visual Studio Achievements Program Brings Gamification to Development

Staff Writer/The Financial

January 19, 2012

Visual Studio Adds Game Mechanic to Keep Devs Engaged

Jason Cartwright/TechAU

January 19, 2012

Microsoft Introduces Xbox-like Achievements for Developers

Tom Warren/The Verge

January 18, 2012

The Coding Game: Microsoft’s Visual Studio Gets Badges, Achievements and Leaderboard

Todd Bishop/GeekWire

January 18, 2012

Channel9’s Visual Studio Achievements Now Available

Long Zheng/istartedsomething

January 18, 2012

Microsoft Visual Studio Brings Gamification to App Development

Chris Burns/Slashgear

January 18, 2012

Microsoft Announces Visual Studio Achievements Beta

Pradeep Viswav/WMPoweruser

January 18, 2012

Microsoft Adds Achievements for Developers

Justin Rubio/IGN

January 18, 2012

Achievement Unlocked… For Developers

Paul Thurrott/Supersite for Windows

January 18, 2012

Microsoft Announces ‘Visual Studio Achievements’ for Developers

Simon LR/Techie Buzz

January 18, 2012

Visual Studio Achievements, Now a Reality!

Rudi/While True Blog

January 18, 2012

Visual Studio Achievements Program Brings Gamification to Development

Staff Writer/Adafruit Industries Blog

January 18, 2012

Visual Studio Achievements – The Beta Goes Live!

Alvin Ashcraft/Alvin Ashcraft’s Morning Dew

January 18, 2012

Visual Studio Achievements – Remember Kids They’re Just for Fun

Bill Simser/Fear and Loathing

January 18, 2012

Don’t Brag About Your Visual Studio Achievements! (Yet?)

Maarten Balliauw/Maarten Balliauw {blog}

January 18, 2012!-%28yet%29.aspx

Querying The Archivist API With JSON.NET

December 16 2011

Someone recently asked me for a sample of how to query The Archivist API to programmatically get a list of the top users of a given search term.

So I put a quick sample together, using the most excellent JSON.NET library’s LINQ provider for querying JSON. 

I used an archive on Wittgenstien.  To get the JSON that generated this chart, I simply append ?format=json to the URL, like this:  which results in the following response:



Here’s the program I wrote to parse this:

string json = string.Empty; using (WebClient webClient = new WebClient()) { json = webClient.DownloadString(""); } JArray result = JArray.Parse(json); foreach (JObject user in result) { Console.WriteLine((string)user["Name"] + " - " + (float)user["Count"]); } Console.ReadLine();

You can see how I simply pass the JSON to the static .Parse method hanging off the JArray class. If the root of the JSON wasn’t an array, I would have used the JObject.Parse() method. Once it is in the array, I can loop it and extract the values. You can also write LINQ queries using JSON.NET. For example, if I only wanted users whose tweet count was greater than 100, I could write this:

var greaterThan100 = from r in result where (float)r["Count"] > 100 select r; foreach (JObject user in greaterThan100) { Console.WriteLine((string)user["Name"] + " - " + (float)user["Count"]); }

Creating A Weighted Average User Defined Aggregate in StreamInsight 1.1

November 16 2011

I’ve been doing some prototyping StreamInsight lately and hit the following issue. I needed to write a weighted average so that I could roll up result sets.  Let me elaborate on the scenario.  I had a query whose HoppingWindow has a WindowSize of 1 minute.  One of the things the query returns is an average of one of the values returned by the event payload:

AvgLoadTime = win.Avg(e => e.LoadTime)

I then want to take all the results of that query and roll them up into a larger window of 1 hour. So that means writing a new query that uses the results of the first query for its calculation. But, if I were to simply take the average of the average, I’d get inaccurate results. What I need was the weighted mean.

To do so, I changed the code to look like this:

AvgLoadTime = win.Sum(e => e.AvgLoadTime * e.RecordCount)/win.Sum(e => e.RecordCount)

But when I deployed to StreamInsight 1.1 and got no love:

Microsoft.ComplexEventProcessing.Linq.QueryGenerationException: Expression '(e.Sum(e => (e.AvgLoadTime * Convert(e.RecordCount))) / Convert(e.Sum(e => e.RecordCount)))' contains more than one aggregate method call, which is not supported.

Doh. But then I got hip to user-defined aggregates. I found the documentation on this MSDN page essential and it helped me to write my extension, which looks like this:

public class WeightedAverage : CepAggregate<WqAggregate, double> { public override double GenerateOutput(IEnumerable<WqAggregate> win) { var AvgLoadTime = win.Sum(e => e.AvgLoadTime*e.RecordCount)/win.Sum(e => e.RecordCount); return AvgLoadTime; } } public static class ExtensionMethods { [CepUserDefinedAggregate(typeof(WeightedAverage))] public static double WeightedAvg(this CepWindow<WqAggregate> window) { throw CepUtility.DoNotCall(); } }

Where WqAggregate is just a class with the various fields that are returned as a result of my query.

So, I can now write this code:

AvgLoadTime = win.WeightedAvg()

Nice! I just had to deploy the .dll where I wrote the extension to the StreamInsight server and I was good to go. Now, with StreamInsight 1.2, the need to do this all goes away with but for now, this turns about to be a reasonable fix.

Dynamic Keyword and Dotfuscator = Very Unhappy

October 3 2011

I’ve been using Dotfuscator to obfuscate an assembly.  I thought all was good, but when I tried to use the obfuscate assembly, the assembly didn’t work like the unobfuscated one did and was throwing exceptions all over the place. After much debugging, I finally discovered that the use of the dynamic keyword was causing bigtime problems with Dotfucator.

Understanding what is going on with a simple sample is useful.  Say I have the following method to be obfuscated:

static void MyMethod() { int num = 1; int i = DoSomething(num); }

When obfuscated and then inspected in Reflector, here’s what it looks like:

private static void a() { int num = 1; int num2 = a(num); }

That’s what you’d expect.  But, now let’s say you change that method to use dynamic types instead of int. So it looks like this:

static void MyMethod() { dynamic num = 1; int i = DoSomething(num); }

When obfuscated and then disassembled in Reflector, here’s what you get:

private static void a() { object obj2 = 1; if (a.a == null) { a.a = CallSite<Func<CallSite, object, int>>.Create(Binder.Convert(CSharpBinderFlags.None, typeof(int), typeof(a))); } if (a.b == null) { a.b = CallSite<Func<CallSite, Type, object, object>>.Create(Binder.InvokeMember(CSharpBinderFlags.None, "DoSomething", null, typeof(a), new CSharpArgumentInfo[] { CSharpArgumentInfo.Create(CSharpArgumentInfoFlags.IsStaticType | CSharpArgumentInfoFlags.UseCompileTimeType, null), CSharpArgumentInfo.Create(CSharpArgumentInfoFlags.None, null) })); } int num = a.a.Target(a.a, a.b.Target(a.b, typeof(a), obj2)); }

Eek! Not only is it ugly as can be, look at how the method name DoSomething is passed as a  string to the InvokeMember method. But, DoSomething has been ofbuscated as a, so the assembly no longer has any notion of the DoSomething name for that method.

I’m not the only one who’s hit this as you can see from this post in the Dotfuscator forums.  Unfortunately, I couldn’t find a workaround other than removing the use of the dynamic keyword.

Background Transfer Download Sample For C# In Windows 8

September 27 2011
Was hitting buffer limits with the HttpClient even when I set the MaxResponseContentBufferSize property.  I even tried to get around the buffer problem by manually reading each byte out of the stream into my own byte array but still no luck. Then I found the Javascript sample for the Background Downloader:
I realized I could use it from managed code too using the Windows.Networking.BackgroundTransfer namespace.  Check out the code -- much niftier than the HttpClient. The code is pretty straight forward: 
StorageFolder localFolder = KnownFolders.MusicLibrary; 
StorageFile sampleFile = await localFolder.CreateFileAsync("my.mp3", CreationCollisionOption.ReplaceExisting);
var downloader = new BackgroundDownloader() 
d = downloader.StartDownloadAsync(
new Uri("http://linkto.mp3", UriKind.Absolute), sampleFile);
What's nifty is that once you pass the handle to the IStorageFile, all the i/o as far as saving the file gets handled for you. And, you can queue them up, so your app can start downloading n number of files and the BackgroundDownloader handles it all for you.
There are Progress and Completed events wired up to the DownloadOperation although I can't quite figure out how to wire them up -- don't see any sample code and I can't seem to get the signature of the method handler right. It wants to be an AsyncActionProgressHandler with the BackgroundDownloadProgress type as its parameter, but when I create a method with that signature, the compiler isn't happy. I'm having the same problem with the Completed handler as well -- if anyone figures out how to do it in C# (its simple in Javascript, thanks to the promise object!) let me know!

.NET Metro Surface Area

September 22 2011
Pretty interesting slide in Krzysztof Cwalina's talk: about the surface area for .NET metro apps. Note that the count on the .NET Metro APIs do not include the WinRT apis.
  .NET Metro .NET 4.5 Windows Phone
# assemblies 15+ 120 22
# namespaces 60 400 88
# types ~1k ~14k ~2k

Navigation Sample For Windows 8 Metro Style Apps Using .NET

September 22 2011

Doing navigation in Windows 8 apps is a little confusing and not well documented. With that in mind, I put together a sample which should help clarify things.

Joe Stegman outlines the basics in this forum response but let me drill in a bit. The confusing bit is that the templates in VS make your pages use UserControl as the root instead of Page.  So, you have to manually change the root element to Page.  The other confusing bit is that you need to handle things in the OnNavigatedTo override. That's how you can get the NavigationEventArgs and get at parameters that you've passed.

Beware of putting any logic into the Page_Loaded event when using Navigation. I hit a really weird bug where my i/o calls were mysteriously failing when I tried to do them in the Loaded event after I'd committed to the Page navigation paradigm. (114.02 kb)

Adding References To System Libraries In Windows 8

September 21 2011

I've been starting to hack on Windows 8 a bit, using C# and XAML and the .NET 4.5 Framework. Y'all will probably figure this out, but if you don't, here's the deal: when you go to add references to system libraries, none of your favorites (System.Web, System.Net, System.Security, PresentationCore, etc.) will show up.  You have to manually browse to C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5 and walla, there they all are in their .NET glory. Party on.

Or not.  Didn’t realize this could get you in trouble when you go to submit.  Look here for the libraries available to you: 

Beware though: .NET in Windows 8 is not exactly .NET.  This reference page starts to explain and this presentation by the eminent Krzysztof Cwalina definitely helps. 

More coming...

Visual Studio Achievements
Karsten Januszewski (207 Points)