Visual Studio Achievements For Windows Azure Buzz

May 1 2012

Here’s some links to posts and stories about the 15 new Windows Azure achievements added to Visual Studio Achievements:

Visual Studio Adds Windows Azure Achievements

Azure Gamified - More Badges!
I Programmer

Visual Studio Achievements Brings Gamification to Windows Azure Development
MSFT post

Announcing Windows Azure Achievements For Visual Studio
Windows Azure Post

Cloud Cover Video

Announcing Visual Studio Achievements For Windows Azure

Using Parallel.ForEach To Aggregate Results From JSON Files Stored in Windows Azure Blob Storage

April 17 2012

I love NoSQL, except when it comes to reporting.  Then I miss those handy SQL aggregation calls. I recently had a situation where I needed to look at a whole bunch of JSON files stored in Windows Azure Blob Storage and aggregate values from within those JSON files. The exact scenario was to get a count of how many people had achieved each achievement as part of the Visual Studio Achievements project.

Turned out to be an ideal use case for using some of the parallel programming features in .NET 4.0. Rather than download and process N JSON blobs linearly, I could throw the loop in a Parallel.ForEach block and gain speed from the multi-core machine I was using. The code was pretty straightforward, especially once I discovered the System.Collections.Concurrent namespace with its handy ConcurrentDictionary.

And, most significantly, it increased performance by 400 percent!

Below is the code in it’s entirety; I’ll walk through it here:

For JSON deserialization, I’m using a library provided by the WCF team up on CodePlex which is now part of the ASP.NET Web API. It provides some nifty features for turning JSON into dynamic objects.

The first thing I do is download a JSON file that has all the achievements, which is publically available (line 25).

I then put all the achievements in a ConcurrentDictionary<string, int> which I’ll use to build my report (line 33).

Then, I get the blobs and start my Parallel.ForEach loop (line 41). Inside the Action(TSource), I walk the list of the user’s earned achievements, incrementing the count in the ConcurrentDictionary of each achievement (line 57).

Finally, once I exit the loop, I turn the dictionary into an Excel spreadsheet (line 69) – folks tend to like that format.

You’ll notice the remmed out code; that’s the old non-parallelized code if you’d like to compare.

1 using System; 2 using System.Collections.Concurrent; 3 using System.IO; 4 using System.Json; 5 using System.Net; 6 using System.Threading.Tasks; 7 using Microsoft.WindowsAzure; 8 using Microsoft.WindowsAzure.StorageClient; 9 10 namespace AchievementsReporting 11 { 12 class Program 13 { 14 15 static void Main(string[] args) 16 { 17 var cloudBlobClient = new CloudBlobClient(new Uri("", UriKind.Absolute), 18 new StorageCredentialsAccountAndKey("---", 19 "---")); 20 var container = cloudBlobClient.GetContainerReference("users"); 21 22 string masterJson = string.Empty; 23 using (var webClient = new WebClient()) 24 { 25 masterJson = 26 webClient.DownloadString(new Uri("")); 27 } 28 dynamic masterList = JsonValue.Parse(masterJson); 29 var statisticsDictionary = new ConcurrentDictionary<string, int>(); 30 //var statisticsDictionary = new Dictionary<string, int>(); 31 foreach (var achieve in masterList.Achievements) 32 { 33 statisticsDictionary.GetOrAdd(achieve.Name.ToString(), 0); 34 //statisticsDictionary.Add(achieve.Name.ToString(), 0); 35 } 36 BlobRequestOptions options = new BlobRequestOptions(); 37 options.UseFlatBlobListing = true; 38 options.BlobListingDetails = BlobListingDetails.Snapshots; 39 Console.WriteLine("Starting..."); 40 DateTime start = DateTime.Now; 41 Parallel.ForEach(container.ListBlobs(options), blobListItem => 42 //foreach (var blobListItem in container.ListBlobs(options)) 43 { 44 CloudBlob blob = 45 container.GetBlobReference( 46 blobListItem.Uri.AbsoluteUri); 47 string json = blob.DownloadText(); 48 if (!string.IsNullOrEmpty(json)) 49 { 50 dynamic achievementsDynamic = 51 JsonValue.Parse(json) as dynamic; 52 foreach ( 53 var achieve in achievementsDynamic.Achievements) 54 { 55 if (achieve.DateEarned != null) 56 { 57 statisticsDictionary[achieve.Name.ToString()] = 58 statisticsDictionary[achieve.Name.ToString()] + 1; 59 } 60 } 61 } 62 } 63 ); 64 65 using (StreamWriter writer = new StreamWriter("report.xls")) 66 { 67 foreach (var key in statisticsDictionary.Keys) 68 { 69 writer.WriteLine(key + "\t" + statisticsDictionary[key].ToString()); 70 } 71 } 72 TimeSpan diff = DateTime.Now - start; 73 Console.WriteLine("done - took: "); 74 Console.WriteLine(diff.TotalMinutes); 75 } 76 } 77 } 78

Migrating VSIX Extensions To Dev11

March 30 2012

Hit an interesting issue with porting Visual Studio VSIX extensions to Dev11. 

To get my extension to work in Dev11, I tweaked the source.extension.vsixmanifest file. While there is a GUI for editing that file, it doesn’t expose the ability to add Dev11 as a supported edition. However, you can open the source.extension.vsixmanifest up in your favorite xml editor and party on it there. In my case, I changed it as follows:

<SupportedProducts> <VisualStudio Version="10.0"> <Edition>Ultimate</Edition> <Edition>Premium</Edition> <Edition>Pro</Edition> <Edition>IntegratedShell</Edition> </VisualStudio> <VisualStudio Version="11.0"> <Edition>Ultimate</Edition> <Edition>Premium</Edition> <Edition>Pro</Edition> <Edition>IntegratedShell</Edition> </VisualStudio> </SupportedProducts>

Easy enough – the vsix installer will then install your extension for both versions. Nice!


However, I experienced some odd behavior that may catch some people.  There’s another section of the vsixmanifest called the <Content> section.  You are supposed to put an element into that section called the <VsPackage> element, with the name of the pkgdef file (which is an .ini file).  However, in VS2010, you could get away without having this element. But, in Dev11, if you don’t have this element, your extension will install but never initialize.  So make sure you have this element, either as macro, aka


Or as a hardcoded value:


Finally, be sure to test test test in Dev11, as you never know what might get ya.

StreamInsight Nuggets

March 22 2012

Here’s some choice quotes from a whitepaper on StreamInsight:

Relational database applications typically acquire data and store it to disk before it can be analyzed. We therefore call analysis with traditional relational database systems query-driven. Query-driven analysis is well-suited for historical data. … To reach the necessary performance and scale, [some] applications need to analyze the data in near real time while it is being acquired from the source. We denote these applications as event-driven applications because new event data arriving at the system triggers the necessary analysis.

I think this is a great intro to help wrap yr head around the difference between query-driven and event-driven analysis.

Microsoft StreamInsight is Microsoft’s platform to build high-throughput, low-latency event-driven analytics applications.

That’s a great one liner.

With StreamInsight, business insight is delivered at the speed at which data is produced, as opposed to the speed at which traditional reports are processed or consumed.

Nicely worded value prop.

StreamInsight’s runtime performs calculations incrementally whenever possible. This means that the processing only involves the data for the current result and the new event. Unlike in traditional databases, updating a report with aggregates or KPIs with StreamInsight does not require to re-iterate through past data once a new event comes in. Instead, StreamInsight answers continuous queries with a single pass over all the data, which is an important capability for long-running, potentially infinite, standing queries. Incremental processing is one key performance benefit of StreamInsight.

That is rad engineering methinks.

StreamInsight automatically distributes the processing across the available processor cores on the system as well. Thread management and query parallelization are performed automatically by the system.

How cool is that!

Speaking At GSummit

March 19 2012

Gonna be speaking at GSummit, a conference about gamefication, in June, down in San Francisco.  More here.

Visual Studio Achievements Buzz

January 26 2012

The project I’ve been working on recently finally shipped: Visual Studio Achievements.

It’s been great to see the reaction to it, including posts in Wired, Ars Technica, Life Hacker, BoingBoing, Gamasutra and more. It also sparked quite a discussion in Reddit and Slashdot. Here’s a list of all the buzz that the project has generated:

Microsoft Crossbreeds Programming Kit with Fantasy Game

Caleb Garling/Wired

January 24, 2012

Coders, Motivate Yourself with Achievements

Logan Booker/Lifehacker Australia

January 21, 2012

Microsoft Keeps It Old-School with a Pricey Text Adventure Game, Visual Studio 2010

Peter Bright/Ars Technica

January 20, 2012

Microsoft Text Adventure Game!

Rob Beschizza/BoingBoing

January 20, 2012

Microsoft Turns Coding Into a game with New Visual Studio Plug-In

Matt Williams/

January 20, 2012

Microsoft to Make Programming Fun

David Stellmack/Fudzilla

January 20, 2012

Microsoft Corporation Adding Visual Studio

Rachael Brunelli/eMoneyDaily

January 20, 2012

Microsoft Gamifies Visual Studio with Achievements

Mike Rose/Gamasutra

January 19, 2012

Microsoft Now Has Achievements for… Developers

Luke Plunkett/Kotaku

January 19, 2012

Visual Studio Gets Achievements

Chris Duckett/TechRepublic

January 18, 2012

Microsoft Adds Achievements to Visual Studio Software

Tom Bramwell/

January 19, 2012

Microsoft Adds Developer Achievements to Visual Studio

Nathan Brown/Edge

January 19, 2012

Visual Studio Achievements Program

Sue Gee/I Programmer

January 19, 2012

Microsoft Brings Achievements to Visual Studio

Earnest “Nex” Cavalli/ Escapist Magazine

January 19, 2012

Achievement Unlocked: Microsoft Gamifies Development

Craig Chapple/Develop

January 19, 2012

Microsoft Adds Visual Studio Achievements for Developers

Laurentiu Stan/Social Barrel

January 19, 2012

Microsoft Announces Visual Studio Achievements Beta, A Pat on the Back for Dev


January 19, 2012

Visual Studio Achievements Program Brings Gamification to Development

Staff Writer/The Financial

January 19, 2012

Visual Studio Adds Game Mechanic to Keep Devs Engaged

Jason Cartwright/TechAU

January 19, 2012

Microsoft Introduces Xbox-like Achievements for Developers

Tom Warren/The Verge

January 18, 2012

The Coding Game: Microsoft’s Visual Studio Gets Badges, Achievements and Leaderboard

Todd Bishop/GeekWire

January 18, 2012

Channel9’s Visual Studio Achievements Now Available

Long Zheng/istartedsomething

January 18, 2012

Microsoft Visual Studio Brings Gamification to App Development

Chris Burns/Slashgear

January 18, 2012

Microsoft Announces Visual Studio Achievements Beta

Pradeep Viswav/WMPoweruser

January 18, 2012

Microsoft Adds Achievements for Developers

Justin Rubio/IGN

January 18, 2012

Achievement Unlocked… For Developers

Paul Thurrott/Supersite for Windows

January 18, 2012

Microsoft Announces ‘Visual Studio Achievements’ for Developers

Simon LR/Techie Buzz

January 18, 2012

Visual Studio Achievements, Now a Reality!

Rudi/While True Blog

January 18, 2012

Visual Studio Achievements Program Brings Gamification to Development

Staff Writer/Adafruit Industries Blog

January 18, 2012

Visual Studio Achievements – The Beta Goes Live!

Alvin Ashcraft/Alvin Ashcraft’s Morning Dew

January 18, 2012

Visual Studio Achievements – Remember Kids They’re Just for Fun

Bill Simser/Fear and Loathing

January 18, 2012

Don’t Brag About Your Visual Studio Achievements! (Yet?)

Maarten Balliauw/Maarten Balliauw {blog}

January 18, 2012!-%28yet%29.aspx

Querying The Archivist API With JSON.NET

December 16 2011

Someone recently asked me for a sample of how to query The Archivist API to programmatically get a list of the top users of a given search term.

So I put a quick sample together, using the most excellent JSON.NET library’s LINQ provider for querying JSON. 

I used an archive on Wittgenstien.  To get the JSON that generated this chart, I simply append ?format=json to the URL, like this:  which results in the following response:



Here’s the program I wrote to parse this:

string json = string.Empty; using (WebClient webClient = new WebClient()) { json = webClient.DownloadString(""); } JArray result = JArray.Parse(json); foreach (JObject user in result) { Console.WriteLine((string)user["Name"] + " - " + (float)user["Count"]); } Console.ReadLine();

You can see how I simply pass the JSON to the static .Parse method hanging off the JArray class. If the root of the JSON wasn’t an array, I would have used the JObject.Parse() method. Once it is in the array, I can loop it and extract the values. You can also write LINQ queries using JSON.NET. For example, if I only wanted users whose tweet count was greater than 100, I could write this:

var greaterThan100 = from r in result where (float)r["Count"] > 100 select r; foreach (JObject user in greaterThan100) { Console.WriteLine((string)user["Name"] + " - " + (float)user["Count"]); }

Creating A Weighted Average User Defined Aggregate in StreamInsight 1.1

November 16 2011

I’ve been doing some prototyping StreamInsight lately and hit the following issue. I needed to write a weighted average so that I could roll up result sets.  Let me elaborate on the scenario.  I had a query whose HoppingWindow has a WindowSize of 1 minute.  One of the things the query returns is an average of one of the values returned by the event payload:

AvgLoadTime = win.Avg(e => e.LoadTime)

I then want to take all the results of that query and roll them up into a larger window of 1 hour. So that means writing a new query that uses the results of the first query for its calculation. But, if I were to simply take the average of the average, I’d get inaccurate results. What I need was the weighted mean.

To do so, I changed the code to look like this:

AvgLoadTime = win.Sum(e => e.AvgLoadTime * e.RecordCount)/win.Sum(e => e.RecordCount)

But when I deployed to StreamInsight 1.1 and got no love:

Microsoft.ComplexEventProcessing.Linq.QueryGenerationException: Expression '(e.Sum(e => (e.AvgLoadTime * Convert(e.RecordCount))) / Convert(e.Sum(e => e.RecordCount)))' contains more than one aggregate method call, which is not supported.

Doh. But then I got hip to user-defined aggregates. I found the documentation on this MSDN page essential and it helped me to write my extension, which looks like this:

public class WeightedAverage : CepAggregate<WqAggregate, double> { public override double GenerateOutput(IEnumerable<WqAggregate> win) { var AvgLoadTime = win.Sum(e => e.AvgLoadTime*e.RecordCount)/win.Sum(e => e.RecordCount); return AvgLoadTime; } } public static class ExtensionMethods { [CepUserDefinedAggregate(typeof(WeightedAverage))] public static double WeightedAvg(this CepWindow<WqAggregate> window) { throw CepUtility.DoNotCall(); } }

Where WqAggregate is just a class with the various fields that are returned as a result of my query.

So, I can now write this code:

AvgLoadTime = win.WeightedAvg()

Nice! I just had to deploy the .dll where I wrote the extension to the StreamInsight server and I was good to go. Now, with StreamInsight 1.2, the need to do this all goes away with but for now, this turns about to be a reasonable fix.

Dynamic Keyword and Dotfuscator = Very Unhappy

October 3 2011

I’ve been using Dotfuscator to obfuscate an assembly.  I thought all was good, but when I tried to use the obfuscate assembly, the assembly didn’t work like the unobfuscated one did and was throwing exceptions all over the place. After much debugging, I finally discovered that the use of the dynamic keyword was causing bigtime problems with Dotfucator.

Understanding what is going on with a simple sample is useful.  Say I have the following method to be obfuscated:

static void MyMethod() { int num = 1; int i = DoSomething(num); }

When obfuscated and then inspected in Reflector, here’s what it looks like:

private static void a() { int num = 1; int num2 = a(num); }

That’s what you’d expect.  But, now let’s say you change that method to use dynamic types instead of int. So it looks like this:

static void MyMethod() { dynamic num = 1; int i = DoSomething(num); }

When obfuscated and then disassembled in Reflector, here’s what you get:

private static void a() { object obj2 = 1; if (a.a == null) { a.a = CallSite<Func<CallSite, object, int>>.Create(Binder.Convert(CSharpBinderFlags.None, typeof(int), typeof(a))); } if (a.b == null) { a.b = CallSite<Func<CallSite, Type, object, object>>.Create(Binder.InvokeMember(CSharpBinderFlags.None, "DoSomething", null, typeof(a), new CSharpArgumentInfo[] { CSharpArgumentInfo.Create(CSharpArgumentInfoFlags.IsStaticType | CSharpArgumentInfoFlags.UseCompileTimeType, null), CSharpArgumentInfo.Create(CSharpArgumentInfoFlags.None, null) })); } int num = a.a.Target(a.a, a.b.Target(a.b, typeof(a), obj2)); }

Eek! Not only is it ugly as can be, look at how the method name DoSomething is passed as a  string to the InvokeMember method. But, DoSomething has been ofbuscated as a, so the assembly no longer has any notion of the DoSomething name for that method.

I’m not the only one who’s hit this as you can see from this post in the Dotfuscator forums.  Unfortunately, I couldn’t find a workaround other than removing the use of the dynamic keyword.

Background Transfer Download Sample For C# In Windows 8

September 27 2011
Was hitting buffer limits with the HttpClient even when I set the MaxResponseContentBufferSize property.  I even tried to get around the buffer problem by manually reading each byte out of the stream into my own byte array but still no luck. Then I found the Javascript sample for the Background Downloader:
I realized I could use it from managed code too using the Windows.Networking.BackgroundTransfer namespace.  Check out the code -- much niftier than the HttpClient. The code is pretty straight forward: 
StorageFolder localFolder = KnownFolders.MusicLibrary; 
StorageFile sampleFile = await localFolder.CreateFileAsync("my.mp3", CreationCollisionOption.ReplaceExisting);
var downloader = new BackgroundDownloader() 
d = downloader.StartDownloadAsync(
new Uri("http://linkto.mp3", UriKind.Absolute), sampleFile);
What's nifty is that once you pass the handle to the IStorageFile, all the i/o as far as saving the file gets handled for you. And, you can queue them up, so your app can start downloading n number of files and the BackgroundDownloader handles it all for you.
There are Progress and Completed events wired up to the DownloadOperation although I can't quite figure out how to wire them up -- don't see any sample code and I can't seem to get the signature of the method handler right. It wants to be an AsyncActionProgressHandler with the BackgroundDownloadProgress type as its parameter, but when I create a method with that signature, the compiler isn't happy. I'm having the same problem with the Completed handler as well -- if anyone figures out how to do it in C# (its simple in Javascript, thanks to the promise object!) let me know!