A Simple Box.Com C# API Wrapper

August 25 2014

I had a need to access Box.com programmatically to do a daily pull of log files that were posted to Box from a third party service. At first I thought the Box .NET SDK would be helpful, but I quickly realized it is entirely oriented to be used by apps with a UI, not headless apps like this.  So, I dove into the documentation

My first stumble was that the Box developer token expires in one hour. I was hoping I’d be able to use it as a const for my headless server application, but no luck.

So, I need to actually generate an access_token and a refresh_token. The only way to do that is go through their UI. Thanks to a very helpful post on StackOverflow I was able to generate a code that could be used to generate both an access_token and refresh_token (which last for 60 days).

By persisting the refresh_token, you can write code that gets a fresh access_token programmatically. So, basically, my wrapper has a method called Bootstrap which you pass the code that you get from copy/pasting it out of the querystring. And then it has RefreshBoxToken, which gets a new access_token if the access_token is expired.

Then, there are two additional wrappers that actually do work which I called GetFolderByID and GetFileAsStream. GetFolderByID assumes you have the folderID, which you can figure out from the Box UI itself.  Then, with a little JSON.NET, you can parse the response and get the list of files as a JArray:

JObject jObject = JObject.Parse(folderContents);
JArray jArray = jObject["item_collection"]["entries"] as JArray;

Then, you’ve got the power do download files!

I wrapped both calls in a generic DoBoxAPICall method. Below is the entire class that encapsulates the logic:

using ExportConvivaLogsToHadoopWebJob.Properties;
using Newtonsoft.Json.Linq;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net.Http;
using System.Text;
using System.Threading.Tasks;

namespace ExportConvivaLogsToHadoopWebJob

    public static class BoxAPIHelper
        private const string boxApiUrl = "https://api.box.com/2.0/";
        private const string boxClientId = "YOUR_ID";
        private const string boxClientSecret = "YOUR_SECRET";
        private static readonly HttpClient _httpClient = new HttpClient();
        private static int retryCount = 0;
        private static Stream DoBoxCall(string url, HttpMethod httpMethod)
            Stream stream;
            var request = new HttpRequestMessage() { RequestUri = new Uri(url), Method = httpMethod };
            request.Headers.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Authorization", "Bearer " + Settings.Default.boxAccessToken);
            var response = _httpClient.SendAsync(request).Result;
            if (!response.IsSuccessStatusCode && response.StatusCode == System.Net.HttpStatusCode.Unauthorized)
                if (retryCount < 2)
                    stream = DoBoxCall(url, httpMethod);
                    return stream;
                    throw new Exception("Failed to connect to Box.");

            retryCount = 0;

            return response.Content.ReadAsStreamAsync().Result;
        private static void RefreshBoxToken()
            using (var request = new HttpRequestMessage() { RequestUri = new Uri("https://www.box.com/api/oauth2/token"), Method = HttpMethod.Post })
                HttpContent content = new FormUrlEncodedContent(new[] 
                 new KeyValuePair<string, string>("grant_type", "refresh_token"), 
                 new KeyValuePair<string, string>("refresh_token", Settings.Default.boxRefreshToken),
                 new KeyValuePair<string, string>("client_id", boxClientId),
                 new KeyValuePair<string, string>("client_secret", boxClientSecret)

                request.Content = content;
                using (var response = _httpClient.SendAsync(request).Result)
                    if (!response.IsSuccessStatusCode)
                        throw new Exception("Box refresh token failed. A human needs to go to a browser and generate a fresh authorization code.");
                    JObject jObject = jObject = JObject.Parse(response.Content.ReadAsStringAsync().Result);
                    Settings.Default.boxAccessToken = (string)jObject["access_token"];
                    Settings.Default.boxRefreshToken = (string)jObject["refresh_token"];

        public static string GetFolderById(string folderId)
            string url = string.Format("{0}folders/{1}", boxApiUrl, folderId);
            Stream stream = DoBoxCall(url, HttpMethod.Get);

            StreamReader reader = new StreamReader(stream);

            return reader.ReadToEnd();
        public static void Bootstrap(string boxAccessCode)
            using (var request = new HttpRequestMessage() { RequestUri = new Uri("https://www.box.com/api/oauth2/token"), Method = HttpMethod.Post })
                HttpContent content = new FormUrlEncodedContent(new[] 
                 new KeyValuePair<string, string>("grant_type", "authorization_code"), 
                 new KeyValuePair<string, string>("code", boxAccessCode),
                 new KeyValuePair<string, string>("client_id", boxClientId),
                 new KeyValuePair<string, string>("client_secret", boxClientSecret)

                request.Content = content;
                var response = _httpClient.SendAsync(request).Result;
                if (response.IsSuccessStatusCode)
                    JObject jObject = jObject = JObject.Parse(response.Content.ReadAsStringAsync().Result);
                    Settings.Default.boxAccessToken = (string)jObject["access_token"];
                    Settings.Default.boxRefreshToken = (string)jObject["refresh_token"];



        public static Stream GetFileAsStream(string fileId)
            string url = string.Format("{0}files/{1}/content", boxApiUrl, fileId);
            return DoBoxCall(url, HttpMethod.Get);

Maybe that’ll help someone out there…

HDInsight Hadoop Hive Job Decompresses CSV GZIP Files By Default

August 8 2014

Been working with Hadoop (2.4.0) and Hive (0.13.0) with HDInsight (3.1) and it decompresses GZIP files into CSV by default.  Nice!  So, loading data with a Hive query in Powershell:

$response = Invoke-Hive -Query @"
LOAD DATA INPATH 'wasb://$container@$storageAccountName.blob.core.windows.net/file.csv.gz' 



No additional work or arguments to pass. I thought I had to do something like specified in this post with the io.compression.codecs=org.apache.hadoop.io.compress.GzipCodec but apparently not.


UPDATE: Just found this link: https://cwiki.apache.org/confluence/display/Hive/CompressedStorage which goes into keeping compressed data in Hive which has a recommendation to create a SequenceFile.

Microsoft Azure and Visual Studio Online Server Builds – Tips & Tricks

April 22 2014

I’m all about deploying web sites and cloud services via server builds. Say goodbye to deployments from a developer’s box that can’t be reproduced on another box. Say goodbye to deploying code that isn’t checked in. Say goodbye to deployments that aren’t fully tested.  It is indeed super cool.

So how to get it set up? There are some good tutorials out there, but it can get a little tricky. Here’s how I did it.

First, make sure that you’ve linked your Azure account to your Visual Studio Online repository as explained in Step 3 here: http://azure.microsoft.com/en-us/documentation/articles/cloud-services-continuous-delivery-use-vso/

Then, if you open VS from the Azure website, it will generate a build template for you that is the name of your Azure deployment with an _CD at the end. You’ll have to tweak some things to get it happy though.

First, go the Build section of Team Explorer window:


Then, right click on the Build definition file and click Edit Build Definition.

There’s a bunch of things you will need to change.

In the general tab, make sure you enable the build definition. By default it is disabled:


In the source settings tab, make sure it is pointing to the right repository:


In the Trigger tab, you may want to tweak when the deployments happen:


And, in the Process tab, make sure you point the Project it to the right .sln to build as well as the Configuration you want aka Release | Any CPU.

And, in the deployment settings, make sure that the the Path To Deployment Settings points to your .pubxml file and the Windows Azure Deployment Environment points to the name of the Cloud Service in Windows Azure. 


With that all set, you can now build and deploy using server builds!

The New Iteration

April 7 2014

cover Having just gotten back from Build 2014, I felt inspired by conversations and sessions that were all about XAML and developer/designer workflow, and I started thinking about a paper Jaime Rodriguez and I wrote six years ago called The New Iteration: How XAML Transforms The Collaboration Between Designers and Developers. I went to go re-read it and, ack, I got a 404!

Well, that’s not okay, so here it is in all its glory. Still a lot of great content in there methinks.

Adding A Custom Header When Posting JSON Using HttpClient

November 8 2013

UPDATE: See the comments for a better way to do this!

HttpClient comes with handy methods for the very common task of posting JSON to a web service using the various PostAsJsonAsync methods. It handles serializing your object and crafting up the http request for you, aka

var gizmo = new Product() { Name = "Gizmo", Price = 100, Category = "Widget" };
Uri gizmoUri = null;
response = client.PostAsJsonAsync("api/products", gizmo).Result;
if (response.IsSuccessStatusCode)
    gizmoUri = response.Headers.Location;
    Console.WriteLine("{0} ({1})", (int)response.StatusCode, response.ReasonPhrase);

However, what happens if you need to add a custom header to your request? The only way to add a custom header is to craft up an HttpRequestMessage. And the nifty PostAsJsonAsync won’t take a HttpRequestMessage as a parameter; you have to use the SendAsync method.  No sweat; you have to write a little more code but it is no big deal. Here’s what it looks like:

var gizmo = new Product() { Name = "Gizmo", Price = 100, Category = "Widget" }; 
Uri gizmoUri = null;
MediaTypeFormatter jsonFormatter = new JsonMediaTypeFormatter(); HttpContent content = new ObjectContent<Product>(gizmo , jsonFormatter); var request = new HttpRequestMessage() { RequestUri = new Uri("api/products"), Method = HttpMethod.Post, Content = content }; request.Headers.Add("My-Special-Header", "xx-oo-xx-oo"); var response = _httpClient.SendAsync(request).Result;

Using Parallel.ForEach To Aggregate Results From JSON Files Stored in Windows Azure Blob Storage

April 17 2012

I love NoSQL, except when it comes to reporting.  Then I miss those handy SQL aggregation calls. I recently had a situation where I needed to look at a whole bunch of JSON files stored in Windows Azure Blob Storage and aggregate values from within those JSON files. The exact scenario was to get a count of how many people had achieved each achievement as part of the Visual Studio Achievements project.

Turned out to be an ideal use case for using some of the parallel programming features in .NET 4.0. Rather than download and process N JSON blobs linearly, I could throw the loop in a Parallel.ForEach block and gain speed from the multi-core machine I was using. The code was pretty straightforward, especially once I discovered the System.Collections.Concurrent namespace with its handy ConcurrentDictionary.

And, most significantly, it increased performance by 400 percent!

Below is the code in it’s entirety; I’ll walk through it here:

For JSON deserialization, I’m using a library provided by the WCF team up on CodePlex which is now part of the ASP.NET Web API. It provides some nifty features for turning JSON into dynamic objects.

The first thing I do is download a JSON file that has all the achievements, which is publically available (line 25).

I then put all the achievements in a ConcurrentDictionary<string, int> which I’ll use to build my report (line 33).

Then, I get the blobs and start my Parallel.ForEach loop (line 41). Inside the Action(TSource), I walk the list of the user’s earned achievements, incrementing the count in the ConcurrentDictionary of each achievement (line 57).

Finally, once I exit the loop, I turn the dictionary into an Excel spreadsheet (line 69) – folks tend to like that format.

You’ll notice the remmed out code; that’s the old non-parallelized code if you’d like to compare.

1 using System; 2 using System.Collections.Concurrent; 3 using System.IO; 4 using System.Json; 5 using System.Net; 6 using System.Threading.Tasks; 7 using Microsoft.WindowsAzure; 8 using Microsoft.WindowsAzure.StorageClient; 9 10 namespace AchievementsReporting 11 { 12 class Program 13 { 14 15 static void Main(string[] args) 16 { 17 var cloudBlobClient = new CloudBlobClient(new Uri("https://---.blob.core.windows.net", UriKind.Absolute), 18 new StorageCredentialsAccountAndKey("---", 19 "---")); 20 var container = cloudBlobClient.GetContainerReference("users"); 21 22 string masterJson = string.Empty; 23 using (var webClient = new WebClient()) 24 { 25 masterJson = 26 webClient.DownloadString(new Uri("http://channel9.msdn.com/achievements/visualstudio?json=true")); 27 } 28 dynamic masterList = JsonValue.Parse(masterJson); 29 var statisticsDictionary = new ConcurrentDictionary<string, int>(); 30 //var statisticsDictionary = new Dictionary<string, int>(); 31 foreach (var achieve in masterList.Achievements) 32 { 33 statisticsDictionary.GetOrAdd(achieve.Name.ToString(), 0); 34 //statisticsDictionary.Add(achieve.Name.ToString(), 0); 35 } 36 BlobRequestOptions options = new BlobRequestOptions(); 37 options.UseFlatBlobListing = true; 38 options.BlobListingDetails = BlobListingDetails.Snapshots; 39 Console.WriteLine("Starting..."); 40 DateTime start = DateTime.Now; 41 Parallel.ForEach(container.ListBlobs(options), blobListItem => 42 //foreach (var blobListItem in container.ListBlobs(options)) 43 { 44 CloudBlob blob = 45 container.GetBlobReference( 46 blobListItem.Uri.AbsoluteUri); 47 string json = blob.DownloadText(); 48 if (!string.IsNullOrEmpty(json)) 49 { 50 dynamic achievementsDynamic = 51 JsonValue.Parse(json) as dynamic; 52 foreach ( 53 var achieve in achievementsDynamic.Achievements) 54 { 55 if (achieve.DateEarned != null) 56 { 57 statisticsDictionary[achieve.Name.ToString()] = 58 statisticsDictionary[achieve.Name.ToString()] + 1; 59 } 60 } 61 } 62 } 63 ); 64 65 using (StreamWriter writer = new StreamWriter("report.xls")) 66 { 67 foreach (var key in statisticsDictionary.Keys) 68 { 69 writer.WriteLine(key + "\t" + statisticsDictionary[key].ToString()); 70 } 71 } 72 TimeSpan diff = DateTime.Now - start; 73 Console.WriteLine("done - took: "); 74 Console.WriteLine(diff.TotalMinutes); 75 } 76 } 77 } 78

Visual Studio Achievements Buzz

January 26 2012

The project I’ve been working on recently finally shipped: Visual Studio Achievements.

It’s been great to see the reaction to it, including posts in Wired, Ars Technica, Life Hacker, BoingBoing, Gamasutra and more. It also sparked quite a discussion in Reddit and Slashdot. Here’s a list of all the buzz that the project has generated:

Microsoft Crossbreeds Programming Kit with Fantasy Game

Caleb Garling/Wired

January 24, 2012


Coders, Motivate Yourself with Achievements

Logan Booker/Lifehacker Australia

January 21, 2012


Microsoft Keeps It Old-School with a Pricey Text Adventure Game, Visual Studio 2010

Peter Bright/Ars Technica

January 20, 2012


Microsoft Text Adventure Game!

Rob Beschizza/BoingBoing

January 20, 2012


Microsoft Turns Coding Into a game with New Visual Studio Plug-In

Matt Williams/Games.on.net

January 20, 2012


Microsoft to Make Programming Fun

David Stellmack/Fudzilla

January 20, 2012


Microsoft Corporation Adding Visual Studio

Rachael Brunelli/eMoneyDaily

January 20, 2012


Microsoft Gamifies Visual Studio with Achievements

Mike Rose/Gamasutra

January 19, 2012


Microsoft Now Has Achievements for… Developers

Luke Plunkett/Kotaku

January 19, 2012


Visual Studio Gets Achievements

Chris Duckett/TechRepublic

January 18, 2012


Microsoft Adds Achievements to Visual Studio Software

Tom Bramwell/EuroGamer.net

January 19, 2012


Microsoft Adds Developer Achievements to Visual Studio

Nathan Brown/Edge

January 19, 2012


Visual Studio Achievements Program

Sue Gee/I Programmer

January 19, 2012


Microsoft Brings Achievements to Visual Studio

Earnest “Nex” Cavalli/ Escapist Magazine

January 19, 2012


Achievement Unlocked: Microsoft Gamifies Development

Craig Chapple/Develop

January 19, 2012


Microsoft Adds Visual Studio Achievements for Developers

Laurentiu Stan/Social Barrel

January 19, 2012


Microsoft Announces Visual Studio Achievements Beta, A Pat on the Back for Dev


January 19, 2012


Visual Studio Achievements Program Brings Gamification to Development

Staff Writer/The Financial

January 19, 2012


Visual Studio Adds Game Mechanic to Keep Devs Engaged

Jason Cartwright/TechAU

January 19, 2012


Microsoft Introduces Xbox-like Achievements for Developers

Tom Warren/The Verge

January 18, 2012


The Coding Game: Microsoft’s Visual Studio Gets Badges, Achievements and Leaderboard

Todd Bishop/GeekWire

January 18, 2012


Channel9’s Visual Studio Achievements Now Available

Long Zheng/istartedsomething

January 18, 2012


Microsoft Visual Studio Brings Gamification to App Development

Chris Burns/Slashgear

January 18, 2012


Microsoft Announces Visual Studio Achievements Beta

Pradeep Viswav/WMPoweruser

January 18, 2012


Microsoft Adds Achievements for Developers

Justin Rubio/IGN

January 18, 2012


Achievement Unlocked… For Developers

Paul Thurrott/Supersite for Windows

January 18, 2012


Microsoft Announces ‘Visual Studio Achievements’ for Developers

Simon LR/Techie Buzz

January 18, 2012


Visual Studio Achievements, Now a Reality!

Rudi/While True Blog

January 18, 2012


Visual Studio Achievements Program Brings Gamification to Development

Staff Writer/Adafruit Industries Blog

January 18, 2012


Visual Studio Achievements – The Beta Goes Live!

Alvin Ashcraft/Alvin Ashcraft’s Morning Dew

January 18, 2012


Visual Studio Achievements – Remember Kids They’re Just for Fun

Bill Simser/Fear and Loathing

January 18, 2012


Don’t Brag About Your Visual Studio Achievements! (Yet?)

Maarten Balliauw/Maarten Balliauw {blog}

January 18, 2012


Querying The Archivist API With JSON.NET

December 16 2011

Someone recently asked me for a sample of how to query The Archivist API to programmatically get a list of the top users of a given search term.

So I put a quick sample together, using the most excellent JSON.NET library’s LINQ provider for querying JSON. 

I used an archive on Wittgenstien.  To get the JSON that generated this chart, I simply append ?format=json to the URL, like this: http://archivist.visitmix.com/irhetoric/75/user?format=json  which results in the following response:



Here’s the program I wrote to parse this:

string json = string.Empty; using (WebClient webClient = new WebClient()) { json = webClient.DownloadString("http://archivist.visitmix.com/irhetoric/75/user?format=json"); } JArray result = JArray.Parse(json); foreach (JObject user in result) { Console.WriteLine((string)user["Name"] + " - " + (float)user["Count"]); } Console.ReadLine();

You can see how I simply pass the JSON to the static .Parse method hanging off the JArray class. If the root of the JSON wasn’t an array, I would have used the JObject.Parse() method. Once it is in the array, I can loop it and extract the values. You can also write LINQ queries using JSON.NET. For example, if I only wanted users whose tweet count was greater than 100, I could write this:

var greaterThan100 = from r in result where (float)r["Count"] > 100 select r; foreach (JObject user in greaterThan100) { Console.WriteLine((string)user["Name"] + " - " + (float)user["Count"]); }

Dynamic Keyword and Dotfuscator = Very Unhappy

October 3 2011

I’ve been using Dotfuscator to obfuscate an assembly.  I thought all was good, but when I tried to use the obfuscate assembly, the assembly didn’t work like the unobfuscated one did and was throwing exceptions all over the place. After much debugging, I finally discovered that the use of the dynamic keyword was causing bigtime problems with Dotfucator.

Understanding what is going on with a simple sample is useful.  Say I have the following method to be obfuscated:

static void MyMethod() { int num = 1; int i = DoSomething(num); }

When obfuscated and then inspected in Reflector, here’s what it looks like:

private static void a() { int num = 1; int num2 = a(num); }

That’s what you’d expect.  But, now let’s say you change that method to use dynamic types instead of int. So it looks like this:

static void MyMethod() { dynamic num = 1; int i = DoSomething(num); }

When obfuscated and then disassembled in Reflector, here’s what you get:

private static void a() { object obj2 = 1; if (a.a == null) { a.a = CallSite<Func<CallSite, object, int>>.Create(Binder.Convert(CSharpBinderFlags.None, typeof(int), typeof(a))); } if (a.b == null) { a.b = CallSite<Func<CallSite, Type, object, object>>.Create(Binder.InvokeMember(CSharpBinderFlags.None, "DoSomething", null, typeof(a), new CSharpArgumentInfo[] { CSharpArgumentInfo.Create(CSharpArgumentInfoFlags.IsStaticType | CSharpArgumentInfoFlags.UseCompileTimeType, null), CSharpArgumentInfo.Create(CSharpArgumentInfoFlags.None, null) })); } int num = a.a.Target(a.a, a.b.Target(a.b, typeof(a), obj2)); }

Eek! Not only is it ugly as can be, look at how the method name DoSomething is passed as a  string to the InvokeMember method. But, DoSomething has been ofbuscated as a, so the assembly no longer has any notion of the DoSomething name for that method.

I’m not the only one who’s hit this as you can see from this post in the Dotfuscator forums.  Unfortunately, I couldn’t find a workaround other than removing the use of the dynamic keyword.

Table Storage, Continuation Tokens, Windows Azure

September 8 2011

By default, queries to table storage will return a maximum of 1000 rows, whether you are using the REST APIs or the LINQ provider.  If you want more rows, you need to get fancy.

It’s pretty easy.  Let’s say you have created some data access classes similar to how the Azure Hands On Labs and samples encourage you to do so. 

So say you’ve got a class that inherits from TableServiceContext that looks like this:

class RawDataContext : TableServiceContext { public RawDataContext (string baseAddress, StorageCredentials credentials) : base (baseAddress,credentials) { } public IQueryable<RawEntity> RawEntity { get { return this.CreateQuery<RawEntity>("RawEntity"); } } }

Well, you are going to want to change that, so your data context returns a CloudTableQuery instead.  So, now, your class should look like this:

class RawDataContext : TableServiceContext { public RawDataContext (string baseAddress, StorageCredentials credentials) : base (baseAddress,credentials) { } public CloudTableQuery<RawEntity> RawEntity { get { return this.CreateQuery<RawEntity>("RawEntity").AsTableServiceQuery(); } } }
Notice how the CreateQuery method has been appended with the AsTableServiceQuery() extension method. Notice also that the return type is CloudTableQuery instead of IQueryable.

But that’s not all. You’ll also need to make some changes when you issue your queries.  So, for example, say you have a DataSource class with a select query that looks like this:

public class RawDataSource { private static CloudStorageAccount storageAccount; private RawDataContext context; public RawDataSource() { storageAccount = CloudStorageAccount.FromConfigurationSetting("StorageConnection"); this.context = new RawDataContext(storageAccount.TableEndpoint.AbsoluteUri, storageAccount.Credentials); } public List<RawEntity> Select(string site) { var results = (from g in this.context.RawEntity where g.PartitionKey == site select g).ToList(); return results; } }

You’ll need to change that query such that it issues the Execute method off the CloudTableQuery result, like this:

public List<RawEntity> Select(string site) { var results = (from g in this.context.RawEntity.Execute() where g.PartitionKey == site select g).ToList(); return results; }

By adding the Execute method, the query will issue the continuation token on your behalf to Azure and concatenate the data to the result set, returning > 1000 rows.