Querying The Archivist API With JSON.NET

December 16 2011

Someone recently asked me for a sample of how to query The Archivist API to programmatically get a list of the top users of a given search term.

So I put a quick sample together, using the most excellent JSON.NET library’s LINQ provider for querying JSON. 

I used an archive on Wittgenstien.  To get the JSON that generated this chart, I simply append ?format=json to the URL, like this: http://archivist.visitmix.com/irhetoric/75/user?format=json  which results in the following response:

[{"Date":"\/Date(-62135596800000+0000)\/","Count":667.0,"Name":"lucasofri"},{"Date":"\/Date(-62135596800000+0000)\/","Count":439.0,"Name":"3PennyMovies"},{"Date":"\/Date(-62135596800000+0000)\/","Count":326.0,"Name":"quotemeal"},{"Date":"\/Date(-62135596800000+0000)\/","Count":260.0,"Name":"SiegenerZeitung"},{"Date":"\/Date(-62135596800000+0000)\/","Count":237.0,"Name":"lucasarrimada"},{"Date":"\/Date(-62135596800000+0000)\/","Count":236.0,"Name":"Double_bliss"},{"Date":"\/Date(-62135596800000+0000)\/","Count":175.0,"Name":"praashok"},{"Date":"\/Date(-62135596800000+0000)\/","Count":139.0,"Name":"PhilosophyQuotz"},{"Date":"\/Date(-62135596800000+0000)\/","Count":135.0,"Name":"NihilDeNada"},{"Date":"\/Date(-62135596800000+0000)\/","Count":132.0,"Name":"wilsonvoight"},{"Date":"\/Date(-62135596800000+0000)\/","Count":126.0,"Name":"imamrasyidi"},{"Date":"\/Date(-62135596800000+0000)\/","Count":117.0,"Name":"Iceburner"},{"Date":"\/Date(-62135596800000+0000)\/","Count":112.0,"Name":"emilsantosiii"},{"Date":"\/Date(-62135596800000+0000)\/","Count":111.0,"Name":"unwetterwarner"},{"Date":"\/Date(-62135596800000+0000)\/","Count":104.0,"Name":"LogicalAnalysis"},{"Date":"\/Date(-62135596800000+0000)\/","Count":102.0,"Name":"macondo83"},{"Date":"\/Date(-62135596800000+0000)\/","Count":93.0,"Name":"de_340513031"},{"Date":"\/Date(-62135596800000+0000)\/","Count":90.0,"Name":"neofibot"},{"Date":"\/Date(-62135596800000+0000)\/","Count":86.0,"Name":"hw1"},{"Date":"\/Date(-62135596800000+0000)\/","Count":83.0,"Name":"wirsiegen"},{"Date":"\/Date(-62135596800000+0000)\/","Count":82.0,"Name":"souvik_rt"},{"Date":"\/Date(-62135596800000+0000)\/","Count":79.0,"Name":"asmiather"},{"Date":"\/Date(-62135596800000+0000)\/","Count":76.0,"Name":"ogallardov"},{"Date":"\/Date(-62135596800000+0000)\/","Count":72.0,"Name":"pr0ject2501"},{"Date":"\/Date(-62135596800000+0000)\/","Count":71.0,"Name":"ggrigoriadis"}]

 

Here’s the program I wrote to parse this:

string json = string.Empty; using (WebClient webClient = new WebClient()) { json = webClient.DownloadString("http://archivist.visitmix.com/irhetoric/75/user?format=json"); } JArray result = JArray.Parse(json); foreach (JObject user in result) { Console.WriteLine((string)user["Name"] + " - " + (float)user["Count"]); } Console.ReadLine();

You can see how I simply pass the JSON to the static .Parse method hanging off the JArray class. If the root of the JSON wasn’t an array, I would have used the JObject.Parse() method. Once it is in the array, I can loop it and extract the values. You can also write LINQ queries using JSON.NET. For example, if I only wanted users whose tweet count was greater than 100, I could write this:

var greaterThan100 = from r in result where (float)r["Count"] > 100 select r; foreach (JObject user in greaterThan100) { Console.WriteLine((string)user["Name"] + " - " + (float)user["Count"]); }

Taking The M out of MVC - JSON, The dynamic Keyword, LINQ

May 25 2011

Okay, so I’ve been working on a new MVC project in all the data is stored as JSON. The plan is to have JSON deserialized in the controller (using the WCF JSON classes) and then pass the deserialized JSON to the view. The general thinking is that we won't have any models, just controllers and views that work with dynamically created object graphs from JSON.

So, I started implementing.  Works great. For example, here’s controller code that grabs some JSON out of blob storage, deserializes it, and returns it to the view:

public ActionResult Index(string name) { ICloudBlobContainer container = cloudBlobClient.GetContainerReference("applications"); ICloudBlob blob = container.GetBlobReference(string.Format("{0}.txt",name)); string json = blob.DownloadText(); JsonObject achievementModel = JsonValue.Parse(json) as JsonObject; return View(achievementModel); }

And here's what the view looks like:

@model dynamic <h2>@Model.Name</h2> <h3>@Model.Description</h3>

Okay, so far so good. Notice how the JSON gets to the view as dynamic and the view can party on it.

But, here's where things got tricky. What happens when I want to LINQ queries over the JSON and then pass it to my view? This is going to be a common scenario for the application, where the controller filters the model, if you will, before handing it off to the view.

For example, consider the following code:

var achievementsByCategory = from JsonValue v in achievementModel["Achievements"] group v by v["Category"].ReadAs<string>("") into grp select new { Category = grp.Key, Items = grp }; return View(achievementsByCategory);


Basically, I'm doing a group by clause on the object tree created in the deserialized JSON and creating an anonymous type, then returning that to the view. Nice idea, but MVC and anonymous types are not friends. Run this code and when you attempt to read the model and look at the properties , you'll get an exception of the like:

Microsoft.CSharp.RuntimeBinder.RuntimeBinderException was unhandled by user code -- 'object' does not contain a definition for 'Category'

So, what to do? One solution would be to create a typed object as part of the LINQ result and return that to the view. But that defeats the whole point of the architecture, which is to stay flexible, have no strict typing, but still getting the goodness of LINQ. What I realized is that I need to trick LINQ into returning a dynamic type, not an anonymous type. How to do this? Well, I realized that all the Json classes implement IDynamicMetaObjectProvider.

 Awesome! I could have LINQ return an IEnumerable of JsonObjects, which I could then cast as dynamic in my view and party on. So, the code looks like this:

var achievementsByCategory = from JsonValue v in achievementModel["Achievements"] group v by v["Category"].ReadAs<string>("") into grp select new JsonObject { {"Category", grp.Key}, {"Items", new JsonArray(grp)} }; return View(achievementsByCategory);

What's clever about this code is that my LINQ query is returning a set of JsonValue objects, which I can pass directly to the ctor of JsonArray.  And my view is happy, because it is getting something it knows how to handle: an IEnumerable. I can then walk the JsonObject results in the view, casting them back to dynamic and getting all the lovely property syntax:

@foreach (dynamic group in Model) { <h3>@group.Category</h3> <ul> @foreach (dynamic item in group.Items) { <li>@item.Name - @item.Description </li> } </ul> }
Pretty slick, methinks. My JSON can now change, morph, etc. and my code is rock solid. But at the same time, I get all the goodness of LINQ. And my front end Razor code ends up feeling more like Javascript than C#. It is the weird, trippy marriage of JSON and dynamic. I'm stoked about it. In this bizarre way, it takes the model out of MVC.

JSON Deserialization Made Simple With System.Web.Helpers and List<dynamic>

February 3 2011

Been playing with WebMatrix and have come across some crazy and beautiful code that I really liked in the FourSquare helper on CodePlex. Check this method out, which returns a graph of friends from the fourquare api

/// Returns a list of friends
    /// </summary>
    /// <param name="accessToken">The access token of the authenticating user.</param>
    /// <param name="userId">The Id of the person for whom to pull a friend graph. if not specified, the authenticating user's list of friends will be returned.</param>
    public static IList<dynamic> GetFriends(string accessToken, int userId = 0) {
        var url = "https://api.foursquare.com/v2/users/{0}/friends?oauth_token={1}";

        var client = new WebClient();
        var jsonResult = client.DownloadString(string.Format(url, userId == 0 ? "self" : userId.ToString(), accessToken));
        var result = Json.Decode(jsonResult);
        
        return new List<dynamic>(result.response.friends.items);
    }

Notice how tight this code is. And note the crazy dynamic typing that C#4 provides in combination with the Json.Decode method. If you are wondering where that JSON.Decode method came from, it is from System.Web.Helpers -- you can use this dll anywhere, doesn't have to be with WebMatrix or Razor. Here's a link to the docs on System.Web.Helpers.Json: http://msdn.microsoft.com/en-us/library/system.web.helpers.json(VS.99).aspx 

 Serialization and JSON have long been painful in .NET because of .NET's strong typing.  This got better with the C# dynamic keyword (see this post and this post) but nothing is as elegant as the code above.

Check this example out, which returns a user's FourSquare badges:

/// <summary>
    /// Returns badges for a given user.
    /// </summary>
    /// <param name="accessToken">The access token of the authenticating user</param>
    /// <param name="userId">The Id of the user</param>
    public static IList<dynamic> GetBadges(string accessToken, int userId = 0) {
        var url = "https://api.foursquare.com/v2/users/{0}/badges?oauth_token={1}";
        
        var client = new WebClient();
        var jsonResult = client.DownloadString(string.Format(url, userId == 0 ? "self" : userId.ToString(), accessToken));
        var result = Json.Decode(jsonResult);
        
        var badges = new List<dynamic>();
var groups = new List<dynamic>(result.response.sets.groups); var allBadgesGroup = groups.FirstOrDefault(g => g.type == "all"); if (allBadgesGroup != null) { foreach(var badgeId in allBadgesGroup.items) { badges.Add(result.response.badges[badgeId]); } } return badges; }

Notice how the two dynamic lists get created which are then manipulated with a LINQ lambda query.  Nice! Having dealt so often with either manually mapping JSON to CLR objects and/or dealing with arrays nested in arrays nested in arrays, I'm absolutely loving this.