Table Storage, Continuation Tokens, Windows Azure

September 8 2011

By default, queries to table storage will return a maximum of 1000 rows, whether you are using the REST APIs or the LINQ provider.  If you want more rows, you need to get fancy.

It’s pretty easy.  Let’s say you have created some data access classes similar to how the Azure Hands On Labs and samples encourage you to do so. 

So say you’ve got a class that inherits from TableServiceContext that looks like this:

class RawDataContext : TableServiceContext { public RawDataContext (string baseAddress, StorageCredentials credentials) : base (baseAddress,credentials) { } public IQueryable<RawEntity> RawEntity { get { return this.CreateQuery<RawEntity>("RawEntity"); } } }

Well, you are going to want to change that, so your data context returns a CloudTableQuery instead.  So, now, your class should look like this:

class RawDataContext : TableServiceContext { public RawDataContext (string baseAddress, StorageCredentials credentials) : base (baseAddress,credentials) { } public CloudTableQuery<RawEntity> RawEntity { get { return this.CreateQuery<RawEntity>("RawEntity").AsTableServiceQuery(); } } }
Notice how the CreateQuery method has been appended with the AsTableServiceQuery() extension method. Notice also that the return type is CloudTableQuery instead of IQueryable.

But that’s not all. You’ll also need to make some changes when you issue your queries.  So, for example, say you have a DataSource class with a select query that looks like this:

public class RawDataSource { private static CloudStorageAccount storageAccount; private RawDataContext context; public RawDataSource() { storageAccount = CloudStorageAccount.FromConfigurationSetting("StorageConnection"); this.context = new RawDataContext(storageAccount.TableEndpoint.AbsoluteUri, storageAccount.Credentials); } public List<RawEntity> Select(string site) { var results = (from g in this.context.RawEntity where g.PartitionKey == site select g).ToList(); return results; } }

You’ll need to change that query such that it issues the Execute method off the CloudTableQuery result, like this:

public List<RawEntity> Select(string site) { var results = (from g in this.context.RawEntity.Execute() where g.PartitionKey == site select g).ToList(); return results; }

By adding the Execute method, the query will issue the continuation token on your behalf to Azure and concatenate the data to the result set, returning > 1000 rows.

Default Username and Password For A Cisco Router G Broadband Router WRT54G Firmware Version: v2.02.7

September 1 2011

If you are trying to connect to your Cisco Router G Broadband WRT54G, you’ve got to hit in your browser. You could hit it from this post even.  So that’ll take you to the UI, but first you’ll be prompted by a password.  If you never changed the defaults, they are:

Username: admin

Password: admin


That’s  for Cisco Router G Broadband Router WRT54G Firmware Version: v2.02.7.

Then, once you are in, you can change things like the security key., etc.

MVC3 and Windows Azure Tools 1.4 -- Better But Still Messed Up

August 24 2011

Getting MV3 on to Azure used to be really painful. The new Windows Azure Tools 1.4 make things better, but they are still a bit bungled up.  The tools fix the missing assemblies problem (although I still maintain that this shouldn't even be an issue and IT should have them installed on the servers, but that's another issue) in that if you create a new Azure project and choose ASP.NET MVC 3 Web Role, it will set all the assemblies you need to 'copy local.' Thanks guys. 

But, you're still going to have problems deploying to Azure, even you pick the empty template.  Your web.config will be full of things that can trip you up, the most egregious being a default session provider, which points to SQL Express, which no instance of Azure will ever have.  So, if you deploy without fixing that, you're screwed. Additionally, if you use the other providers that are in the web.config, you'll need to tweak them, as they rely on providers that Azure doesn't support.  

I'm calling out the folks who built these tools: this is sloppy. If you really want the gory details on this, you can read more Nathan Trotten's post.

While I'm ranting, if I choose an 'empty web template', why are 20 javascripts included with the project? This happens both with Azure MVC3 projects as well as non-Azure MV3 projects.  Empty means empty.

Oh, one other thing: if you create an MVC3 project not from the Azure Tools and then attempt to deploy it to Azure, you'll run into the missing assemblies problem.  Your life gets a little better if you have SP1 installed because you can invoke the Add Deployable Assemblies feature (explained here), but if you don't have SP1 installed, you'll have to do it manually.

Rudy Rucker on Programming and Communing With The Muse

August 17 2011

From a recent interview by Rudy Rucker:

Some textbooks make it sound as if software engineering is a formal process of making out lists of specifications, milestones, and the like, the process is also an experiential hands-on endeavor. You don’t ride a bicycle by making out lists of part numbers. You have to get on it and lurch around.

There are any number of ways to be creative in CS. In terms of theory, you might come up with a new higher-level way of thinking about computations. In terms of practice, deciding what kinds of programs to create can be creative. Much creativity (and low cunning) comes into play in finding ways to make one’s programs run faster. And designing a program’s interface is an artful process as well.

Just as is the case for writers and painters, programmers will often find that their projects are mutating while they work on them. Certain pathways close up, and newer opportunities emerge. At some point it can feel as everything you see all day long is in some way part of the creative process—as if everything is helping you to get the project done. This is known as communing with the Muse. And, make no mistake, there is a Muse of programming as well as there are Muses of writing and painting.

I'm currently reading Jim and the Flims and, as usual, find myself caught up in the bizarre logic of Rudy's imagination. Highly recommended.

How To Suppress The Gravatar Default Icon

August 4 2011

Gravatar is a nifty service. But if someone on your site doesn’t have a Gravatar, Gravatar will display a default icon.  You have some choices in what gets displayed (documented here), but what if you want nothing displayed?  Well, you can have Gravatar return you a 404, but you need to handle that 404 somehow. Not a big deal, but here’s how I did it.

First off, I started by using this code, which is an ASP.NET control that encapsulates all the Gravatar logic. Then, I mucked around with the override of Render, adding the following:

if (!string.IsNullOrEmpty(DefaultImage))
    imageUrl += "&default=404"; // +HttpUtility.UrlEncode(DefaultImage);
using (WebClient client = new WebClient())

    catch (WebException webException)
        noImage = true;


if (noImage)


Yes, it means I have to make a call to Gravatar in order to determine to not provide an image, but it is worth it. If performance is an issue, an optimization would be to cache the emails that have already been checked. In fact, I’ll probably add that.  No more default gravatars!

DebuggerNonUserCodeAttribute Saves The Day

July 14 2011

Sometimes Visual Studio is too smart. Hit a situation yesterday where I has a solution with an .exe and a .dll. In the .dll, there's a method called by the .exe that throws an exception.  I needed to test the logic of how the .exe handled that exception. But, while debugging, the project would always break on the throw in the .dll and I couldn't ever get it to continue on to test my logic in the .exe. 

Finally, I got it to bypass the class in the .dll by attibuting the class dll with the DebuggerNonUserCodeAttribute.  Walla! Now I could check the logic in the .exe when the exception was thrown.

MVC3 and Azure -- Back To Classic Dev Versus IT

June 13 2011

Ugh, the evil "Instance 0 of role is initializing" when deploying to Azure. Turns out MVC3 isn't installed on the Azure VMs.  The issue is the same as getting PHP installed into Azure: the Azure virtual machines don't have MVC3 installed by default, so it is up to you to get it on the box. Beware -- just changing the references to MVC3 in your project to "Copy Local" = true will not do the trick, as the MVC references rely on additional assemblies.  So, you have to manually add these assemblies:

  • Microsoft.Web.Infrastructure
  • System.Web.Helpers
  • System.Web.Mvc
  • System.Web.Razor
  • System.Web.WebPages
  • System.Web.WebPages.Deployment
  • System.Web.WebPages.Razor

Which can be found in the C:\Program Files\Microsoft ASP.NET\ASP.NET Web Pages\v1.0\Assemblies\ directory.

The other options are to use the WebPI command line tool to install MVC for you, which is the same idea as my PHP deployment solution.  O, you can send up the MVC installer with your website.  I'm not too keen on that solution, as it means deploying an additional 18MB with each deployment.

Steve Marx goes into all of this more in this post.

I don't get why they can't provision the Azure VMs with the latest/greatest shipping Microsoft runtimes and assemblies to avoid this problem, just as I think the Azure VMs should be provisioned with PHP by default.  That's what you can expect from the top tier ISPs so why not Azure? I suppose the counter argument is that the VMs get bloated with runtimes that aren't used. But the cost is only disk space. Really, I think it comes down to classic IT vs. Dev conflict. I expect IT to have boxes provisioned; that's their job.  But Azure makes it my job as a developer.  And they probably think the exact opposite.


June 1 2011

Supporting JSON-P with MVC is easy. Just derive a class from ActionResult, like this:

public class JsonpResult : ActionResult { public string Callback { get; set; } public string Content { get; set; } public override void ExecuteResult(ControllerContext context) { context.HttpContext.Response.ContentType = &quot;application/json&quot;; if (this.Callback != null) { context.HttpContext.Response.Write(string.Format(&quot;{0}({1})&quot;, this.Callback, this.Content)); } else { context.HttpContext.Response.Write(this.Content); } } }

Then, in your controller, do the following, assuming you are looking for a querystring called callback (the json-p) convention as a method parameter to your action, and assuming that you have a string of json to send back, return this:

if (callback != null) return new JsonpResult { Content = json, Callback = callback};

Note that I'm not having MVC handle serialization of my objects to JSON. I'm doing that all myself using the JSON classes from Microsoft.Serialization.Json from this project:

Taking The M out of MVC - JSON, The dynamic Keyword, LINQ

May 25 2011

Okay, so I’ve been working on a new MVC project in all the data is stored as JSON. The plan is to have JSON deserialized in the controller (using the WCF JSON classes) and then pass the deserialized JSON to the view. The general thinking is that we won't have any models, just controllers and views that work with dynamically created object graphs from JSON.

So, I started implementing.  Works great. For example, here’s controller code that grabs some JSON out of blob storage, deserializes it, and returns it to the view:

public ActionResult Index(string name) { ICloudBlobContainer container = cloudBlobClient.GetContainerReference("applications"); ICloudBlob blob = container.GetBlobReference(string.Format("{0}.txt",name)); string json = blob.DownloadText(); JsonObject achievementModel = JsonValue.Parse(json) as JsonObject; return View(achievementModel); }

And here's what the view looks like:

@model dynamic <h2>@Model.Name</h2> <h3>@Model.Description</h3>

Okay, so far so good. Notice how the JSON gets to the view as dynamic and the view can party on it.

But, here's where things got tricky. What happens when I want to LINQ queries over the JSON and then pass it to my view? This is going to be a common scenario for the application, where the controller filters the model, if you will, before handing it off to the view.

For example, consider the following code:

var achievementsByCategory = from JsonValue v in achievementModel["Achievements"] group v by v["Category"].ReadAs<string>("") into grp select new { Category = grp.Key, Items = grp }; return View(achievementsByCategory);

Basically, I'm doing a group by clause on the object tree created in the deserialized JSON and creating an anonymous type, then returning that to the view. Nice idea, but MVC and anonymous types are not friends. Run this code and when you attempt to read the model and look at the properties , you'll get an exception of the like:

Microsoft.CSharp.RuntimeBinder.RuntimeBinderException was unhandled by user code -- 'object' does not contain a definition for 'Category'

So, what to do? One solution would be to create a typed object as part of the LINQ result and return that to the view. But that defeats the whole point of the architecture, which is to stay flexible, have no strict typing, but still getting the goodness of LINQ. What I realized is that I need to trick LINQ into returning a dynamic type, not an anonymous type. How to do this? Well, I realized that all the Json classes implement IDynamicMetaObjectProvider.

 Awesome! I could have LINQ return an IEnumerable of JsonObjects, which I could then cast as dynamic in my view and party on. So, the code looks like this:

var achievementsByCategory = from JsonValue v in achievementModel["Achievements"] group v by v["Category"].ReadAs<string>("") into grp select new JsonObject { {"Category", grp.Key}, {"Items", new JsonArray(grp)} }; return View(achievementsByCategory);

What's clever about this code is that my LINQ query is returning a set of JsonValue objects, which I can pass directly to the ctor of JsonArray.  And my view is happy, because it is getting something it knows how to handle: an IEnumerable. I can then walk the JsonObject results in the view, casting them back to dynamic and getting all the lovely property syntax:

@foreach (dynamic group in Model) { <h3>@group.Category</h3> <ul> @foreach (dynamic item in group.Items) { <li>@item.Name - @item.Description </li> } </ul> }
Pretty slick, methinks. My JSON can now change, morph, etc. and my code is rock solid. But at the same time, I get all the goodness of LINQ. And my front end Razor code ends up feeling more like Javascript than C#. It is the weird, trippy marriage of JSON and dynamic. I'm stoked about it. In this bizarre way, it takes the model out of MVC.

More On PHP Deployment To Windows Azure

May 12 2011

Just got hip to this post by Maarten Balliauw, who did something very similar to what I did for doing PHP deployment to Azure. He adds a few extra goodies (automated .bat files, etc.) and goes into a little more detail, so definitely worth reading.

He also calls out that there is more than one way to skin this cat, which Jas Sandu called out as a comment to my blog post, directing folks to the PHP Commandline tools for Azure. There's a few differences between the methodologies.  The PHP Commandline tools assume you are building/deploying your Azure PHP package from the same box that you are developing your PHP application on.  It picks up your instance of PHP -- and all necessary modules that you may require -- right from the box.  The other big difference is that the PHP command line tools install PHP using the web.roleconfig methodology, pointing the <fastCgi> application element to the php-cgi-exe.  This is different from the way Maarten and I do it, which uses the Web Platform Installer to install PHP and then installs any modules using a custom script.  You'll notice that Maarten and my method never actually references the locaiton of PHP in the web.config or web.roleconfig.

Boy, wouldn't this all be easier if the Azure folks would just provision PHP to any and all of the Azure VMs?