Prototyping With Hive Using HDInsight

I’ve been doing a lot of prototyping with Hive lately and I really wanted to use an emulator to do the work.  For awhile, I was trying to do the prototyping using Powershell but kept getting NotSupported exceptions.  I finally decided to simply program against Hive itself which has turned out to work just great. I’m using Visual Studio to write my HQL as .sql files [Read More]

Connecting To The Azure Storage Emulator From The HDInsight Emulator

I was following the Getting Started instructions on using the HDInsight emulator and got stuck trying to connect to the Azure Storage Emulator:   hadoop fs -ls wasb://temp@storageemulator ls: `wasb://temp@storageemulator': No such file or directory Turns out that you must have a trailing slash, like this: hadoop fs -ls wasb://temp@storageemulator/ Maybe that’ll help someone out there… [Read More]

A Simple Box.Com C# API Wrapper

I had a need to access Box.com programmatically to do a daily pull of log files that were posted to Box from a third party service. At first I thought the Box .NET SDK would be helpful, but I quickly realized it is entirely oriented to be used by apps with a UI, not headless apps like this.  So, I dove into the documentation.  My first stumble [Read More]

Beware: Hadoop C# SDK Inserts Line Breaks, Tabs That Break Your Queries

After banging my head against the wall for many hours, I finally figured out that .NET is adding escaped carriage returns, aka \r\n when the queries are sent to HDInsight, which is causing the queries to fail. My code was loading the queries from files on disk like this: string query = string.Empty; using (var fs = new StreamReader("CreateTempTable.hql")) { query = fs.ReadToEnd(); } I figured this out [Read More]

Submitting HDInsight Jobs From An Azure Webjob or WorkerRole Using the C# Hadoop SDK

All the samples for submitting jobs programmatically to HDInsight assume that you are doing so from a desktop working station that has been set up with a management certificate. The code gets your cert out of the cert store and creates a JobSubmissionCertificateCredential as such: // Get the certificate object from certificate store using the friendly name to identify it X509Store store = new X509Store(); store.Open(OpenFlags.ReadOnly); X509Certificate2 cert = store. [Read More]