Azure real time analytics

I have been thinking of exploring Azure PAAS offering in Data warehousing/analytics for a while and finally decided to go ahead with 2-3 examples that will help me cover most of the PAAS offering.

In the first example, I am using Event hubs and see if i can try some real time analytics there using azure stream analytics jobs. I assumed that I am sitting in a big mall where thermometers keep sending temperature readings frequently to Azure Event Hubs. Then i am using this data to check temperature every 2 minutes and see if there is no drop/rise in data that indicate some issue with cooling system/heaters.

I started off by provisioning a event hub with 4 partitions in Azure.s1

Then I made a code in c# which sends seed data to this Azure Event Hub in the same pattern a real thermometer/thermostat would do.s2.jpg

Data was send in JSON format. Here is the example of data send.

s3

Once I am sure data is being received by Events Hubs, I want to do some real time stream analytics and make sure that i get updates for every few minutes for each device.

For Stream Analytics in this example, I Went ahead with azure PAAS offering, Stream Analytics Jobs.

s4

Every Stream Analytics has 3 important setups. Input, thats where our real time data will be picked from (here an Event Hub). Output, where the results of analysis will be saved, (here Power BI) and a query/function which manipulates input data.

See below on how we have setup these 3 in stream analysis.

On the Event Hub Dashboard in portal click input and add these details to attach it to an event hub for input.

s5.jpg

Click on test to make sure input endpoint is valid. Then go back to Event Hub Portal and click Output. And then add Power BI as output.

s6

In this case we will setup a query to check Min, Max and Average Temperature every 2 minutes and publish results on power bi so that end user can check if temperature stats are ok.

See below for query –

s7

Save the Query and Then go back to Event Hub Portal. Start the Stream Analytics Service and Check The graph to make sure Input is processed and Output is generated.

s8.jpg

Once Output is generated, go to Power BI and see if dataset is created for this exercise and if its getting refreshed.

You should see a new Dataset created.

s9.jpg

We can now use this dataset to created any report. I have made a simple table to show data produced every 2 minutes.

s11.jpg

Get going with ExceptionLess

The other day, I was trying to come up with a centralized, real time error and log reporting solution and came across this https://exceptionless.com/. On some r&d I found out it to be a perfect in premise free log management tool that goes really well with entire .NET stack and javascript.

There is an online version of same provided by ExceptionLess, but i am going to describe the process for self hosting below.
ExceptionLess has 2 key components :
API to receive logs, messages and events. The core is written in .NET as WEB API and supported by mangoDB, redis and ElasticSearch in backend.
Second is a dashboard application written in angular.js that makes calls to the API and provides a decent enough dashboard to browse logs.

Each project has its own API key so we can be sure that the logs are separate for each project/module and dashboard has enough filters to separate then out. So we can have logs of different projects with proper access control on the same dashboard.
At the time of this blog, I used version 2.1 and the notes below are based on that. Version 3.0 will be out soon and make internal hosting better. For those who do not want internal hosting, I recommend using ExceptionLess hosting. more details here : http://exceptionless.com/pricing

The installation is really straight forward as with any .NET site.
– install mangoDB, redis server, ElasticSearch and IIS url rewrite on the server.

Download the API code from here. https://github.com/exceptionless/Exceptionless
then rebuild/publish the API on IIS.
Update the connection strings in the Web.config file to point to your Elasticsearch and Redis servers. If you are doing everything on same server then standard port are 6379 for redis and 9200 for elasticSearch.

Update the app settings (BaseURL, EnableSSL, WebsiteMode, etc..) in the Web.config file. For me these were http://serverDNS or http://serverIP , SSL = false and mode = “Production”

Update the mail settings in the Web.config file. I was using my own SMTP server so I updated it to this and all the emails worked fine.
<mailSettings>
<smtp from=”no-reply@no-reply.com”>
<network host=”smtp.live.com” password=”mypasswordgoeshere” port=”587″ userName=”MyEmail@live.co.uk” enableSsl=”true”/>
</smtp>
</mailSettings>

Update the machineKey in web.config file. You can use this link to generate a new one. http://www.a2zmenu.com/utility/Machine-Key-Generator.aspx

Update the ‘BaseURL’ to the UI url I am setting up next.

To install the angular frontend using the release here :
https://github.com/exceptionless/Exceptionless.UI/releases

Update the app.config.*.js file with your settings. the key here is the BaseUrl. This should point out to the API url.

The first user I signed as, became the UI admin.
So my final API layer web.config is like this :

<connectionStrings>
<add name=”RedisConnectionString” connectionString=”127.0.0.1:6379″ />
<add name=”ElasticSearchConnectionString” connectionString=”http://localhost:9200&#8243; />
</connectionStrings>
<appSettings>
<!– Base url for the ui used to build links in emails and other places. –>
<add key=”BaseURL” value=”http://xxx.xxx.xxx.xxx:8081/&#8221; />
<!– Controls whether SSL is required. Only enable this if you have SSL configured. –>
<add key=”EnableSSL” value=”false” />
<!–
Dev: Use this mode when debugging. (Outbound emails restricted)
QA: Use this mode when deployed to staging. (Outbound emails restricted)
Production: Use this mode when deployed to production.
–>
<add key=”WebsiteMode” value=”Production” />

and app.config file on the UI layer is like this :
(function () {
“use strict”;

angular.module(‘app.config’, [])

.constant(‘BASE_URL’, ‘http://xxx.xxx.xxx.xxx/api/v2&#8217;)

.constant(‘FACEBOOK_APPID’, ”)

.constant(‘GITHUB_APPID’, ”)

.constant(‘GOOGLE_APPID’, ”)

.constant(‘INTERCOM_APPID’, ”)

.constant(‘LIVE_APPID’, ”)

.constant(‘STRIPE_PUBLISHABLE_KEY’, ”)

.constant(‘SYSTEM_NOTIFICATION_MESSAGE’, ”)

.constant(‘USE_HTML5_MODE’, true)

.constant(‘USE_SSL’, false)

.constant(‘VERSION’, ‘2.0.441’)

;
}());
you can also enable API layer debugging by using nLog.
to do that i just updated my web.config with this setting :
<configSections>
<section name=”exceptionless” type=”Exceptionless.ExceptionlessSection, Exceptionless.Extras” />
<section name=”nlog” type=”NLog.Config.ConfigSectionHandler, NLog”/>
</configSections>
<nlog xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance”&gt;
<targets>
<!– Daily log files, deleted after 7 days –>
<target name=”Search.Web” xsi:type=”File” fileName=”C:\inetpub\wwwroot\errors.web.log” archiveFileName=”C:\inetpub\wwwroot\errors.web.{#}.log” archiveEvery=”Day” archiveNumbering=”Rolling” maxArchiveFiles=”7″ />
</targets>
<rules>
<logger name=”*” minlevel=”Info” writeTo=”Search.Web” />
</rules>
</nlog>

Then there are a lot of samples to make a call to the api and see results on dashboard.

https://github.com/exceptionless/Exceptionless.Net

but in summary you have to do the settings as per below –

Install ExceptionLess using Nuget package manager.

Then update config file

<exceptionless apiKey=”YOUR_API_KEY” serverUrl=”http://localhost&#8221; enableSSL=”false” />

Or

using Exceptionless.Configuration;
[assembly: Exceptionless(“YOUR_API_KEY”, ServerUrl = “http://localhost&#8221;, EnableSSL = false)]

API_KEY here is provided to you when you create a new project on UI.

and then make calls to the API to submit logs,events or messages.