coding

inotify Instances .Net Core on Linux

Now you get the exception "The configured user limit (128) on the number of inotify instances has been reached" and you wonder what the heck is going on.  For me I never got this exception on my local instance of Docker running on my Mac or in Windows but I would see my app just stop running in Linux.  Even weirder I could run the app fine in Windows hosted by Kestrel.  So where does this error start coming from?  

After much digging, I found the problem was with the IConfiguration interface.  Specificially, I was calling pulling the JSON configuration files multiple times which was reloading the IConfiguration interface.  The issue was that it was rebuilding the configuration and re-attaching the "reloadOnChange" setting.  Since the reloadOnChange was set to true it was attaching a new file watch multiple times and was hitting the limit.  That all being said, I never saw this issue locally on my Mac or Windows or my local instances of Docker on either which I think is related to the way the underlying Docker Linux host manages the watch on the files.  

Something else as a side note, since the files were being tracked, even though I was disposing of the class it wasn't releasing the watch and thus keeping these connections open.  The simple fix was to set the reloadOnChange to false but to make it even better was to just create a singleton of the data and thus prevent the data from being ever reloaded.  I don't think it is necessarily a bug with .Net but it is a weird side effect of how the host can cause problems with a framework.  

I hope this helps some people save some time that I lost dealing with this crazy issue.

 

Sending Mass Emails in C#

I like many developers am tasked with creating batch emails to send to a large population of users.  And for this I am not talking about spam, I am referring to sending emails to a large poplulation within an organization such as annual notification reminders or custom links to some application.  I have tried sending these emails synchroniously and if you are sending a few hundred that isn't a problem, but when I started sending several thousand, I was noticing a bottleneck with my SMTP server and waiting for the callback showing the SMTP server receved the message.  

So moving forward from that thought the best option for sending mass emails greater than a few hundred is using threading and asynchronious sending.  I have tried this a few different ways in the past with event listeners and different techniques, but moving to the async and await world of modern .Net I thought I would re-write it with some more modern tooling.  Since this took me a little trial and error I thought it would be a good thing to post for my fellow developers out there. 

From a performance standpoint using the same SMTP server I was getting about three emails per second with standard synchronous methods and when I went to the attached code, I went up to about 18 (running locally over VPN to our corporate offices) and I will expect to see an even bigger jump on a VM in the datacenter.  Overall there isn't much impact to performance on the machine with minimal CPU and RAM overhead.  

Anyway, click here to get to my code.  

Two things I want to point out:

  1. I know I have a weird try / catch in there.  That is because the mail server was choking with that many emails going through at one time and I was getting random errors.  Retrying one time usually fixed it.
  2. I know there are async methods with the SMTP client, but when you use that it still only allows one email to be processed at a time by the general SMTP client thus making it an async method but not async in the since that I wanted which was mass emails with threading.

Yammer OAuth for data export

Anyone familiar with Yammer will know that many of the APIs are a bit lacking.  Also, all API connections will need an oAuth token that has access to the data in question.  So what do you do if you have a console application that needs to use Yammer Data Export API and drop the into a database or more to the point, how do you get the oAuth token?  I am not going to post the specifics on how to get the data and extract it but I will post some code on how to get the oAuth token.

To start there is some information you will need from Yammer which is only obtained after creating a Yammer app (https://developer.yammer.com/v1.0/docs/app-registration).  Your redirect URL for the included code needs to be http://localhost:80/Temporary_Listen_Addresses/  (if you get an error relating to access denied the port may be being used or it could be an issue with your privileges on the device).  *NOTE you do not need to publish your app for this functionality to work, so you can create the app then never finalize it.  Yammer's documentation https://developer.yammer.com/v1.0/docs/authentication-1 does not include the specifics on how to implement this on a console application and only includes the basic information.

The included code makse the assumptions that you will have the following app seetings in your App.config: "ClientID", "RedirectURL", "ClientSecret", and "oAuthToken" with all of these fields populated, with the exception of the "oAuthToken" field, from the information in your Yammer app settings.

I have included the complete class to perform the login here and to implement it you just need to call the constructor. 

  1. var yammerLogin = new YammerLogin.YammerAuthentication();
  2. string token = yammerLogin.oAuthToken;

The oAuthToken will be stored in the App.config and it will only re-auth if the token has failed to login. 

I hope this will help save a few hours of work for a few people out there.

Migrating Drupal from MySQL to MSSQL

So I am going to start by ranting a little bit about Microsoft Azure cloud services.  Something odd about Azure is that when you are creating a new Drupal or WordPress site it will automatically crate a MySQL database but if there is already a MySQL database it will not create it automatically.  You can however create a MySQL database then install your CMS and associate it with said database.  The second part of my rant is that with a MSSQL database the smallest size is 2 GB and it is less expensive than the 20 MB MySQL database, which just seems odd to me.

 

Okay, now that I have that over with, let me get to the heart of what I wanted to talk about and discuss installing Drupal on a MSSQL and migrating (if need be) from MySQL in Azure.  So, to start there is one very easy config change to allow Drupal to run on MSSQL.  Download the drivers from https://www.drupal.org/project/sqlsrv then extract the files.  With the files extracted copy the sqlsrv folder and drop it in the includes/databases directory in the root of your Drupal installation.  After the files are uploaded update the configuration settings in the settings.php file and update the connection for the $databases to have the driver “'driver' => 'sqlsrv'”.  Something to note, if you are performing this on Azure the PHP MSQL Drivers are already installed but if you are running this install on a different system you may need to install PHP MSSQL drivers.

 

One thing that was missing on my installation was a SUBSTRING function that is used for Drupal when performing a post.  The site worked fine but when I tried to post as an authenticated user I received an error.  To solve the problem I needed to add the function.  Click Here for the specific TSQL code.  (UPDATE)Even with the substring SQL function installed I was still having trouble.  So I ended up hitting the install.php page and then running a database update.  This seems to have fixed my missing SQL functions, but only time will tell if I am not missing anymore.  I have included all of the functions that are created in the link referenced in this paragaraph.

 

So if you need to migrate from MySQL to MSSQL there are a few steps that you need to take.  After creating the new database in MSSQL download the MSSQL Migration Assistant for MySQL from http://www.microsoft.com/en-us/download/details.aspx?id=42657.  Point the MySQL to your old MySQL instance and point the MSSQL to the new MSSQL instance.  Start by changing the target schema to dbo (this will make the migration easy).  Then right click on your database and click “Convert Schema” this will create the local metadata of the schema.  Once the schema conversion is done, on the target database right click and select “Synchronize with Database.”  Once the synchronization is complete (this will take a while) your tables will be created but without data.  Lastly right click on your original MySQL database and select “Migrate Data.” 

 

If you have not updated your connection in your settings.php file, you can update that now to point to the MSSQL instance and you should be good.  I tested my change by making a small change to my site and seeing how it reflected in the database (specifically I changed my locale on my user account). 

 

I hope this will help the few that run Azure sites under your MSDN and do not want to pay for additional MySQL storage.