Introduction
A screen grab of the monitoring control whilst monitoring a process
Often when creating a WebApplication, WebService, Website or any Internet based system, you will need to perform tasks that take a long time to run. This may be processing uploaded files or processing data from a database. If the execute time for the task takes longer than 10 seconds to complete, you are asking a lot of your users expecting them to wait for the process to complete and the page to reload. Making people wait just doesn’t cut it.
I recently had to write an app that processed large amounts of data from uploaded CSV text files, multiple MySQL databases and a MS SQL Server database. The data needed to be filtered and sorted, then the subset of data needed to a be submitted to a WebService. All this had to be controlled from an aspx page, and the processes also needed the ability to be run by users at any time (not just one off tasks). In addition I wanted the ability to cancel the operations at any point in time.
Due to the large amounts of data being processed, and the transport overheads grabbing the data from multiple sources, the execution time of the operations was considerable, even after optimization of the sorting and filtering algorithms.
To solve the problem of keeping the user informed of the progress of each operation and providing them with information on totals, error counts, the number of duplicates etc.. I decided to build an asynchronous, threaded worker process manager and an UpdatePanel control to monitor the progress of the working processes. I wanted some code that would allow me to reuse the classes in other projects and an architecture that makes it easy to add worker process classes to the system as needed in the future. It turned out to be an interesting exercise. So I thought I would share my code so that others can use it as well.
I looked around for free controls and libraries that would serve the purpose but I found nothing that fit the bill. Well… now my code is out there.
Technology Overview
The code for this project is written in C#. I’m using ASP .NET and Microsoft Visual Studio 2008 running on version 3.5 of the .NET framework. I am using Linq because I like being able to use List<> object collections. The code I am providing is like scaffolding, or a shell that you can build on. If you post questions in the comments I’ll do my best to answer them for you.
How to use the code
[the file is hosted on mediafire.com]
Source code: AsyncProcessor.zip
To implement my code in your project, you’ll need to download the source, extract the contents and include the files in your project. The project will run on your system if you have everything set up to run Visual Studio 2008 WebApplications and .NET 3.5. I’ve used static classes where possible and no namespaces so the classes should be good to go in your project once they are included. You’ll need to change the namespace on the aspx page and the ascx control though. The code is commented to a certain degree, and I’ve highlighted where you should add your code to do the work that your threaded worker processes need to do. I’ve included a basic CSS file for the example, however I have not spent time on making the example project look great. I am providing this code “as-is”. I have tested it. However I am not responsible for your implementation of it. Use it at your own risk.
Important files
Now onto the code. For the sake of brevity I am not going to include the entire source here for all classes. I will explain the main functions in the important classes. I am assuming you have some knowledge of coding in C# and you understand the concept of Threads. To view the complete class files you’ll need to download the source.
AsyncProcessorDetail.cs
This is the class that tells the AsyncProcessManager what it needs to do. It is used to store the details of the worker process. You’ll need to create one of these and pass it to the AsyncProcessManager.StartAsyncProcess method to kick off a new threaded process. This class is used to store the status of the running process and it keeps track of the running totals and counts. The worker process updates this class when it’s running and the asyncProcessingMonitor control reads this class before it displays the current status of the running process to the user.
The constructor is discussed in more detail in the Default.aspx.cs section a bit later because that’s where the object is instantiated.
AsyncProcessorDetail.cs
using System;
public class AsyncProcessorDetail
{
// types, identifiers and control properties
public int ID { get; set; }
public string ProcessorType { get; set; }
public bool Process { get; set; }
public bool Processing { get; set; }
public bool Stop { get; set; }
public bool Complete { get; set; }
public bool Test { get; set; }
// properties for storing counts
public int Total { get; set; }
public int Processed { get; set; }
public int ErrorCount { get; set; }
public int Duplicates { get; set; }
// Status properties
public int RefreshStatusCount { get; set; }
public string StatusText { get; set; }
// properties for dealing with files
public string FullFilePath { get; set; }
public string FileName { get; set; }
//Timing Information
public DateTime StartTime { get; set; }
public DateTime EndTime { get; set; }
public AsyncProcessorDetail()
{
}
public AsyncProcessorDetail(int id, string full_file_path, string filename, int total, string processor_type, int refresh_status_count)
{
Process = true;
Processing = false;
Stop = false;
Complete = false;
RefreshStatusCount = refresh_status_count;
ID = id;
FullFilePath = full_file_path;
FileName = filename;
Total = total;
ProcessorType = processor_type;
}
public void Begin()
{
StartTime = DateTime.Now;
Process = false;
Processing = true;
}
public void UpdateCounts(int processed, int error_count, int duplicates, int total)
{
Processed = processed;
ErrorCount = error_count;
Duplicates = duplicates;
Total = total;
}
public void UpdateStatusText(string status_text)
{
if (StatusText == null)
StatusText = "";
if (StatusText.Length > 0)
StatusText += "<br />" + DateTime.Now.ToLongTimeString() + ": " + status_text;
else
StatusText = DateTime.Now.ToLongTimeString() + ": " + status_text;
}
public void End()
{
Stop = true;
Processing = false;
Complete = true;
EndTime = DateTime.Now;
TimeSpan.FromTicks(EndTime.Ticks - StartTime.Ticks).TotalSeconds.ToString();
UpdateStatusText("Processing completed in "
+ TimeSpan.FromTicks(EndTime.Ticks - StartTime.Ticks).TotalSeconds.ToString()
+ " seconds");
}
}
AsyncProcessManager.cs
This is the main class that manages the worker processes and starts the worker threads. It contains a static List<> of worker AsyncProcessorDetail objects. It contains methods and functions that are exposed to the pages in your WebApplication and the asyncProcessingMonitor control. It also contains the main AsyncProcessorThread class that instantiates the worker processor objects.
Firstly we have a List to store the AsyncProcessorDetail objects. The List is used to store all the processes that are currently running or are complete.
public static List<AsyncProcessorDetail> ProcDetails { get; set; }
The StartAsyncProcess() method is called by Default.aspx.cs or one of your pages to initiate processing. It first checks to see if another process with the same ID is running. If there is another process running with the same ID this function will return a 1, otherwise it will add the new AsyncProcessorDetail to the List and start a new Thread by calling the StartProcessorThread() method and return 0 for success.
public static int StartAsyncProcess(AsyncProcessorDetail input_detail)
{
if (ProcDetails == null)
ProcDetails = new List<AsyncProcessorDetail>();
bool error = false;
foreach (AsyncProcessorDetail detail in ProcDetails)
{
if (detail.ID == input_detail.ID)
{
// task is already being processed
if (detail.Processing)
{
error = true;
}
}
}
if (!error)
{
// removing any existing instances of this detail in the list
ProcDetails.RemoveAll(z => z.ID == input_detail.ID);
// Add the detail to the list
ProcDetails.Add(input_detail);
// Start processor thread
Thread processorThread = new Thread(new ThreadStart(StartProcessorThread));
processorThread.IsBackground = false;
processorThread.Start();
return 0;
}
else
return 1;
}
Next we have the AsyncProcessorThread class itself. This class consists of one public constructor: AsyncProcessorThread(). This grabs the next AsynchProcessorDetail object from the List that has a Process value of true. It then instantiates a new worker process class based on the ProcessorType value that was set when the AsynchProcessorDetail was created.
public class AsyncProcessorThread
{
// Main processor thread
public int ID { get; set; }
public AsyncProcessorThread()
{
try
{
ID = GetNextIDToProcess();
ProcDetails.SingleOrDefault(z => z.ID == ID).Begin();
AsyncProcessorDetail fd = ProcDetails.SingleOrDefault(z => z.ID == ID);
// Add a case statement for each of the processor classes you create
switch (fd.ProcessorType)
{
case "ProcessorTemplate":
ProcessorTemplate processorTemplate = new ProcessorTemplate(fd);
break;
case "ProcessorExample":
ProcessorExample processorExample = new ProcessorExample(fd);
break;
default:
ProcDetails.SingleOrDefault(z => z.ID == ID).StatusText = "No ProcessorType found. Exiting..";
ProcDetails.SingleOrDefault(z => z.ID == ID).End();
break;
}
}
catch (Exception ex) { }
}
}
The following methods are utility methods that start, stop and finalize the running process. There is also the Continue function that the worker class will call at set intervals based on the AsyncProcessorDetail.RefreshStatusCount to determine if it should continue processing or not.
private static void StartProcessorThread()
{
AsyncProcessorThread processor = new AsyncProcessorThread();
}
private static void UpdateCounts(int file_id, int processed, int error_count, int duplicates, int total)
{
ProcDetails.SingleOrDefault(z => z.ID == file_id).UpdateCounts(processed, error_count, duplicates, total);
}
private static int GetNextIDToProcess()
{
return ProcDetails.SingleOrDefault(z => z.Process == true).ID;
}
public static AsyncProcessorDetail GetProcessorDetail(int file_id)
{
try
{
return ProcDetails.SingleOrDefault(z => z.ID == file_id);
}
catch (Exception ex)
{
return null;
}
}
public static bool Continue(int id)
{
return !ProcDetails.SingleOrDefault(z => z.ID == id).Stop;
}
public static void StopProcessor(int id)
{
ProcDetails.SingleOrDefault(z => z.ID == id).Stop = true;
ProcDetails.SingleOrDefault(z => z.ID == id).Processing = false;
}
public static void FinalizeProcess(int id)
{
ProcDetails.SingleOrDefault(z => z.ID == id).End();
}
Default.aspx.cs
Default.aspx before monitoring begins
The most important method in the Default.aspx.cs class is StartAsynchProcess(). This is called from the start button click event handler. You will notice that this method creates a new AsyncProcessorDetail object. The constructor takes in the following values:
- int id – A unique id for the process. In the example I have used a random number however in a real implementation I insert the details of the process into a database and use a unique id from the inserted row.
- string full_file_path – If you are processing files this should be the full path to the file. The worker process can then use this to grab the file.
- string filename – The filename if you are processing a file.
- int total – If you know the total number of iterations your process will complete at this point pass the value in. In practice though the total is normally determined by the processor class and is not known before hand.
- string processor_type – This is a string that identifies what processor class you want to use. The AsyncProcessorThread class uses this string to determine which class to instantiate to begin processing.
- int refresh_status_count = This int determines how often the running process should update the status of the AsyncProcessorDetail for display to the user. For example if your process has to run through around 10 iterations you would probably set this to 1, however if you are expecting large files to be processed with say 100,000 lines in the file, you might want to set this to 1000. Therefore in the case of a large file, the process would only update the progress counts on every 1,000th iteration.
The StartAsynchProcess() method calls the StartAsyncProcess() method on the static AsyncProcessManager and passes it the created detail which in turn kicks off the new thread and processing will begin.
The last line of code in this method is the function call on the asyncProcessingMonitor custom control. It gets passed the ID of the process so it knows what to monitor and two other variables that tell the monitor control what to do when it is finished monitoring the process: redirectWhenComplete and the redirectURL. These are discussed in more detail later.
Default.aspx.cs
private void StartAsynchProcess()
{
// generate a sudo random number for this process
// I suggest adding the process to a database and using the unique id
// generated. that way you have a record of the process running
int seed = int.Parse(DateTime.Now.Ticks.ToString().Substring(0, 9));
int unique_proc_id = new Random(seed).Next();
// create a new AsyncProcessorDetail
AsyncProcessorDetail detail = new AsyncProcessorDetail(unique_proc_id, "c:\\path\\to\\your\\file", "filename", 0, "ProcessorExample", 1);
// call the AsyncProcessManager and pass it the AsyncProcessorDetail to be processed
AsyncProcessManager.StartAsyncProcess(detail);
// start the asyncProcessingMonitor control
asyncProcessingMonitor1.Start(unique_proc_id, true, "Default.aspx");
}
ProcessorExample.cs
I’ve included two processor classes in the example project. This ProcessorExample class is really simple but it illistrates how to use the template class provided. Your implementations of these classes will no doubt me much more complex. These classes are where the actual work is done. Examples are reading from a file, grabbing data from a database, calling webservices, updating other systems, basically any task that needs to be executed by a seperate process to the UI.
This class provides an example of calling the UpdateStatusText method to provide feedback via the asyncProcessingMonitor control.
In my example I simply create a loop that counts to ten. On each iteration the processor updates the AsyncProcessorDetail counts and sleeps for 1 second.
The Update() function calculates wether or not to update the counts or check if the process has been canceled. You should always include the Update function call and a Continue check in for, foreach or while loops so that processing can be canceled if the user clicks the Cancel button on the monitoring control.
ProcessorExample.cs
using System;
using System.Net;
using System.IO;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
public class ProcessorExample
{
// This class is an example of how to use the ProcessorTemplate.
// This class will simply count to ten in ten seconds
private AsyncProcessorDetail ProcessorDetail;
public ProcessorExample(AsyncProcessorDetail processor_detail)
{
ProcessorDetail = processor_detail;
Process();
}
public void Process()
{
AsyncProcessManager.GetProcessorDetail(ProcessorDetail.ID).UpdateStatusText("Processing has started");
try
{
int rowcount = 0;
int duplicates = 0;
int errorCount = 0;
int processed = 0;
int total = 0;
total = 10;
AsyncProcessManager.GetProcessorDetail(ProcessorDetail.ID).UpdateStatusText("Commencing processing loop");
while (true)
{
rowcount++;
processed++;
if (UpdateStatus(processed, total))
{
AsyncProcessManager.GetProcessorDetail(ProcessorDetail.ID).UpdateCounts(processed, errorCount, duplicates, total);
if (!AsyncProcessManager.Continue(ProcessorDetail.ID))
break;
}
if (processed == total)
break;
// sleep for a second
Thread.Sleep(1000);
}
if (!AsyncProcessManager.Continue(ProcessorDetail.ID))
AsyncProcessManager.GetProcessorDetail(ProcessorDetail.ID).UpdateStatusText("Processing cancelled");
AsyncProcessManager.GetProcessorDetail(ProcessorDetail.ID).UpdateStatusText("Processing is complete");
}
catch (Exception ex)
{
string error = ex.Message + ex.StackTrace;
}
AsyncProcessManager.FinalizeProcess(ProcessorDetail.ID);
}
private bool UpdateStatus(int processed, int total)
{
if (processed > 0)
if (((processed % ProcessorDetail.RefreshStatusCount) == 0) || (processed == total))
return true;
else return false;
else
return false;
}
}
ProcessorTemplate.cs
The base template for creating a worker process including examples of how to call the UpdateStatusText and UpdateCounts methods in the AsyncProcessManager class. Be aware that the AsyncProcessorDetail declared in the worker process classes you create is not the same object that is stored in the AsyncProcessManager, it is just a copy that was passed in when the worker class was instantiated.
ProcessorTemplate.cs
using System;
using System.Net;
using System.IO;
using System.Collections.Generic;
using System.Linq;
public class ProcessorTemplate
{
private AsyncProcessorDetail ProcessorDetail;
public ProcessorTemplate(AsyncProcessorDetail processor_detail)
{
ProcessorDetail = processor_detail;
Process();
}
public void Process()
{
// use the following method to update the status on the AsyncProcessorDetail
// in the AsyncProcessManager
AsyncProcessManager.GetProcessorDetail(ProcessorDetail.ID).UpdateStatusText("Processing has started");
try
{
int rowcount = 0;
int duplicates = 0;
int errorCount = 0;
int processed = 0;
int total = 0;
// in loops, call the UpdateStatus method to determine if you should update the counts
// on the current iteration. This will save processing time so your not updating the
// counts on every iteration
if (UpdateStatus(processed, total))
{
// use the following method to update the counts on the AsyncProcessorDetail
// in the AsyncProcessManager. This should be included inside any worker loops
// you may have or between code blocks to update the user
AsyncProcessManager.GetProcessorDetail(ProcessorDetail.ID).UpdateCounts(processed, errorCount, duplicates, total);
}
// check to see if the process has been cancelled by calling AsyncProcessManager.Continue
// this method should be called within an UpdateStatus condition within any loops you have
// and should be followed by a break;
if (!AsyncProcessManager.Continue(ProcessorDetail.ID))
AsyncProcessManager.GetProcessorDetail(ProcessorDetail.ID).UpdateStatusText("Processing cancelled");
AsyncProcessManager.GetProcessorDetail(ProcessorDetail.ID).UpdateStatusText("Processing is complete");
}
catch (Exception ex)
{
string error = ex.Message + ex.StackTrace;
}
AsyncProcessManager.FinalizeProcess(ProcessorDetail.ID);
}
private bool UpdateStatus(int processed, int total)
{
// determins if the status should be updated based on the RefreshStatusCount and the current
// value of the processed counter
if (processed > 0)
if (((processed % ProcessorDetail.RefreshStatusCount) == 0) || (processed == total))
return true;
else return false;
else
return false;
}
}
asyncProcessingMonitor.ascx.cs
A screen grab of the monitoring control whilst monitoring a process
This is the code behind the monitor control. The ascx file contains an UpdatePanel and a Timer that is used to trigger the UpdatePanel to refresh and display the latest info provided in the AsyncProcessorDetail for the process being monitored.
The Start method below initiates the monitoring of the process and enables the UpdateStatusTimer. It also displays the Cancel button to allow the user to cancel the running process. It displays the animated pleasewait.gif. Note: this is just a simple animated gif, not a 0 to 100% progress bar.
Default.aspx and the monitoring control when processing is complete
The TimerStatusUpdate_Tick event handler is run each time the Timer Tick event occurs and it grabs a copy of the AsyncProcessorDetail from the AsyncProcessManager to update the labels on the control. If the AsyncProcessorDetail.Complete property is true, the monitor knows that the process is complete or has been canceled. So it then calls the Stop method discussed below.
asyncProcessingMonitor.cs
public void Start(int process_id, bool redirectWhenComplete, string redirectURL)
{
// start monitoring the process
ProcessID = process_id;
ButtonRedirect.Visible = false;
Redirect = redirectWhenComplete;
RedirectURL = redirectURL;
ImageProgressGif.Visible = true;
ButtonCancel.Visible = true;
// start the refresh timer
TimerStatusUpdate.Enabled = true;
}
protected void TimerStatusUpdate_Tick(object sender, EventArgs e)
{
// grab a copy of the AsyncProcessorDetail to update the control
AsyncProcessorDetail ifd = AsyncProcessManager.GetProcessorDetail(ProcessID);
if (ifd != null)
{
LabelStartTime.Text = ifd.StartTime.ToLongTimeString();
LabelFilename.Text = ifd.FileName;
LabelTotal.Text = ifd.Total.ToString();
LabelCurrent.Text = ifd.Processed.ToString();
LabelDuplicates.Text = ifd.Duplicates.ToString();
LabelErrorCount.Text = ifd.ErrorCount.ToString();
LabelStatus.Text = ifd.StatusText;
// if the process is complete stop monitoring
if (ifd.Complete)
{
Stop();
if (ProcessingComplete != null)
ProcessingComplete(sender, e);
}
}
else
{
LabelStatus.Text += "<br />" + DateTime.Now.ToLongTimeString() + ": Error reading status";
Stop();
if (ProcessingError != null)
ProcessingError(sender, e);
}
}
The Stop function is called if the AsyncProcessorDetail.Complete property is true. It disables the UpdateStatusTimer to stop the UpdatePanel refreshing and finalizes the state of the panels and buttons on the control. If the Redirect bool is true, the redirect button is displayed in instances when you want to provide a Continue button to redirect the user to another page or onto another process monitor in a sequence of processes. The user will be redirected to the redirectURL supplied when the Start method was called.
The Cancel function updates the status text on the AsyncProcessorDetail and then calls the AsyncProcessManager.StopProcessor method. This will trigger the running process to stop executing as it sets the AsyncProcessorDetail.Stop property to true.
public void Stop()
{
// stop monitoring the process
TimerStatusUpdate.Enabled = false;
ImageProgressGif.Visible = false;
ButtonCancel.Visible = false;
// show the redirect button if its required
if (Redirect)
ButtonRedirect.Visible = true;
}
private void Cancel()
{
// update the status on the AsyncProcessorDetail and stop it from running
AsyncProcessManager.GetProcessorDetail(ProcessID).UpdateStatusText("Cancelling process...");
AsyncProcessManager.StopProcessor(ProcessID);
}
Conclusion
I hope that you find the code I’ve provided useful and that it saves time implemeting a system that requires asyncronous tasks to be kicked off and monitored from a webpage. There are of course enhancements and functional improvements that can be made to what I’ve provided. My aim is to provide a bare bones implementation and working example of the concept so that it may be built on.
Just writting up this post I thought that it is probably a good idea to remove the AsyncProcessorDetail from the worker classes altogther and simply store the ID of the process and therefore use only the detail stored in the manager class. The UpdatePanel also needs some refinement and possibly a postback trigger that executes when processing is complete. In addition, the animated gif refreshes each time the UpdatePanel refreshes, this needs to be changed.
Use the comments section to post any questions or suggestions and I’ll do my best to respond.
Source code: AsyncProcessor.zip
This annoying little gadget will generate phantom keystrokes on the unfortunate PC it’s connected to. It will send random keystrokes, turn on caps lock or randomly move the mouse. You can also adjust the frequency of the random annoyances. Hilarious joke to play on someone? Or absolutely juvenile fun? Why would someone waste their time producing a gadget like this. Profit? A fun little practical joker gadget regardless
Technology is truth 