First a quick review of what I mean by a parallel file system.  There are any number of different types of parallel file systems available.  Some allow multiple systems and applications to share common pools of storage as in a clusered filesystem.  Some split the data across two or more nodes to improve access time and redundancy.  Other variants split files into lots of small chunks, stores these chunks on different disks in a round-robin fashion, and re-combine them upon reading to get back the original file.

The earliest instance of Microsoft Windows-specific parallel fileystem that I have found to date is the parallel filesystem developed by the ARGOS group at Universidad Carlos 111 de Madrid, Madrid, ES.  This research group developed a prototype of a parallel file system for a network of Microsft Windows nodes which they called WinPFS.  They presented their work at COSET 2004 and a number of other workshops.  WinPFS was implemented as a new fileyystem type fully integrated within the Microsoft Windows kernel.  This has the advantage that no modification or recompilation of user applications is needed to take advantage of the parallel filesystem.

The goal of this research group was to build a parallel file system for networks of Microsoft Windows computers using Microsoft Windows shared folders to access remote data in parallel. The implementation is based on file system redirectors which redirect requests to remote nodes using UNC (Universal Naming Convention) and the SMB and/or CIFS protocols. WinPFS is registered as a virtual remote file system and access to remote data is through a new shared folder \\PFS.  The basic file operation primatives are: create, read, write, and create directory.

The prototype was developed on the Windows XP platform, and has been tested with a cluster of seven Windows XP nodes and a Windows 2003 Server node in various configurations.  Maximum throughput for write operations were 250 Mbit/s and 1200 Mbit/s for read operations.  The research team reported that the bottleneck for writes was the disks and for reads was the network.  As far as I can tell this ]]> http://blog.fpmurphy.com/2009/05/windows-parallel-filesystems.html/feed 0 http://blog.fpmurphy.com/2009/04/porting-waitforsingleobject-to-linux.html http://blog.fpmurphy.com/2009/04/porting-waitforsingleobject-to-linux.html#comments Thu, 30 Apr 2009 13:38:00 +0000 fpmurphy http://blog.fpmurphy.com/2009/04/porting-waitforsingleobject-to-linux-part-2.html In my last post I discussed the use of WaitForSingleObject in relation to mutexes and possible ways to implement equivalent functionality when porting such code to GNU/Linux.  In this post I will describe the use of this API with event objects in Microsoft Windows and suggest possible ways of posting such code to GNU/Linux or Unix.

First, some background on event objects.  An event object is just another type of Windows kernel dispatcher object.  From a coding prespective, an event object is a synchronization object which encapsulates one or more kernel dispatcher objects and whose synchronization semantics are accessable via WaitForSingleObject and its cousins.  At any given time a synchronization object is either nonsignaled or signaled, i.e. the object can only be in one of two possible states.

All of the WaitFor family of APIs including WaitForSingleobject wait on an object handle or handles until some specified criteria is met.  The two basic criteria for all these APIs are the signaled state of the object on whose handle it is waiting and a time-out value.  Thus a thread which calls this API waits till the specified object enters the signaled state or the specified time-out has expired.  Little or no CPU time is used when such a thread is in the wait state.

In the case of events, a CreateEvent or OpenEvent returns a handle to an event object.  When an event is in the signaled state it means that that the event has the capacity to release one or more threads waiting for this particular event to be signaled.  When an event is in the nonsignaled state it will not release any waiting thread.  Initially the state of an event is nonsignaled.  An event object’s state is set explicitly to signaled by SetEvent or PulseEvent.  Event objects are also used in overlapped operations such as reading from a socket, in which case the event object state is set to signaled by the kernel rather than by an application.

Events also come in two reset types.  If an event is a manual-reset event, then all WaitForSingleObjects return that wait for that event if so configured.  In other words a manual-reset event can trigger action by one or more WaitForSingleObject or its cousins.  A manual-reset event object’s state must be reset explicitly to nonsignaled by ResetEvent.

For an auto-reset event object, WaitForSingleObject and it’s relations reset the state ]]> http://blog.fpmurphy.com/2009/04/porting-waitforsingleobject-to-linux.html/feed 0 http://blog.fpmurphy.com/2009/04/porting-waitforsingleobject-to-gnulinux.html http://blog.fpmurphy.com/2009/04/porting-waitforsingleobject-to-gnulinux.html#comments Wed, 29 Apr 2009 20:01:00 +0000 fpmurphy http://blog.fpmurphy.com/2009/04/porting-waitforsingleobject-to-linux-part-1.html Recently I was involved in porting a 32-bit application which was initially written for Microsoft Windows NT to GNU/Linux.  This application contained a large number of calls to NtWaitForSingleObject and a smaller number of calls to NtWaitForMultipleObject. 

Now anybody who has had to port code containing more than a few instances of these particular Win32 APIs, or their close cousins WaitForSingleObjectEx, MsgWaitForMultipleObjects, MsgWaitForMultipleObjectsEx, etc. to Unix or GNU/Linux is probably already shuddering with the recollection of long arduous days and nights of trial and error coding to try and correctly mimic the semantics and functionality of these particular Microsoft Windows specific APIs, but for the reader who has not yet had to attempt to port such an application, this post and my next post may help you save your sanity (and possibly your hair!) sometime in the future.

By the way, both of these APIs are marked deprecated in MSDN by Microsoft but still work as expected in Windows NT and Windows XP.  I am not sure about Windows Vista or Windows 7 as I have not tested them on these operating systems.  The two deprecated APIs have been replaced by the equivalant APIs WaitForSingleObject and WaitForMultipleObject respectively.  For the remainder of this post I shall just discuss the replacement APIs but most of what I say will be valid for either the deprecated or the replacement API.

On first examination WaitForSingleObject seems fairly benign.  The description in MSDN states that “This function returns when the specified object is in the signaled state or when the time-out interval elapses”.  Sounds like a fairly simple and innocuous API, right?  Maybe something similar to the POSIX.1 API pthread_cond_timedwait.  Well, you are dead wrong and this post and the following will explain why.

WaitForSingleObject and its cousins can wait for a signal from any or all of the following “objects”: change notification, console input, event, job, memory resource notification, mutex, process, semaphore, thread and waitable timer and in limited circumstances on files and file I/O.  When appropriately signaled, a thread is unblocked and continues.  No published standardized API in the GNU/Linux or Unix world comes come to handling this range of objects in a single API.

This is probably the one single area where a Win32 API is better designed than the GNU/Linux or Unix API set.  In GNU/Linux and Unix there are specific APIs to wait for different kinds of events: ]]> http://blog.fpmurphy.com/2009/04/porting-waitforsingleobject-to-gnulinux.html/feed 2 http://blog.fpmurphy.com/2008/12/more-on-powershell.html http://blog.fpmurphy.com/2008/12/more-on-powershell.html#comments Fri, 19 Dec 2008 10:38:00 +0000 fpmurphy http://blog.fpmurphy.com/2008/12/more-on-powershell.html Microsoft’s PowerShell is radically different than shells on UNIX or GNU Linux systems in that Powershell can deal in objects rather than just plain text.

A concrete example may help you more quickly understand the difference.  Suppose you want to get and save information about all the files in a certain subdirectory.   We want to get not only the names of the files but as much metadata as possible relating to each file such as date of creation, date of modification, etc.  This information also needs to be stored in a single XML document.  To keep the size of this post manageable, our subdirectory contains only two files, i.e. file.xml and file.xsl, as shown below.

On a UNIX or GNU Linux system, this would be a somewhat difficult task to quickly accomplish.  Using Powershell, however, it becomes a simple one or two line task.

Because Powershell is fundamentally object-orientated, it regards a filesystem as being an object representated as a location exposed by provider.  More generally, a location can be a file directory, a registry hive, a certificate store or some other “thing” exposed by a provider.  The Powershell get-childItem cmdLet is used to retrieve information, including metadata, about items in a specified location.  Thus get-childItem can be used for a lot of different types of operations including enumerating directory contents, listing registry values or showing the current values of variables.  For our purposes, get-childItem exposes information about files in a specified subdirectory or the current directory if no directory is specified.

Here is a single line Powershell script which does what we want to do.   It retrieves metadata about the two files in the current directory using the get-childItem cmdLet, pipes that data to a cmdLet called ConvertTo-XML which in turn converts the metadata into a valid XML document.  The save method is then used to write this stream out to a file called file.out in the current directory. (get-childItem | ConvertTo-XML -NoTypeInformation).save("$(convert-path "X:")\file.out")

Alternatively, we could store the retrieved metadata in a variable and then write it out to a file as shown below.  Similar to bash and ksh93, Powershell supports the concept of aliases and defines ls as one of the aliases for the get-childItem cmdLet,dir and gci being the other two. $a=(ls | ConvertTo-XML) $a.save("$(convert-path "X:")\file.out")

Here is the contents of file.out.  As you can see, quite a bit of information about each of the two files is exposed. <?xml version="1.0"?> <Objects> <Object Type="System.IO.FileInfo"> ]]> http://blog.fpmurphy.com/2008/12/more-on-powershell.html/feed 0 http://blog.fpmurphy.com/2008/12/vista-snipping-tool-rant.html http://blog.fpmurphy.com/2008/12/vista-snipping-tool-rant.html#comments Sat, 06 Dec 2008 02:22:00 +0000 fpmurphy http://blog.fpmurphy.com/2008/12/vista-snipping-tool-rant.html My version of Microsoft Windows Vista comes with a screen capture tool called the Snipping Tool.  It is a very useful tool which I often use.  With this tool you can capture a screen shot (snip) of any part of your screen and save it to the clipboard or to a file in a number of formats (HTML, PNG, GIF, or JPEG).  Available snip types include free-form, rectangular, window or full-screen.

One day last week the Snipping Tool suddenly stopped allowing me to save a snip to a file.  No error messages or message of any kind.  Pressing the Save option simply did not nothing.  The warning message about saving a unsaved snip still appeared when exiting the Snipping Tool but did not allow me to save the snip.

After some research I stumbled across the answer.  It turns out that I had deleted the Pictures folder in my user folder as part of a cleanup of my folders.  I never use that folder so I naively assumed that I could delete it.&nbsp However if you delete that folder, the Snipping Tool does not display a Save File dialog box.  Note that the Snipping Tool does not force you to save a snip in your Pictures folder but does require you to have the folder!  You can save the snip anywhere you like provided you have the right permissions.

What a poor user interface design! As part of Vista QA did nobody test this scenario?  If it was a deliberate design decision to require the Pictures folder, why not display a dialog box warning the user that the folder is missing and is required?