A GPU is not Always Fastest

There has been a huge amount of interest in GPU computing (GPGPU) over the last couple of years.  Unsurprisingly, a number of image processing algorithms have been implemented using this technology.  In most cases, large performance gains are reported.  However, whilst I have been writing image processing algorithms that leverage the GPU performance for some time now, I have often found that the GPU is not the best solution.  As a rule of thumb, I aim for a x10 increase in speed to justify the development, if I can't achieve a x4 increase in speed then its just not worth the effort.  

Sometimes, the performance gains are misleading for practical applications.  NVidia themselves are guilty of this in their SDK with their image processing examples.  For instance, in many of their SDK demonstration applications they use the SDK functions to load an 8-bit image and then pre-convert it on the host to a packed floating point format before uploading to the GPU.  They then show large gains in speed, but ignore the huge time penalty of the CPU-side format conversion.  In another example they have to unpack 24-bit RGB data into 128bit packed quads of floating point data on the host before they can process it.  In the real-world this is not practical.  I do wonder how many other people have used some constructive accounting in their reported acceleration factors.

So, despite generally being a GPU evangelist for accelerating image processing, I wanted to write a bit about the downsides to provide a balanced view.

Architecture constraints.   You need to be doing a lot of work on the image data to make the architecture work for you.  Many (Most?) practical algorithms just don't fit into a GPU very well.  For example, it may be the case that a GPU can do a brute-force template correlation faster than a quad-core CPU, but brute-force correlation for pattern matching isn't the method of choice these days.  Contemporary vision libraries have extremely sophisticated algorithms that do a far superior job of pattern matching than correlation, plus they are highly optimised for multi-threading on the CPU.  These algorithms simply do not fit into the GPU 'brute force' computational model. 


By way of a painful example, I have been developing a complete JPG conversion library for NVidia GPUs.  This is blazingly fast at RGB-YUV conversion, DCT and Quantisation, but falls down on the Huffman coding which is a sequential algorithm.  Add in the transfer overheads and it gets slower.  At the time of writing, hand-optimised multi-threaded CPU version is almost as fast.  All is not lost on this development, but its a tough sell at this point.

Multi-threading.  Whilst a GPU is massively parallel internally, it cannot run multiple algorithms (kernels) in parallel*.  So if your application is used to doing multiple operations in parallel, e.g. processing the images from multiple sensors in parallel, then it will have to change and serialize the images into GPU work chunks.  So whilst your quad-core CPU could be doing four images at once, the GPU is doing them in serial.  This means the GPU has to process at least four times the rate than a single CPU core in order to break even.  

*I believe the new NVidia Fermi architecture can run multiple Kernels simultaneously but most don't.

Transfer Overheads.  It takes time to transfer data across the PCIe bus to and from the GPU.  If the algorithm already runs quickly on the CPU (e.g. a few milliseconds) then GPU acceleration is usually a non-starter.
 

Algorithm development time.  It takes longer to write and debug a massively parallel GPU algorithm than it does to parallelize the algorithm on the CPU to make use of a fast quad-core.  Development time is expensive.

Hardware cost.  You do get a lot of horsepower for your money with a GPU, and a good performance card can be purchased for £150.  That still has to be factored into the system cost.   
Hardware obsolescence.  Whilst NVidia have confirmed that CUDA will be available in every new GPU they produce, the exact same GPU card quickly becomes obsolete.  Code should be forward compatible, but I don't think this has really been put to the test yet.




Of course, there are still lots of good things about this new technology and it really can accelerate the big number crunching algorithms like optic flow and deconvolution and FFTs.  But you have to choose carefully.


Vision Experts

High Throughput for High Resolution

We've been using the ProSilica/AVT GE4900 recently to get super high resolution 16megapixel images at about 3Hz.  It's a nice camera, but that resolution tends to demand high performance from the processor.

We have about 45MB/sec of raw image data we have to process.  In order to chew through all this data we've been pushing the raw bayer mosaic images onto an NVidia GTX260 GPU and performing colour conversion, gamma correction and even the sensors flat field correction on the GPU at high speed.  We also use the GPU to produce reduced size greyscale images for processing and alalysis alongside the regular colour converted image for display.  The ability to process such high resolution images using the GPU has really made the difference for this application and it would not be possible without this capability. 
Vision Experts

Interface Acceleration

Machine vision sensors are getting big, and Camera are increasingly available with a number of pixels that is truly enormous by historical standards.  Cameras in the 10+ Megapixel range seem to be increasing in popularity for industrial inspection, possibly driven by the consumer market in which such large sensors are now the norm, partly due to price decreases, and possibly because processing and storing the data is just about feasible these days.  

The bandwidth between cameras and computer is also increasing, which it needs to.  Already, it seems that a single GigE connection just isnt enough bandwidth for tomorrows applications.  For example, AVT have a dual GigE output on a camera to give 2Gbits/sec of bandwidth.  The CoaXPress digital interface is capable of 6.25 Gbits/sec over 50m of pretty much bog-standard coax cable, a capability I find incredible.  Likewise, the HSLINK standard, proposed by DALSA, uses InfiniBand to achieve 2100Mbytes/sec.   Most of these standards even permit using multiple connections to double, or quadruple the bandwidth.  With all this data flying around, trying to process this on a PC is going to be like taking a drink from a hose pipe.  Or two, or four. 

Think about it, at 2Gbits/sec, the computational demand will be 250Mpix/sec (assuming 8bit pixels).  Using a 3GHz processor core, thats 12 clock cycles availble per pixel.  You can't do a whole lot of processing with that.  Even if you scale up to Quad-core and make sure you use as many SSE SIMD instructions as you can, you still aren't going to be doing anything sophisticated with that data.  It could be like machine vision development 15 years ago, when I remember the only realistic goal was to count the number pixels above threshold to take a measurement!


I feel that the new generation of ultra-high resolution cameras streaming data at ultra-high bandwidths are going to require a new generation of processing solutions.  I suspect this will be in the form of massivley parallel processors - such as GPU's and perhaps Intel's Larrabee processor (when it finally materialises).


In the mean time, I'm plugging away writing GPU accelerated algorithms just for format conversion so that we can even display and store this stuff.


Vision Experts

GPU Supercluster

I was interested to see this GPU system doing some biologically inspired processing at Harvard.  Whilst I doubt that there will be any practical industrial applications to emerge from this, it does show how inexpensive it can be to build a minor supercomputer. To quote from their website...

...With peak performance around 4 TFLOPS (4 trillion floating point operations per second), this little 18”x18”x18” cube is perhaps one of the world’s most compact and inexpensive supercomputers....

Vision Experts

C# Interop

I write image processing algorithms for a (so-called) living, which explains why my posts are so badly written and incomprehensible.  I write quite a few libraries in C/C++ for various industrial tasks and supply them as a trusty old windows dll.   Providing an interface for calling my libraries from a C# application front-end is something I have to do quite a bit of.  C++ delivers the performance needed for image processing and C# gives the quick and easy GUI. 



In order to call a regular 'C' dll from C#, you need to use something in .NET called P/Invoke.  This mechanism defines a function that is callable from C# which maps to a dll function call.  In the definition of the dll function, you can specify things like character sets for string passing, calling conversions etc.  As an example, if you wanted to import the windows kernel32 function Beep into C# using P/Invoke it looks something like:

[DllImport("kernel32.dll")]
public static extern bool Beep(int frequency, int duration);

In the beep example, the integer values passed from managed to unmanaged dll are so-called blittable types and will be passed directly.  Passing arrays is not quite as simple, since they have to be converted (marshalled) by the framework before they are passed to the unmanged dll.  Normally, you don't use unsafe code in your C# GUI, so you don't have pointers to data lying around handy.  Of course, most C libraries for image processing expect a pointer to some image data to be passed in somewhere, not a managed array object.  So somebody has to do some work to turn a managed array into a pointer, without totally screwing up the safe part of C# and all the other stuff going on like the garbage collector.  This is job of P/Invoke.


For example, if we have a C# array declared as:

 float[] CalTgtX = { 58,198,340, 58,198,340, 58,198,340};

which we want to pass into a C++ function that looks like this:

extern "C" __declspec(dllexport) void __stdcall CalibrateProjection(float *pTargetX)

Then we need to carefully define how C# should carry out this conversion.  Heres how to define the function in C# so that we can pass (marshal) that float array object from C# to the C dll function:

[DllImport("MyLib.dll", EntryPoint = "CalibrateProjection", CallingConvention = CallingConvention.StdCall)]
public static extern RETCODE CalibrateProjection([MarshalAs(UnmanagedType.LPArray, SizeConst = 9)] float[] pTargetX)

The first line tells the C# compiler to import a function.  DllImport tells c# we are importing a function from a Dll.  EntryPoint tells c# what the function stub is named. CallingConvention should match that used by the Dll - here it was __stdcall.  
The second line defines the function as it will appear to C#.  The key to converting the float array object to a pointer is in the MarshalAs attribute.  This will involve a copy to an unsafe array on the heap so can be slow for large arrays... very slow.


All the different flavours of managed types and structs can be marshalled this way.  More information on PInvoke can be found at http://msdn.microsoft.com/en-us/library/aa288468%28VS.71%29.aspx

 

Vision Experts

LoaderLock MDA

This post isn't really about accelerated image processing, but the topic is related to deployment of DLLs of any type.  I hope this helps somebody save some time if they encounter this issue.  Whilst developing a C# demo app for one of my CUDA libraries, I encountered a strange error:

LoaderLock was detected
Message: DLL 'Cephalon.dll' is attempting managed execution inside OS Loader lock. Do not attempt to run managed code inside a DllMain or image initialization function since doing so can cause the application to hang.

It took me a while to figure out what was going on, and it was related to how I build CUDA libraries.  

When I make a CUDA enabled library, I wrap up the compiled kernel cubin files as a resource compiled into the DLL itself.  An alternative simpler method is to supply a cubin text file along with each dll and load it directly using the CUDA function:

cuModuleLoad(&cuModule, pszModulePath) 

 

but having two files can lead to version control and maintainance issues. Plus, I try and make a living doing this, and dont really want people reading my precious kernel code that took four months to write too easily. 



So I wrap up the compiled code string neatly inside the DLL as a resource, then get that resource string and compile it on-the-fly (or just-in-time) using the alternative CUDA function:

cuModuleLoadDataEx( &cuModule,pCubinStr,3,&jitOptions[0],&jitOptVals[0]);

This is great but in order to get the string resource from inside the DLL I need to call a varient of LoadResource. And I need to call FindResource to find that resource first.  And I need to call GetModuleHandle("LibraryName.dll") before any of those.  The problem is that GetModuleHandle is a prohibited function to call even indirectly from LoadLibrary when the DLL is first loaded and mapped into the process address space.  

The C# application was loading the DLL when it first encountered one of the functions, this then tried to initialise the CUDA module and load the resource automatically from the dll entry point.  Ultimately, the call to GetModuleHandle raised an alarm back in the managed code.  Not easy to spot.


More on the LoaderLock MDA can be found here





Vision Experts

OpenGL Interop

Looks Matter

Never ignore the display capabilities of the GPU you have.   After all, with the rendering capabililties of a meaty CUDA enabled GPU and the software capabilities of OpenGL you should easily be able to exploit the images and data you've just processed in interesting and engaging ways.  Granted, it should be the result that matters.  But when it's human engineers there comparing your system against the competition and the look-and-feel of your application is slick and polished, it can make a difference.  The key to doing making your CUDA app look slick is OpenGL interop.  

Process vs Display

Image processing and Image generation are two sides of the same coin.   Image processing seeks to take an image and extract information (size, shape, depth, region, motion, identity etc.), image generation seeks to take information (vertices, texture, algorithms) and turn that data into an image.  Looking at a software level, the two sides of the coin can be addressed by using two GPU technologies - image processing using CUDA and image generation using OpenGL.  By using CUDA we can turn an NVidia GPU into a powerful image processor, by using OpenGL we can use the same GPU hardware to generate new images. For example, with CUDA image processing algorithm we could extract the motion and depth from a scene in real-time, then with OpenGL image generation we could re-generate camera stabilized video or generate a panorama or even completely re-render the scene from a novel and augmented perspective.  It is when we combine image processing with image rendering this way that things get really interesting.  

Interop

In my option, CUDA OpenGL interop seems to be under-documented as well as being a bit more complex that it should be.  In sequence - here's how I use CUDA-OpenGL interop:


At program initialisation:

1. Allocate an OpenGL texture that will be compatible with your results (not always easy)
   
2. Allocate an OpenGL Pixel Buffer Object (PBO) using glGenBuffers
   
3. Register the PBO with CUDA using cuGLRegisterBufferObject

Note that (in CUDAv2.2) OpenGL cannot access a buffer that is currently *mapped*. If the buffer is registered but not mapped, OpenGL can do an requested operations on the buffer. Deleting a buffer while it is mapped for CUDA results in undefined behaviour (bad things).  Also, always use the same context to map/unmap as the context used to register the buffer.  This can be difficult with the Runtime API in a multi-threaded app and results in strange behaviour.



A Digression on Pitch
There is a complication with texture allocation and cuda device memory allocation.  With CUDA, you really must allocate pitched device memory (using cuMemAllocPitch) for image processing usage.   This is in order to meet strict alignment requirements for fast coalseced memory access.  You dont have control over the pitch that CUDA will use, but cuMemAllocPitch returns the actual pitch of the device mem that was allocated, which is anything up to 256bytes.  When you allocate a texture in OpenGL, you cannot specify a texture pitch, only width, height and format.  This means that your OpenGL texture buffer may not be pitch-compatible with your CUDA device memory layout.  You can use GL_UNPACK_ALIGNMENT and GL_UNPACK_ROW_LENGTH to help out here, but there are still some fairly common situations when this wont quite give you the control you need.  A symptom of mis-matched texture and device memory pitch is when the image data looks like its made it across the interop but is weirdly diagonally scewed or of the wrong aspect ratio.  Usually, through a combination of modification to your texture width, packing alignment and/or format you can achieve something compatible. 


For now, I'll assume you have managed to allocate a compatible texture, then;



At run-time:

1. Run the CUDA kernel putting the results into device memory (cuDevicePtr)
   
2. Map the PBO using cuGLMapBufferObject, which returns the device pointer of the texture memory (another cuDevicePtr)
   
3. Use cuMemcpy2D to copy from the device memory to the mapped PBO memory.  These are device-to-device copies.
   
4. Unmap the PBO (cuGLUnmapBufferObject)
5. Update the texture from the PBO
   
6. Use OpenGL to draw with your new texture
   

Develop a Generic Interop Class

In most of the NVidia examples, CUDA results are written straight to the mapped texture memory during kernel execution.  In reality I found it much more efficient (from a productivity perspective) to write the code for the above operations once, and package that up as a little interop utility class.  Now, I can always copy from any CUDA device buffer into a suitable OpenGL texture without having to write similar code for every type of kernel launch. Not writing results directly into mapped opengl memory means that you incur an additional copy afterwards, but device-to-device copies are relatively fast in the scheme of things here.


Im putting together some tutorials on interop - they'll be along soon.


In the mean time - take a look at the Interop Release notes from CUDA 2.2...

o OpenGL interoperability
  - OpenGL cannot access a buffer that is currently
    *mapped*. If the buffer is registered but not mapped, OpenGL can do any
    requested operations on the buffer.
  - Deleting a buffer while it is mapped for CUDA results in undefined behavior.
  - Attempting to map or unmap while a different context is bound than was
    current during the buffer register operation will generally result in a
    program error and should thus be avoided.
  - Interoperability will use a software path on SLI
  - Interoperability will use a software path if monitors are attached to
    multiple GPUs and a single desktop spans more than one GPU
    (i.e. WinXP dualview).

Vision Experts