Memory-mapped files using the boost library

The objective of memory mapping files is to increase I/O performance. Memory mapping a file creates a pointer to a segment in virtual memory and the actual loading is performed by the Operating System one page at a time. For large files, this is much faster than using traditional methods in C such as fopen/fread/fwrite.

In this post, I show an example of how to use the boost iostreams library to create a memory mapped file that, unlike mmap, works for both Windows and Linux.

Start with installing the boost iostreams library. On ubuntu this is done by installing the libboost-iostreams-dev package.

sudo apt-get install libboost-iostreams-dev

The example below will create a memory mapping of 1000000 integers for the file filename.raw. The integers will be available from the pointer called data.

#include <boost/iostreams/device/mapped_file.hpp>
#include <iostream>
 
int main() {
 
    boost::iostreams::mapped_file_source file;
    int numberOfElements = 1000000;
    int numberOfBytes = numberOfElements*sizeof(int);
    file.open("filename.raw", numberOfBytes);
 
    // Check if file was successfully opened
    if(file.is_open()) {
        // Get pointer to the data
        int * data = (int *)file.data();
 
        // Do something with the data
        for(int i = 0; i < numberOfElements; i++)
            std::cout << data[i] << " ";
 
        // Remember to unmap the file
        file.close();
    } else {
        std::cout << "could not map the file filename.raw" << std::endl;
    }
}

Here is a minimal CMakeLists.txt file for compiling this example together with the boost iostreams library.

cmake_minimum_required(VERSION 2.8)
find_package(Boost COMPONENTS iostreams REQUIRED)
 
add_executable(memory-map main.cpp)
target_link_libraries(memory-map ${Boost_LIBRARIES})

As usual you can download/clone the code and the sample raw file from my GitHub page

You may also like...

12 Responses

  1. L. F. Ant says:

    You do not need the .close() call, as this will be done in the destructor anyways. (Which is also the only place where an exception safe resource release can happen.)

    This is the way it should be done for every C++ class, google RAII if you want to know more.

  2. Anonymous says:

    Thanks!

  3. Arpan says:

    Hi Eric,
    Thanks a lot for the post. I have tried to tweak it a bit and have run into problems. I am trying to read a file of x,y,z into an cl_float3 array using memory mapped file using the example above but its not giving the format properly and ends up giving access violation errors:

    cl_float3 *data;
    boost::iostreams::mapped_file_source file;
    size = 785444;
    int numBytes = size*sizeof(cl_float)*3;
    file.open(filename,numBytes);
    if(file.is_open()) {
    data = (cl_float3*)file.data();
    file.close();
    }

    Can you help me out on this?

    • Erik Smistad says:

      Try reading regular floats instead and then create a for loop to create the cl_float3 structure. Something like this:

      int numBytes = size*sizeof(float)*3;
      cl_float3 *data = new cl_float3[size];
      float * tmp = (float*)file.data();
      for(int i = 0; i < size;i++) { data[i].x = tmp[i*3]; data[i].y = tmp[i*3+1]; data[i].z = tmp[i*3+2]; } delete[] tmp;

      If that doesn't work, check that size is correct and that you are opening the correct file etc.

      • Arpan says:

        Thanks for the reply Erik. I was actually fooled by the cl_float3 type. It is typedef of cl_float4 which is 16 bytes. I made a custom data type and then, everything worked.

        Now there is another problem I am having regarding boost. If I am trying to map different parts of the same file using the above code, I am getting an error
        “boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector > at memory location 0x003cefdc..”
        This is happening during the second time of the mapping. So its not possible to unmap and map different data onto the same boost::iostreams::mapped_file_source object?

  4. gnthibault says:

    Very usefull tips, I was able to open a raw image file through boost using your snippet.

    I also used boost::filesystem::path wich is compatible with boost::iostreams::mapped_file_source::open

    Thank you again !

  5. Anonymous says:

    Is it thread safe?

  6. Anonymous says:

    I am reading 560 mb file. Using fread it tales 6.41 sec where as using boost it takes 8~10 sec.

  7. Luc says:

    Hey. Thanks for the snippets. I’ve just started using boost::iostreams library with memory mapped files, trying to improve I/O performance in my app, but I found it really degrading the speed.
    Using mapped_file_source with iostream wrapper for reading chunks of 1MB data is in my case more than 20% worse in read speed than using raw C fread function. How it was working in your case? Did you also expect some performance degradation?
    Cheers
    L

    • Erik Smistad says:

      I see a large increase in performance when reading files using memory mapping. However, the files I read using memory mapping are large. Typically several hundred MBs.

      • Luc says:

        Thanks for answer! 🙂 Replying late, as for large data files and small random access reads ( 1MB) direct IO reads with manual buffer allocation here can perform better.
        Cheers,
        L

        • Luc says:

          The message filter cut some info from the message above 😛 By small chunk I meant a chunk less than 100 KB, where big chunk is same or greater than 1 MB.

Leave a Reply

Your email address will not be published.