COM Structured Storage

Not to be confused with Structured storage.

COM Structured Storage (variously also known as COM structured storage or OLE structured storage) is a technology developed by Microsoft as part of its Windows operating system for storing hierarchical data within a single file. Strictly speaking, the term structured storage refers to a set of COM interfaces that a conforming implementation must provide, and not to a specific implementation, nor to a specific file format (in fact, a structured storage implementation need not store its data in a file at all). In addition to providing a hierarchical structure for data, structured storage may also provide a limited form of transactional support for data access. Microsoft provides an implementation that supports transactions, as well as one that does not (called simple-mode storage, the latter implementation is limited in other ways as well, although it performs better).

Structured storage is widely used in Microsoft Office applications, although newer releases (starting with Office 2007) use the XML-based Office Open XML by default. It is also an important part of both COM and the related Object Linking and Embedding (OLE) technologies. Other notable applications of structured storage include SQL Server, the Windows shell, and many third-party CAD programs.

Motivation

Structured storage addresses some inherent difficulties of storing multiple data objects within a single file. One difficulty arises when an object persisted in the file changes in size due to an update. If the application that is reading/writing the file expects the objects in the file to remain in a certain order, everything following that object's representation in the file may need to be shifted backward to make room if the object grows, or forward to fill in the space left over if the object shrinks. If the file is large, this could result in a costly operation. Of course, there are many possible solutions to this difficulty, but often the application programmer does not want to deal with low level details such as binary file formats.

Structured storage provides an abstraction known as a stream, represented by the interface IStream. A stream is conceptually very similar to a file, and the IStream interface provides methods for reading and writing similar to file input/output. A stream could reside in memory, within a file, within another stream, etc., depending on the implementation. Another important abstraction is that of a storage, represented by the interface IStorage. A storage is conceptually very similar to a directory on a file system. Storages can contain streams, as well as other storages.

If an application wishes to persist several data objects to a file, one way to do so would be to open an IStorage that represents the contents of that file and save each of the objects within a single IStream. One way to accomplish the latter is through the standard COM interface IPersistStream. OLE depends heavily on this model to embed objects within documents.

Format

Main article: Compound File Binary Format

Microsoft's implementation uses a file format known as compound files, and all of the widely deployed structured storage implementations read and write this format. Compound files use a FAT-like structure to represent storages and streams. Chunks of the file, known as sectors (these may or may not correspond to sectors of the underlying file system), are allocated as needed to add new streams and to increase the size of existing streams. If streams are deleted or shrink, leaving unallocated sectors, those sectors can be reused for new streams.

The following applications use the OLE Structured Storage (Compound Document Format)

• Microsoft Office 97–2003 documents:
  • Word documents (.DOC, .DOT)
  • Excel spreadsheets (.XLS, .XLT)
  • PowerPoint presentations (.PPT, .POT)
  • Publisher files (.PUB)
  • Visio files (.VSD)
  • Project files (.MPP)
  • Microsoft PhotoDraw files (.MIX)
  • Microsoft Outlook files (.MSG)
• Windows Installer files (.MSI, .MSP, .MST)
• Microsoft Picture It! / Microsoft Digital Image files (.MIX)
• Internet Explorer RSS Feeds Windows RSS Platform files (.feed-ms)
• Windows 7 StickyNotes (.SNT)
• Windows 7 jumplists files
• Thumbs.db
• Microsoft SQL 2000 Server DTS packages
• Autodesk Revit
• Autodesk Inventor
• FlashPix
• Altium Designer

Native Structured Storage

During the beta testing phase of Windows 2000, it included a feature titled Native Structured Storage (NSS) for storage of Structured Storage documents (like the binary Microsoft Office formats and the thumbs.db file Windows Explorer uses to cache thumbnails) with each Stream that makes up a document stored in a separate NTFS data stream. It included utilities that automatically split up the streams in a regular Structured Storage document into NTFS data streams and vice versa. However, the feature was withdrawn after Beta 3 due to incompatibilities with other OS components, and any NSS files automatically converted to the single data stream format.

Implementations

• For Microsoft .NET:
  • OpenMCDF – Free .NET component for accessing OLE structured storage files, MPL licensed.
• For Linux:
  • GNOME Structured File Library – Can read Microsoft structured storage files.
  • POLE.
• Cross platform C++ for Window/MacOSX/Linux:
  • POLE v3 and up.
• For Java:
  • POIFS – Java implementation of the OLE 2 Compound Document format, part of Apache POI.
• For Perl:
  • LAOLA Binary Structures
• For JavaScript:
  • js-cfb – JavaScript implementation of the OLE 2 Compound Document format.
• For Python:
  • compoundfiles – Python implementation of the Microsoft Compound File Binary (CFB) format.

External links

• Official website
• Open Specifications: Compound File Binary File Format