Whole document tree

Chapter 27: Input/Output

Chapter 27 deals with iostreams and all their subcomponents and extensions. All kinds of fun stuff.

Contents

• Copying a file
• The buffering is screwing up my program!
• Binary I/O
• What is this <sstream>/stringstreams thing?
• Deriving a stream buffer
• More on binary I/O
• Pathetic performance? Ditch C.
• Threads and I/O

Copying a file

So you want to copy a file quickly and easily, and most important, completely portably. And since this is C++, you have an open ifstream (call it IN) and an open ofstream (call it OUT):

   #include <fstream>

   std::ifstream  IN ("input_file");
   std::ofstream  OUT ("output_file"); 

Here's the easiest way to get it completely wrong:

   OUT << IN;

   The quick brown fox jumped over the lazy dog.

Seriously, go do it. Get surprised, then come back. It's worth it.

The thing to remember is that the basic_[io]stream classes handle formatting, nothing else. In particular, they break up on whitespace. The actual reading, writing, and storing of data is handled by the basic_streambuf family. Fortunately, the operator<< is overloaded to take an ostream and a pointer-to-streambuf, in order to help with just this kind of "dump the data verbatim" situation.

Why a pointer to streambuf and not just a streambuf? Well, the [io]streams hold pointers (or references, depending on the implementation) to their buffers, not the actual buffers. This allows polymorphic behavior on the part of the buffers as well as the streams themselves. The pointer is easily retrieved using the rdbuf() member function. Therefore, the easiest way to copy the file is:

   OUT << IN.rdbuf();

So what was happening with OUT<<IN? Undefined behavior, since that particular << isn't defined by the Standard. I have seen instances where it is implemented, but the character extraction process removes all the whitespace, leaving you with no blank lines and only "Thequickbrownfox...". With libraries that do not define that operator, IN (or one of IN's member pointers) sometimes gets converted to a void*, and the output file then contains a perfect text representation of a hexidecimal address (quite a big surprise). Others don't compile at all.

Also note that none of this is specific to o*f*streams. The operators shown above are all defined in the parent basic_ostream class and are therefore available with all possible descendents.

Return to top of page or to the FAQ.

The buffering is screwing up my program!

First, are you sure that you understand buffering? Particularly the fact that C++ may not, in fact, have anything to do with it?

The rules for buffering can be a little odd, but they aren't any different from those of C. (Maybe that's why they can be a bit odd.) Many people think that writing a newline to an output stream automatically flushes the output buffer. This is true only when the output stream is, in fact, a terminal and not a file or some other device -- and that may not even be true since C++ says nothing about files nor terminals. All of that is system-dependent. (The "newline-buffer-flushing only occurring on terminals" thing is mostly true on Unix systems, though.)

Some people also believe that sending endl down an output stream only writes a newline. This is incorrect; after a newline is written, the buffer is also flushed. Perhaps this is the effect you want when writing to a screen -- get the text out as soon as possible, etc -- but the buffering is largely wasted when doing this to a file:

   output << "a line of text" << endl;
   output << some_data_variable << endl;
   output << "another line of text" << endl; 

   output << "a line of text\n"
          << some_data_variable << '\n'
          << "another line of text\n"; 

If you do need to flush the buffer above, you can send an endl if you also need a newline, or just flush the buffer yourself:

   output << ...... << flush;    // can use std::flush manipulator
   output.flush();               // or call a member fn 

On the other hand, there are times when writing to a file should be like writing to standard error; no buffering should be done because the data needs to appear quickly (a prime example is a log file for security-related information). The way to do this is just to turn off the buffering before any I/O operations at all have been done, i.e., as soon as possible after opening:

   std::ofstream    os ("/foo/bar/baz");
   std::ifstream    is ("/qux/quux/quuux");
   int   i;

   os.rdbuf()->pubsetbuf(0,0);
   is.rdbuf()->pubsetbuf(0,0);
   ...
   os << "this data is written immediately\n";
   is >> i;   // and this will probably cause a disk read 

Since all aspects of buffering are handled by a streambuf-derived member, it is necessary to get at that member with rdbuf(). Then the public version of setbuf can be called. The arguments are the same as those for the Standard C I/O Library function (a buffer area followed by its size).

A great deal of this is implementation-dependent. For example, streambuf does not specify any actions for its own setbuf()-ish functions; the classes derived from streambuf each define behavior that "makes sense" for that class: an argument of (0,0) turns off buffering for filebuf but has undefined behavior for its sibling stringbuf, and specifying anything other than (0,0) has varying effects. Other user-defined class derived from streambuf can do whatever they want. (For filebuf and arguments for (p,s) other than zeros, libstdc++ does what you'd expect: the first s bytes of p are used as a buffer, which you must allocate and deallocate.)

A last reminder: there are usually more buffers involved than just those at the language/library level. Kernel buffers, disk buffers, and the like will also have an effect. Inspecting and changing those are system-dependent.

Return to top of page or to the FAQ.

Binary I/O

The first and most important thing to remember about binary I/O is that opening a file with ios::binary is not, repeat not, the only thing you have to do. It is not a silver bullet, and will not allow you to use the <</>> operators of the normal fstreams to do binary I/O.

Sorry. Them's the breaks.

This isn't going to try and be a complete tutorial on reading and writing binary files (because "binary" covers a lot of ground), but we will try and clear up a couple of misconceptions and common errors.

First, ios::binary has exactly one defined effect, no more and no less. Normal text mode has to be concerned with the newline characters, and the runtime system will translate between (for example) '\n' and the appropriate end-of-line sequence (LF on Unix, CRLF on DOS, CR on Macintosh, etc). (There are other things that normal mode does, but that's the most obvious.) Opening a file in binary mode disables this conversion, so reading a CRLF sequence under Windows won't accidentally get mapped to a '\n' character, etc. Binary mode is not supposed to suddenly give you a bitstream, and if it is doing so in your program then you've discovered a bug in your vendor's compiler (or some other part of the C++ implementation, possibly the runtime system).

Second, using << to write and >> to read isn't going to work with the standard file stream classes, even if you use skipws during reading. Why not? Because ifstream and ofstream exist for the purpose of formatting, not reading and writing. Their job is to interpret the data into text characters, and that's exactly what you don't want to happen during binary I/O.

Third, using the get() and put()/write() member functions still aren't guaranteed to help you. These are "unformatted" I/O functions, but still character-based. (This may or may not be what you want, see below.)

Notice how all the problems here are due to the inappropriate use of formatting functions and classes to perform something which requires that formatting not be done? There are a seemingly infinite number of solutions, and a few are listed here:

• "Derive your own fstream-type classes and write your own <</>> operators to do binary I/O on whatever data types you're using." This is a Bad Thing, because while the compiler would probably be just fine with it, other humans are going to be confused. The overloaded bitshift operators have a well-defined meaning (formatting), and this breaks it.
• "Build the file structure in memory, then mmap() the file and copy the structure." Well, this is easy to make work, and easy to break, and is pretty equivalent to using ::read() and ::write() directly, and makes no use of the iostream library at all...
• "Use streambufs, that's what they're there for." While not trivial for the beginner, this is the best of all solutions. The streambuf/filebuf layer is the layer that is responsible for actual I/O. If you want to use the C++ library for binary I/O, this is where you start.

How to go about using streambufs is a bit beyond the scope of this document (at least for now), but while streambufs go a long way, they still leave a couple of things up to you, the programmer. As an example, byte ordering is completely between you and the operating system, and you have to handle it yourself.

Deriving a streambuf or filebuf class from the standard ones, one that is specific to your data types (or an abstraction thereof) is probably a good idea, and lots of examples exist in journals and on Usenet. Using the standard filebufs directly (either by declaring your own or by using the pointer returned from an fstream's rdbuf()) is certainly feasible as well.

One area that causes problems is trying to do bit-by-bit operations with filebufs. C++ is no different from C in this respect: I/O must be done at the byte level. If you're trying to read or write a few bits at a time, you're going about it the wrong way. You must read/write an integral number of bytes and then process the bytes. (For example, the streambuf functions take and return variables of type int_type.)

Another area of problems is opening text files in binary mode. Generally, binary mode is intended for binary files, and opening text files in binary mode means that you now have to deal with all of those end-of-line and end-of-file problems that we mentioned before. An instructive thread from comp.lang.c++.moderated delved off into this topic starting more or less at this article and continuing to the end of the thread. (You'll have to sort through some flames every couple of paragraphs, but the points made are good ones.)

What is this <sstream>/stringstreams thing?

Stringstreams (defined in the header <sstream>) are in this author's opinion one of the coolest things since sliced time. An example of their use is in the Received Wisdom section for Chapter 21 (Strings), describing how to format strings.

The quick definition is: they are siblings of ifstream and ofstream, and they do for std::string what their siblings do for files. All that work you put into writing << and >> functions for your classes now pays off again! Need to format a string before passing the string to a function? Send your stuff via << to an ostringstream. You've read a string as input and need to parse it? Initialize an istringstream with that string, and then pull pieces out of it with >>. Have a stringstream and need to get a copy of the string inside? Just call the str() member function.

This only works if you've written your <</>> functions correctly, though, and correctly means that they take istreams and ostreams as parameters, not ifstreams and ofstreams. If they take the latter, then your I/O operators will work fine with file streams, but with nothing else -- including stringstreams.

If you are a user of the strstream classes, you need to update your code. You don't have to explicitly append ends to terminate the C-style character array, you don't have to mess with "freezing" functions, and you don't have to manage the memory yourself. The strstreams have been officially deprecated, which means that 1) future revisions of the C++ Standard won't support them, and 2) if you use them, people will laugh at you.

Deriving a stream buffer

Creating your own stream buffers for I/O can be remarkably easy. If you are interested in doing so, we highly recommend two very excellent books: Standard C++ IOStreams and Locales by Langer and Kreft, ISBN 0-201-18395-1, and The C++ Standard Library by Nicolai Josuttis, ISBN 0-201-37926-0. Both are published by Addison-Wesley, who isn't paying us a cent for saying that, honest.

Here is a simple example, io/outbuf1, from the Josuttis text. It transforms everything sent through it to uppercase. This version assumes many things about the nature of the character type being used (for more information, read the books or the newsgroups):

    #include <iostream>
    #include <streambuf>
    #include <locale>
    #include <cstdio>

    class outbuf : public std::streambuf
    {
      protected:
	/* central output function
	 * - print characters in uppercase mode
	 */
	virtual int_type overflow (int_type c) {
	    if (c != EOF) {
		// convert lowercase to uppercase
		c = std::toupper(static_cast<char>(c),getloc());

		// and write the character to the standard output
		if (putchar(c) == EOF) {
		    return EOF;
		}
	    }
	    return c;
	}
    };

    int main()
    {
	// create special output buffer
	outbuf ob;
	// initialize output stream with that output buffer
	std::ostream out(&ob);

	out << "31 hexadecimal: "
	    << std::hex << 31 << std::endl;
	return 0;
    }
   

More on binary I/O

Towards the beginning of February 2001, the subject of "binary" I/O was brought up in a couple of places at the same time. One notable place was Usenet, where James Kanze and Dietmar Kühl separately posted articles on why attempting generic binary I/O was not a good idea. (Here are copies of Kanze's article and Kühl's article.)

Briefly, the problems of byte ordering and type sizes mean that the unformatted functions like ostream::put() and istream::get() cannot safely be used to communicate between arbitrary programs, or across a network, or from one invocation of a program to another invocation of the same program on a different platform, etc.

The entire Usenet thread is instructive, and took place under the subject heading "binary iostreams" on both comp.std.c++ and comp.lang.c++.moderated in parallel. Also in that thread, Dietmar Kühl mentioned that he had written a pair of stream classes that would read and write XDR, which is a good step towards a portable binary format.

Pathetic performance? Ditch C.

It sounds like a flame on C, but it isn't. Really. Calm down. I'm just saying it to get your attention.

Because the C++ library includes the C library, both C-style and C++-style I/O have to work at the same time. For example:

     #include <iostream>
     #include <cstdio>

     std::cout << "Hel";
     std::printf ("lo, worl");
     std::cout << "d!\n";
      

Alert members of the audience will immediately notice that buffering is going to make a hash of the output unless special steps are taken.

The special steps taken by libstdc++, at least for version 3.0, involve doing very little buffering for the standard streams, leaving most of the buffering to the underlying C library. (This kind of thing is tricky to get right.) The upside is that correctness is ensured. The downside is that writing through cout can quite easily lead to awful performance when the C++ I/O library is layered on top of the C I/O library (as it is for 3.0 by default). Some patches are in the works which should improve the situation for 3.1.

However, the C and C++ standard streams only need to be kept in sync when both libraries' facilities are in use. If your program only uses C++ I/O, then there's no need to sync with the C streams. The right thing to do in this case is to call

     #include any of the I/O headers such as ios, iostream, etc

     std::ios::sync_with_stdio(false);
      

You must do this before performing any I/O via the C++ stream objects. Once you call this, the C++ streams will operate independently of the (unused) C streams. For GCC 3.0, this means that cout and company will become fully buffered on their own.

Note, by the way, that the synchronization requirement only applies to the standard streams (cin, cout, cerr, clog, and their wide-character counterparts). File stream objects that you declare yourself have no such requirement and are fully buffered.

Threads and I/O

I'll assume that you have already read the general notes on library threads, and the notes on threaded container access (you might not think of an I/O stream as a container, but the points made there also hold here). If you have not read them, please do so first.

This gets a bit tricky. Please read carefully, and bear with me.

Structure

As described here, a wrapper type called __basic_file provides our abstraction layer for the std::filebuf classes. Nearly all decisions dealing with actual input and output must be made in __basic_file.

A generic locking mechanism is somewhat in place at the filebuf layer, but is not used in the current code. Providing locking at any higher level is akin to providing locking within containers, and is not done for the same reasons (see the links above).

The defaults for 3.0.x

The __basic_file type is simply a collection of small wrappers around the C stdio layer (again, see the link under Structure). We do no locking ourselves, but simply pass through to calls to fopen, fwrite, and so forth.

So, for 3.0, the question of "is multithreading safe for I/O" must be answered with, "is your platform's C library threadsafe for I/O?" Some are by default, some are not; many offer multiple implementations of the C library with varying tradeoffs of threadsafety and efficiency. You, the programmer, are always required to take care with multiple threads.

(As an example, the POSIX standard requires that C stdio FILE* operations are atomic. POSIX-conforming C libraries (e.g, on Solaris and GNU/Linux) have an internal mutex to serialize operations on FILE*s. However, you still need to not do stupid things like calling fclose(fs) in one thread followed by an access of fs in another.)

So, if your platform's C library is threadsafe, then your fstream I/O operations will be threadsafe at the lowest level. For higher-level operations, such as manipulating the data contained in the stream formatting classes (e.g., setting up callbacks inside an std::ofstream), you need to guard such accesses like any other critical shared resource.

The future

As already mentioned here, a second choice is available for I/O implementations: libio. This is disabled by default, and in fact will not currently work due to other issues. It will be revisited, however.

The libio code is a subset of the guts of the GNU libc (glibc) I/O implementation. When libio is in use, the __basic_file type is basically derived from FILE. (The real situation is more complex than that... it's derived from an internal type used to implement FILE. See libio/libioP.h to see scary things done with vtbls.) The result is that there is no "layer" of C stdio to go through; the filebuf makes calls directly into the same functions used to implement fread, fwrite, and so forth, using internal data structures. (And when I say "makes calls directly," I mean the function is literally replaced by a jump into an internal function. Fast but frightening. *grin*)

Also, the libio internal locks are used. This requires pulling in large chunks of glibc, such as a pthreads implementation, and is one of the issues preventing widespread use of libio as the libstdc++ cstdio implementation.

But we plan to make this work, at least as an option if not a future default. Platforms running a copy of glibc with a recent-enough version will see calls from libstdc++ directly into the glibc already installed. For other platforms, a copy of the libio subsection will be built and included in libstdc++.

Alternatives

Don't forget that other cstdio implemenations are possible. You could easily write one to perform your own forms of locking, to solve your "interesting" problems.

See license.html for copying conditions. Comments and suggestions are welcome, and may be sent to the libstdc++ mailing list.