Adventures in Redirection, Part Deux

In the previous chapter, I showed how to redirect cout and cerr, and how it turns out that doing so does not also redirect stdout and stderr. As this turned out not to meet my requirements, I went in search of other methods.

In Googling around, the most common way to redirect stdout/stderr involves calling freopen on them. However, this didn’t work for me, for a few reasons. For one thing, freopen is really geared toward redirecting to a file. You can engage in a bit of trickery to redirect to memory instead, but I wasn’t really interested in going down that path. Worse, it’s completely irreversible; there’s no portable way to restore stdout/stderr once you redirect them via freopen. The ability to arbitrarily redirect and restore output was a must-have for me, so freopen couldn’t cut it.

I finally settled on a method which involves calling dup2 to duplicate the stdout/stderr file descriptors. In this way, you can redirect to one end of a pipe, then read from the other end whenever it’s convenient. More importantly, you can save off the old file descriptors so that you can restore them. Let’s get right to it. You might remember the IOutputRedirector interface from last time:

class IOutputRedirector
{
public:
            IOutputRedirector() {}
    virtual ~IOutputRedirector(){}

    virtual void StartRedirecting() = 0;
    virtual void StopRedirecting() = 0;

    virtual std::string GetOutput() = 0;
    virtual void        ClearOutput() = 0;
};

For our new class, we just need a couple of ints to hold our pipe file numbers, a couple of ints for the saved stdout/stderr file numbers, a bool to indicate that we’re redirecting, and a std::string which we redirect into.

class CStdoutRedirector : public IOutputRedirector
{
public:
            CStdoutRedirector();
    virtual ~CStdoutRedirector();

    virtual void StartRedirecting();
    virtual void StopRedirecting();

    virtual std::string GetOutput();
    virtual void        ClearOutput();

private:
    int    _pipe[2];
    int    _oldStdOut;
    int    _oldStdErr;
    bool   _redirecting;
    std::string _redirectedOutput;
};

In the constructor we create a pipe, and call dup to copy the stdout and stderr file descriptors, so that we can restore them later. The setbuf calls are there to disable buffering. stdout is typically line-buffered whilst stderr is typically unbuffered, resulting in the combined output not being properly interleaved otherwise.

CStdoutRedirector::CStdoutRedirector()
: IOutputRedirector()
, _oldStdOut( 0 )
, _oldStdErr( 0 )
, _redirecting( false )
{
    _pipe[READ] = 0;
    _pipe[WRITE] = 0;
    if( pipe( _pipe ) != -1 ) {
        _oldStdOut = dup( fileno(stdout) );
        _oldStdErr = dup( fileno(stderr) );
    }

    setbuf( stdout, NULL );
    setbuf( stderr, NULL );
}

The destructor does some cleanup (stopping the redirection and closing files).

CStdoutRedirector::~CStdoutRedirector()
{
    if( _redirecting ) {
        StopRedirecting();
    }

    if( _oldStdOut > 0 ) {
        close( _oldStdOut );
    }
    if( _oldStdErr > 0 ) {
        close( _oldStdErr );
    }
    if( _pipe[READ] > 0 ) {
        close( _pipe[READ] );
    }
    if( _pipe[WRITE] > 0 ) {
        close( _pipe[WRITE] );
    }
}

StartRedirecting is where part of the magic is. We call dup2 to copy the stdout and stderr file descriptors to the write end of our pipe. This is how the redirection works: after the dup2 calls, any writes to stdout/stderr go instead to the write end of our pipe.

void
CStdoutRedirector::StartRedirecting()
{
    if( _redirecting ) return;

    dup2( _pipe[WRITE], fileno(stdout) );
    dup2( _pipe[WRITE], fileno(stderr) );
    _redirecting = true;
}

StopRedirecting copies the stdout and stderr file descriptors back to the ones we saved in the constructor, resulting in restoring stdout/stderr.

void
CStdoutRedirector::StopRedirecting()
{
    if( !_redirecting ) return;

    dup2( _oldStdOut, fileno(stdout) );
    dup2( _oldStdErr, fileno(stderr) );
    _redirecting = false;
}

GetOutput is where the rest of the magic lies. We call fcntl to set the read end of our pipe to nonblocking mode (otherwise the read call will block if the pipe is empty). Then we call read in a loop into a little temporary buffer, NULL terminate it, and append it to a string, which we then return.

string
CStdoutRedirector::GetOutput()
{
    const size_t bufSize = 4096;
    char buf[bufSize];
    fcntl( _pipe[READ], F_SETFL, O_NONBLOCK );
    ssize_t bytesRead = read( _pipe[READ], buf, bufSize - 1 );
    while( bytesRead > 0 ) {
        buf[bytesRead] = 0;
        _redirectedOutput += buf;
        bytesRead = read( _pipe[READ], buf, bufSize );
    }

    return _redirectedOutput;
}

ClearOutput simply clears out the string.

void
CStdoutRedirector::ClearOutput()
{
    _redirectedOutput.clear();
}

There is one caveat to this method: if the internal buffer fills up, subsequent output (printf, for example) will block until space is freed up in the buffer. This buffer appears to be 16k in size, though I imagine it varies depending on OS. So basically, if you intend to redirect more than 16k at once. well… that won’t work. Calling GetOutput clears the buffer out, so call it often, if you can.

I was thinking that supplying my own larger buffer via setvbuf() would help, but it turns out not to. If anyone has any ideas on how to make this better, let me know. I would also point out that I’ve not used this in production code (I am using it in the project I’m working on now, and it seems to be working perfectly; we just haven’t shipped it yet).

I’ll put the full source and sample project up on github when I have a spare few minutes. Update: Full source and sample project are on github, here.

If you want to follow me, I’m @zpasternack on Twitter and on app.net.

stdout and stderr: Adventures in Redirection, Part 1

For a project I’m working on, I needed to redirect the standard output and standard error of a subprocess I’m launching. While it’s trivial to do this with NSTask (just send it setStandardOutput: and setStandardError:), I had some unique requirements that precluded this. Mainly, I didn’t want the process’ stdout/stderr redirected for its entire lifespan. Rather, I wanted it to redirect only certain parts, and return that data to the GUI through a separate mechanism (a named pipe).

So, what I wanted was to be able to was:

  1. Enable/disable the redirection at any point, while the process is running (this more or less precludes freopen– based solutions, as there’s no portable way to restore the streams once they’re reopened).

  2. Be able to retrieve and clear the redirected output at any time, regardless of whether redirection is enabled or not.

  3. stdout and stderr must both be redirected to the same buffer, and properly interleaved.

  4. The helper app (the app that has this code) is written in C++; no Cocoa, no Objective-C.

Based on those requirements, I wrote the following unit test to determine what my interface should look like.

int RedirectorTest()
{
    IOutputRedirector theRedirector;
    std::cout << "This is not redirected" << std::endl;

    theRedirector.StartRedirecting();
    std::cout << "This is redirected";
    assert( theRedirector.GetOutput() == "This is redirected"
        && "cout not redirected!" );
    theRedirector.ClearOutput();
    assert( theRedirector.GetOutput().length() == 0
        && "redirector not cleared" );

    std::cerr << "Redirected Error";
    assert( theRedirector.GetOutput() == "Redirected Error"
        && "cerr not redirected" );
    theRedirector.ClearOutput();
    assert( theRedirector.GetOutput().length() == 0
        && "redirector not cleared" );

    theRedirector.StopRedirecting();
    std::cout << "This is also not redirected";
    assert( theRedirector.GetOutput().length() == 0
        && "redirection not stopped!" );

    theRedirector.StartRedirecting();
    std::cout << "cout";
    std::cerr << "cerr";
    std::cout << "cout again";
    std::string output = theRedirector.GetOutput();
    assert( theRedirector.GetOutput() == "coutcerrcout again"
        && "cout, err not interleaved properly" );

    return 0;
}

Based on that test I wrote the following interface I’d use for my test class:

class IOutputRedirector
{
public:
               IOutputRedirector() {}
    virtual    ~IOutputRedirector(){}

    virtual void    StartRedirecting() = 0;
    virtual void    StopRedirecting()  = 0;

    virtual std::string GetOutput()    = 0;
    virtual void        ClearOutput()  = 0;
};

I Googled around for various techniques on how to do this, and the most promising one involved calling rdbuf on the cout and cerr streams. In this way, you can specify a different streambuf, and I use that of an ostringstream. It looks a little somethin’ like this:

class CCoutRedirector : public IOutputRedirector
{
public:
            CCoutRedirector();
    virtual ~CCoutRedirector();

    virtual void    StartRedirecting();
    virtual void    StopRedirecting();

    virtual std::string GetOutput();
    virtual void        ClearOutput();

private:
    std::streambuf* m_pOldCoutStreamBuf;
    std::streambuf* m_pOldCerrStreamBuf;
    std::ostringstream m_stream;
};

CCoutRedirector::CCoutRedirector()
: IOutputRedirector()
, m_pOldCoutStreamBuf( NULL )
, m_pOldCerrStreamBuf( NULL )
{
}

CCoutRedirector::~CCoutRedirector()
{
    StopRedirecting();
}

void
CCoutRedirector::StartRedirecting()
{
    if( m_pOldCoutStreamBuf != NULL ||
        m_pOldCerrStreamBuf != NULL )
    {
        return;
    }

    m_pOldCoutStreamBuf = std::cout.rdbuf();
    m_pOldCerrStreamBuf = std::cerr.rdbuf();

    streambuf* newOutbuf = m_stream.rdbuf();
    cout.rdbuf( newOutbuf );
    cerr.rdbuf( newOutbuf );
}

void
CCoutRedirector::StopRedirecting()
{
    if( m_pOldCoutStreamBuf != NULL )
    {
        cout.rdbuf( m_pOldCoutStreamBuf );
        m_pOldCoutStreamBuf = NULL;
    }

    if( m_pOldCerrStreamBuf != NULL )
    {
        cerr.rdbuf( m_pOldCerrStreamBuf );
        m_pOldCerrStreamBuf = NULL;
    }
}

string
CCoutRedirector::GetOutput()
{
    return m_stream.str();
}

void
CCoutRedirector::ClearOutput()
{
    m_stream.str("");
}

It’s pretty self explanatory. StartRedirecting saves off the old cout and cerr’s streambufs, and replaces them with that of our own ostringstream. GetOutput returns the string from that stream; ClearOutput clears it, and finally, StopRedirecting restores the previous streambufs. Easy as pi.

With the test I wrote above this works flawlessly. There’s only one major issue, as I later discovered: while this works for cout and cerr, it does NOT work for stdout and stderr in general. Meaning, calls to printf will not be redirected. I’d not realized that stdout and cout weren’t synonymous. For me this was a deal-breaker, but perhaps others will find it useful.

In the next post I’ll show the technique I ended up using, that works for stdout/stderr as well as cout/cerr.

If you want to follow me, I’m @zpasternack on Twitter and on app.net.

Stuff I Can’t Live Without Part I: conditional NSLogging

I spend a great deal of time watching my code. By that I mean either stepping through it in the debugger, or reading logs generated by it to see what it’s done. You might think you know exactly what your software is doing, but the truth is, if you don’t watch it run, you don’t know.

During the course of development, I usually end up adding a fair amount of logging code, and of course I don’t want that in my production builds. So a while back I went a-Googling for some conditional NSLog calls, and found these great macros, courtesy of Nick Dalton’s iPhoneIncubator. I use them (and repeat them below) nearly verbatim. This is the first thing I add to any project I’m working on. I’m naked without my DebugLog (and not in the good way).

// DebugLog is almost a drop-in replacement for NSLog
// DebugLog();
// DebugLog(@“here”);
// DebugLog(@“value: %d”, x);
// Unfortunately this doesn’t work: DebugLog(aStringVariable); you have to do this instead: DebugLog(@“%@“, aStringVariable);
// Shamelessly stolen (and edited) from http://iphoneincubator.com/blog/debugging/the-evolution-of-a-replacement-for-nslog

#ifdef DEBUG
    #define DebugLog(fmt, ...) NSLog((@“%s:%d “ fmt), __PRETTY_FUNCTION__, __LINE__, ##__VA_ARGS__);
    #define DebugLogFunc() NSLog( @“%s:%d”, __PRETTY_FUNCTION__, __LINE__ );
#else
    #define DebugLog(...)
    #define DebugLogFunc()
#endif

// AlwaysLog always displays output regardless of the DEBUG setting
#define AlwaysLog(fmt, ...) NSLog((@“%s:%d “ fmt), __PRETTY_FUNCTION__, __LINE__, ##__VA_ARGS__);

Of course this assumes you have DEBUG #defined in your debug builds somewhere (in Xcode 4 do this in Preprocessor Macros in Build Settings).

The only changes I made over Nick’s is renaming them (I prefer clarity over brevity), minor formatting, and adding the …Func macros for the common (for me) case of simply outputting the function name with no further description. This is useful for making sure the functions you expect to get called, really are.

If you want to follow me, I’m @zpasternack on Twitter and on app.net.