Skis / Toys / Fun

Recently read this great little article about doing map reduce over music.  What was interesting is that the author put up a test dataset for people to use, but of course had to caveat it with “don’t abuse”.   What’s interesting is that there is a host of interesting datasets out there:

• Music
• Wikipedia
• Some IMDB information (movies)
• …others…

One of the motos that has gotten embedded in corporate culture is “You are not your customer” as an engineer I see this all the time.  A designer builds something, engineering looks at it and says “but…”.  A manager responds with the flip, “The designer is the professional, you’re not the customer”.

With an attitude like that the message is abdicate your responsibility and focus on your job.  How can you build as successful company if all you do is your job, we need to think outside our role, challenge assumptions and build great products.  Since this is Steve Jobs week, he is noted for playing many roles he challenges the assumption that “you are not your customer”.  He is the customer he cares, if I’m building any product I need to put on my customer hat and think what I’m looking for.

How do you challenge “your are not your customer”?  You don’t need to challenge it, you need to demonstrate a care for the product, understand the use (and user) and live in their shoes.  It’s a form of empathy, we’ve been wired for thousands of years to care.  Bring up your experience, your understanding, your passion for great products.

We were working on some customer issues the other day and it another customer was having problems, you know double whammy.  What was interesting is that a more senior person started talking about the customer about “they’re such a problem” and then talking about strange things they did in the past.  As we dug into their problem, yes they were using an API in a very strange way (20 seconds of runtime to yield 6 results), but what was more interesting is that the more junior people started talking about how this customer was bad/wrong – taking on the way leadership has framed them.

In software we have an expectation of how somebody is going to do things, but we don’t have the “customer is always right” ideals that Customer Service/Sales takes to heart.  As wiser leaders we need to make sure that we frame how we talk about customers in ways that help people understand the basics of a customer centered development model (they’re always right).  Since regardless of what we think, it is what they want to do which keeps them happy and keeps things going forward.  Not only that but as we start to understand what they want to do, only then can we build systems that truly support their needs.

There is a whole different discussion about firing your customers, but thats an explicit option, not a cultural issue.

Looking at event IO frameworks and found some really good comments about the “ev” library, the challenge is that I would rather work in C++ than pure C – I like methods.. Found that the documentation is lacking for the C++ side of the library and needed to build a test harness.

This is a very basic TCP Echo server using libev in c++. The C++ side is only using std::list and the ev side is playing with loop, signals and basic socket IO. It should be a useful bread crumb for anybody trying to build their own service, which of course is my next big activity.

#include <unistd.h>
#include <fcntl.h>
#include <string.h>
#include <stdlib.h>
#include <ev++.h>
#include <netinet/in.h>
#include <sys/socket.h>
#include <resolv.h>
#include <errno.h>
#include <list>

//
//  Buffer class - allow for output buffering such that it can be written out
//                 into async pieces
//
struct Buffer {
    char    *data;
    ssize_t len;
    ssize_t pos;

    Buffer(const char *bytes, ssize_t nbytes) {
        pos = 0;
        len = nbytes;
        data = new char[nbytes];
        memcpy(data, bytes, nbytes);
    }

    virtual ~Buffer() {
        delete [] data;
    }

    char *dpos() {
        return data + pos;
    }

    ssize_t nbytes() {
        return len - pos;
    }
};

//
//  A single instance of a non-blocking Echo handler
//
class EchoInstance {
private:
    ev::io      io;
    static int total_clients;

    // Buffers that are pending write
    std::list<Buffer*>   write_queue;

    // Generic callback
    void callback(ev::io &watcher, int revents) {
        if (EV_ERROR & revents) {
            perror("got invalid event");
            return;
        }

        if (revents & EV_READ)
            read_cb(watcher);

        if (revents & EV_WRITE)
            write_cb(watcher);

        if (write_queue.empty()) {
            io.set(ev::READ);
        } else {
            io.set(ev::READ|ev::WRITE);
        }
    }

    // Socket is writable
    void write_cb(ev::io &watcher) {
        if (write_queue.empty()) {
            io.set(ev::READ);
            return;
        }

        Buffer* buffer = write_queue.front();

        ssize_t written = write(watcher.fd, buffer->dpos(), buffer->nbytes());
        if (written < 0) {
            perror("read error");
            return;
        }

        buffer->pos += written;
        if (buffer->nbytes() == 0) {
            write_queue.pop_front();
            delete buffer;
        }
    }

    // Receive message from client socket
    void read_cb(ev::io &watcher) {
        char    buffer[1024];

        ssize_t  nread = recv(watcher.fd, buffer, sizeof(buffer), 0);

        if (nread < 0) {
            perror("read error");
            return;
        }

        if (nread == 0) {
            // Gack - we're deleting ourself inside of ourself!
            delete this;
        } else {
            // Send message bach to the client
            write_queue.push_back(new Buffer(buffer, nread));
        }
    }

    // effictivly a close and a destroy
    virtual ~EchoInstance() {
        // Stop and free watcher if client socket is closing
        io.stop();

        printf("%d client(s) connected.\n", --total_clients);
    }

public:
    EchoInstance(int s) {
        fcntl(s, F_SETFL, fcntl(s, F_GETFL, 0) | O_NONBLOCK); 

        printf("Got connection\n");
        total_clients++;

        io.set<EchoInstance, &EchoInstance::callback>(this);

        io.start(s, ev::READ);
    }
};

class EchoServer {
private:
    ev::io      io;
    ev::sig     sio;
    int         s;

public:

    void io_accept(ev::io &watcher, int revents) {
        if (EV_ERROR & revents) {
            perror("got invalid event");
            return;
        }

        struct sockaddr_in client_addr;
        socklen_t client_len = sizeof(client_addr);

        int client_sd = accept(watcher.fd, (struct sockaddr *)&client_addr, &client_len);

        if (client_sd < 0) {
            perror("accept error");
            return;
        }

        EchoInstance *client = new EchoInstance(client_sd);
    }

    static void signal_cb(ev::sig &signal, int revents) {
        signal.loop.break_loop();
    }

    EchoServer(int port) {
        printf("Listening on port %d\n", port);

        struct sockaddr_in addr;

        s = socket(PF_INET, SOCK_STREAM, 0);

        addr.sin_family = AF_INET;
        addr.sin_port   = htons(port);
        addr.sin_addr.s_addr = INADDR_ANY;

        if (bind(s, (struct sockaddr *)&addr, sizeof(addr)) != 0) {
            perror("bind");
        }

        fcntl(s, F_SETFL, fcntl(s, F_GETFL, 0) | O_NONBLOCK); 

        listen(s, 5);

        io.set<EchoServer, &EchoServer::io_accept>(this);
        io.start(s, ev::READ);

        sio.set<&EchoServer::signal_cb>();
        sio.start(SIGINT);
    }

    virtual ~EchoServer() {
        shutdown(s, SHUT_RDWR);
        close(s);
    }
};

int EchoInstance::total_clients = 0;

int main(int argc, char **argv)
{
    int     port = 8192;

    if (argc > 1)
        port = atoi(argv[1]);

    ev::default_loop    loop;
    EchoServer          echo(port);

    loop.run(0);

    return 0;
}

Waterfall bad, right?  Well, to an extent it is — the good part about waterfall is that you start with a objective and build a team around an objective.  The challenge is that the interm points and progress measurements get caught up in gantt charts, absolute assignments, critical paths, etc.etc.  Of course waterfall is all about time and deadlines, so when things fall behind it’s all about finger pointing and fall guys.

Scrum is good, semi-self organized teams working on short lists of tasks.  Continual measurement of progress, yadda yadda yadda.  But, then once you’ve released what happens…   You have a two week sprint, a bunch of things a product person, if you’re lucky an engineer, has added to the sprint queue…  Rinse and repeat — but somebody has forgotten the biggest piece.

What’s the Goal?

Waterfall – simple goal is a PRD … Scrum to a point is a PRD (yes, lean ideals, it changes).  What you need to remember to create in a series of sprints is a goal for the team — why are they doing this work, to what end… If it is just the features of the week, you need to take a step back and figure out if you’re in a conifer forest or a deciduous forest, since you’re not seeing anything except trees.