I have been working through some of my older pieces lately. Making sure they are a higher resolution and tweaking the designs slightly fixing things I am unhappy with.

First through the changes is Second Lives which has seen significant changes. I will be offering a print of the older version soon for those of you who prefer the older version.

Print is now available through the print store.

Yes indeed, Grey Skies is pleased to be now offering prints of the various poster designs I have worked on.
I have been wanting to offer prints for a while now and have spent the last month or so researching several providers trying to find one that would work. Finally it was narrowed down (with some helpful advice from a fellow artist) to Society6.

Currently there is only a small selection on offer but I hope to grow that over the course of the year.

To the many people who have been poking me about the Skyrim piece, at the moment I have no plans to sell that unless I can work something out with Bethesda.

There is now a Starry Expanse Hood in MOULa. If you’d like to join, catch someone from 59 Volt Entertainment online (like Zib or ironmagma), or find someone who’s already a member. If nobody’s online, just send us an e-mail, and one of us will hop online to help you out. Spread the love!

Hey there – Just wanted to let you guys know that work is continuing, despite our silence. I’ve posted a couple of things to our forum in the last week, and I figured I’d repost them here. They depict (very preliminary) work on the new sky system, and some rigging work on Catherine:

We’re not dead! We haven’t given up! Thanks for continuing to believe in us!

That being said, we could really use another experienced programmer or two. We’d like help developing both the engine, and some backend tools for us to manage the project. If you’re interested in helping out (in this or any other way), just send us an e-mail, and we’ll let you know if we can use you.

We were given the complete set of Harry Potter movies for Christmas. So we’ve been watching our way through them with the kids. As a result this mornings speed painting was of some random dark wizard.

Done in Alchemy with some colour tweaks in Photoshop, took about an hour.

Today we open sourced Dreadnot, our take on a Continuous Deployment tool. Details are posted over on the Rackspace Cloud Blog

Source is up on github.com/racker/dreadnot.

Work Project:

Hobby Project:

2012 Goal

Finish the hobby project.

Created using Dustin’s git-timecard.

I realized the moment I fell into the fissure that the book would not be destroyed as I had planned. It continued falling into that starry expanse, of which I had only a fleeting glimpse. I have tried to speculate where it might have landed, but I must admit that such conjecture is futile. Still, questions about whose hands might one day hold my Myst book are unsettling to me. I know my apprehensions might never be allayed, and so I close, realizing that perhaps the ending has not yet been written.

Bien que mariés depuis quelques temps déjà, T’chérie d’Amûr et moi ne portions pas d’alliance, celles que nous avions vu étaient trop ringardes classiques et nous n’attachons que peu d’importance à ces « marques de propriété ».

 

C’était sans compter avec le Père Noël (en l’occurrence un des frères de ma Mie) qui nous a dégotté Zi Object !

Tous les deux amateurs d’esthétique Steampunk et de la saga Myst, nous ne pouvions que craquer devant cet assemblage de pignons et de crémaillères mobiles. Voilà de bien belles étrennes !

Je crois que je vais m’amuser à la modéliser en 3D, pour le plaisir…

Le Père Noël n’existe pas ? D’accord.

Mais s’est-on jamais donné la peine de démonter pièce par pièce cette supercherie vieille de 150 ans ? Le mensuel new-yorkais Spy l’a fait avec une rigueur toute scientifique :

——————————————————————————————————————–

Croyez-vous au Père Noël ?
C’est une question théologique complexe à laquelle chaque enfant devait jusque-là répondre seul. Plus maintenant en tout cas, puisque nous avons mené une enquête statistique rigoureuse sur la question de l’existence du Père Noël.
 Nous préférons vous en avertir : vous n’allez peut-être pas en apprécier les conclusions…

Nous sommes donc partis du principe que le Père Noël existe vraiment.

Si vous connaissez un tant soit peu la nature humaine, vous savez qu’il est fort improbable qu’un homme normal choisisse, sans raison particulière, de consacrer sa vie à fabriquer des jouets et à les distribuer aux petits garçons et aux petites filles du monde entier. Mais comme il s’agit d’une enquête objective, les questions de motivation n’ont pas à être prises en compte. Nous voulons seulement savoir si un tel homme pourrait accomplir sa mission.

Premier obstacle pour le Père Noël : aucune espèce connue de renne n’est capable de voler.
Cependant, les scientifiques estiment que, sur les quelque deux millions d’espèces vivantes qui habitent la Terre, environ 300 000 n’ont pas encore été répertoriées. Même si la plupart de ces espèces encore inconnues sont des insectes et des bactéries, nous ne pouvons éliminer la faible possibilité qu’une espèce de renne volant existe réellement. Et que personne – à l’exception du Père Noël – n’en ait jamais vu un seul.

Autre obstacle de taille pour le Père Noël : il y a 2 milliards d’enfants de moins de 18 ans dans le monde.
La bonne nouvelle, c’est qu’il n’apporte des cadeaux qu’aux enfants chrétiens, qui sont à peu près 378 millions (selon les chiffres du Population Reference Bureau). Admettons que 15 % de ces enfants chrétiens n’ont pas été sages et que, comme les enfants musulmans, hindous, juifs et bouddhistes, ils ne peuvent recevoir de cadeaux. Il n’en reste pas moins qu’avec une moyenne de 3,5 enfants par famille, pour une nuit de Noël donnée, le Père du même nom a la tache éreintante de visiter 91,8 millions de foyers.

Heureusement, à Noël, grâce à la rotation terrestre, il dispose de 31 heures d’obscurité pour faire sa tournée s’il voyage d’est en ouest. Malheureusement, cela donne encore quelque 822,6 maisonnées a visiter par seconde. Donc, pour chaque foyer chrétien ou les enfants ont été sages, le Père Noël dispose d’un peu plus d’un millième de seconde pour se poser, bondir de son traîneau, déposer ses cadeaux, décoller et voler jusqu’à la prochaine maison.

À quelle vitesse se déplace le Père Noël ? En admettant que les 91,8 millions d’arrêts sont également repartis sur les continents, le Père Noël doit parcourir 1,2 km entre chaque foyer, soit un voyage de 121 155 200 km (estimation modérée, ne tenant pas compte des déplacements transocéaniques, des arrêts fourrage pour les rennes, etc.). Pour accomplir son périple en 31 heures, le traîneau du Père Noël doit maintenir une vitesse de 1 040 km/s, soit 3 000 fois la vitesse du son.

Pour vous donner une idée, le véhicule le plus rapide jamais fabriqué par l’homme, la sonde spatiale Ulysse, se déplace à la vitesse relativement étriquée de 43,8 km/s, et un renne de type conventionnel se déplace sur terre à une vitesse maximale de 24 km/h. Mais on peut supposer que les rennes volants du Père Noël sont capables d’une façon ou d’une autre d’atteindre des vitesses hypersoniques grâce, disons, à l’esprit magique de la nuit de Noël.

Considérons d’un peu plus près le véhicule du Père Noël. D’abord, en partant du principe qu’il transporte un kilo de jouets pas chers par enfant (une boite de Lego minable, par exemple), le traîneau doit quand même avoir une capacité de 321 300 tonnes, plus le Père Noël, qui souffre d’un léger embonpoint. Sur terre, un renne ne peut pas tirer plus de 150 kg de fret et, même en admettant que des rennes volants soient capables de tirer 10 fois cette charge, l’énorme traîneau du Père Noël devrait être remorqué par 214 200 animaux. Ces derniers augmentent la masse totale du véhicule à 353 430 tonnes (sans compter le poids du traîneau lui-même), soit quatre fois plus que le paquebot Queen Elizabeth.

Imaginez le Père Noël faisant du rase-mottes au-dessus des toits dans un gigantesque aéronef hypersonique, encore moins maniable qu’une grande roue. C’est la que les choses deviennent amusantes. Avec 353 430 tonnes de rennes et de cadeaux, le traîneau crée une incroyable résistance à l’air, surtout à 1 040 km/s. Cette résistance échauffera les rennes de la même façon que les vaisseaux spatiaux sont chauffés en entrant dans l’atmosphère terrestre.

D’après nos calculs, la première paire de rennes absorbera 14,3 quintillions (14,3 * 10^30) de joules d’énergie par seconde.
Ce qui signifie qu’ils exploseront presque instantanément dans de magnifiques jets de flammes multicolores, exposant du même coup les rennes suivants. Au fur et à mesure de sa mission, le Père Noël laissera derrière lui un sillage de bangs soniques assourdissants et une cohorte ininterrompue de rennes carbonisés. Les 214 200 bêtes seront tuées en 4,26 millièmes de seconde.

Quand au Père Noël, il sera soumis à des forces centrifuges 17 500,06 fois supérieures à l’attraction terrestre. Un Père Noël d’environ 125 kg sera cloué au fond de son traîneau par une pression de 2 187 507,5 kg
(une fois son poids déduit). Cette force le tuera sur le coup, lui broyant les os, pulvérisant sa chair, le transformant en gelée rose.

En d’autres termes, si le Père Noël essaie de distribuer des cadeaux le soir de Noël à tous les petits garçons et à toutes les petites filles qui le méritent sur la surface de le Terre, il sera liquéfié. Si jamais il existe, il est déjà mort.

CQFD !

Petite traduction rapide du laïus U&I Software :

ArtMatic 5 & ArtMatic Voyager 2 pour Mac OS

ArtMatic 5 et ArtMatic Voyager 2 sont là. La nouvelle génération de nos logiciels graphiques apporte la possibilité de créer de « vrais » objets 3D et des éléments de paysage, en plus d’un lot de nouvelles fonctionnalités.

Utilisez-les séparément ou ensemble pour créer nombre  d’images et d’animations fabuleuses.

Ces vidéos ont été calculées dans ArtMatic Voyager sans aucun compositing.

Pour en savor plus sur ArtMatic 5, et sur ArtMatic Voyager 2.

Samedi dernier aux alentours de 15h00, Tchériedamûr et wam avons vu un truc très rare dans nos contrées : un arc-en-ciel horizontal aussi appelé « Fire rainbow » (littéralement : arc-en-ciel de feu) ou « circumhorizontal arc ».
C’est évidemment dans ces moments-là qu’on regrette de n’avoir pas d’appareil photo sous la main…

Ce phénomène se produit dans des conditions exceptionnelles :

• strictement en présence de cirrus contenant des cristaux de glace plats qui doivent, de plus,  être correctement alignés pour réfracter la lumière
• un soleil à plus de 58° d’élévation
• uniquement entre les latitudes 55° N et S !!

Autrement-dit, gros coup de bol d’avoir vu ça (il semble que ce soit moins rare dans certains états des USA)
C’était superbe et très surprenant !

The Temple of the Falling Star is a small temple within the Vaal’lin city and sister to the Temple of the Rising Star which resides in another Age. The two temples are linked together by a thread that goes beyond space and time.

For Christmas this year I acquired a few props & replicas. (click to embiggen)




From Back to the Future: part II, replicas of the lenticular rainbow hat plus Grays Sports Almanac. The Almanac even came with a replica plastic receipt! I think I'll just leave these lying behind my DeLorean's passenger seat.


I also acquired a collection of screen-used Stargate props. Clockwise, from the top left: Goa’uld mothership wall panel piece, Jaffa armour piece, Replicator block wall piece, actual piece of the Stargate’s Iris, and finally an actual piece of the Stargate itself.

Logging for Humans

A printf style format string is the de facto method of logging for almost all software written in the last 20 years. This style of logging crosses almost all programing language boundaries. Many libraries build upon this, adding log levels and various transports, but they are still centered around a formated string.

I believe the widespread use of format strings in logging is based on two presumptions:

1. The first level consumer of a log message is a human.
2. The programer knows what information is needed to debug an issue.

I believe these presumptions are no longer correct in server side software.

An example of the problem

An example is this classic error message inside the Apache HTTP Server. The following code is called any time a client hits a URL that doesn’t exist on the file system:

ap_log_rerror(APLOG_MARK, APLOG_INFO, 0, r,
  "File does not exist: %s", r->filename);

This would generate a log message like the following in your error.log:

[Mon Dec 26 09:14:46 2011] [info] [client 50.57.61.4]
  File does not exist: /var/www/no-such-file

This is fine for human consumption, and for decades people have been writing Perl scripts to munge it into fields for a computer to understand too. However, the first time you add a field, for example the HTTP User-Agent header, it would break most of those perl scripts. This is one example of where building a log format that is optimized for computer consumption starts to make sense.

Another problem is when you are writing these format string log messages, you don’t always know what information people will need to debug the issue. Since you are targeting them for human consumption you try to reduce the information overload, and you make a few guesses, like the path to the file, or the source IP address, but this process is error prone. From my experience in the Apache HTTP server this would mean opening GDB to trace what is happening. Once you figure out what information is relevant, you modify the log message to improve the output for future users with the relevant information.

What if we logged everything into JSON?

If we produced a JSON object which contained the same message, it might look something like this:

{
    "timestamp": 1324830675.076,
    "status": "404",
    "short_message": "File does not exist: /var/www/no-such-file",
    "host": "ord1.product.api0",
    "facility": "httpd",
    "errno": "ENOENT",
    "remote_host": "50.57.61.4",
    "remote_port": "40100",
    "path": "/var/www/no-such-file",
    "uri": "/no-such-file",
    "level": 4,
    "headers": {
        "user-agent": "BadAgent/1.0",
        "connection": "close",
        "accept": "*/*"
    },
    "method": "GET",
    "unique_id": ".rh-g2Tm.h-ord1.product.api0.r-axAIO3bO.c-9210.ts-1324830675.v-24e946e"
}

This example gives a much richer picture of information about the error. We now have data like the User-Agent in an easily consumable form, we could much more easily figure out that BadAgent/1.0 is the cause of our 404s. Other information like the source server and a mod_unique_id hash can be used to correlate multiple log entries across the lifetime of an request.

This information is also expandable. As the knowledge of what our product needs to log increases, it is easy to add more data, and we can safely do this without breaking our System Admins precious Perl scripts.

Why now?

This idea is not new, it has just never been so easily accessible. Windows has had “Event Logs” for a decade, but in the more recent versions it uses XML. The emergence of JSON as a relatively compact serialization format that can be generated and parsed from almost any programming languages means it makes a great light weight interchange format.

Paralleling the big data explosion, is a growth in machine and infrastructure size. This means logging and the ability to spot errors in a distributed system has become even more valuable.

Logging objects instead of a format string enables you to more easily index and trace operations across hundreds of different machines and different software systems. With traditional format strings it is too fail deadly for the programmer to not log all the necessary information for a later operator to trace an operation.

Generating JSON with Log Magic

Log Magic is a small and fast logging library for Node.js that I wrote early on for our needs at Rackspace. It only has a few features, and it is only about 300 lines of code.

Log Magic has the concept of a local logger instance, which is used by a single module for logging. A logger instance automatically populates information like the the facility in a log entry. Here is an example of creating a logger instance for a module named 'myapp.api.handler and using it:

var log = require('logmagic').local('myapp.api.handler');

exports.badApiHandler = function(req, res) {
  log.dbg("Something is wrong", {request: req});
  res.end();
};

The second feature that Log Magic provides is what I call a “Log Rewriter”. This enables the programmer to just consistently pass in the request object, and we will take care of picking out the fields we really want to log. In this example, we ensure the logged object always has an accountId and txnId fields set:

var logmagic = require('logmagic');
logmagic.addRewriter(function(modulename, level, msg, extra) {
  if (extra.request) {
    if (extra.request.account) {
      extra.accountId = extra.request.account.getKey();
    }
    else {
      /* unauthenticated user */
      extra.accountId = null;
    }
    extra.txnId = extra.request.txnId;
    delete extra.request;
  }
  return extra;
});

The final feature of Log Magic is dynamic routes and sinks. For the purposes of this article, we are mostly interested in the graylog2-stderr, which outputs a GELF JSON format message to stderr:

var logmagic = require('logmagic');
logmagic.route('__root__', logmagic['DEBUG'], "graylog2-stderr");

With this configuration, if we ran that log.dbg example from above, we would get a message like the following:

{
    "version": "1.0",
    "host": "product-api0",
    "timestamp": 1324936418.221,
    "short_message": "Something is wrong",
    "full_message": null,
    "level": 7,
    "facility": "myapp.api.handler",
    "_accountId": "ac42",
    "_txnId": ".rh-3dT5.h-product-api0.r-pVDF7IRM.c-0.ts-1324936588828.v-062c3d0"
}

Other implementations

There are many other libraries that are starting to emerge that can output logs in a JSON or GELF format:

• winston: (Node.js) A more complete (or complex?) logging module compared to Log Magic, but the prolific crew at Nodejitsu have done a great job.
• graypy: (Python) A graylog2 logger that interacts with the standard Python logging module.
• gelf4j (Java) We use a modified version of this library that logs to stderr instead of using UDP.

The Transaction Id

One field we added very early on to our system was what we called the “Transaction Id” or txnId for short. In retrospect, we could of picked a better name, but this is essentially a unique identifier that follows a request across all our of services. When a User hits our API we generate a new txnId and attach it to our request object. Any requests to a backend service also include the txnId. This means you can clearly see how a web request is tied to multiple backend service requests, or what frontend request caused a specific Cassandra query.

We also send the txnId to our user’s in our 500 error messages and the X-Response-Id header, so if a user reports an issue, we can quickly see all of the related log entries.

While we treat the txnId as an opaque string, we do encode a few pieces of information into it. By putting the current time and the origin machine into the txnId, even if we can’t figure out what went wrong from searching for the txnId, we have a place to start deeper debugging.

Transporting Logs

Since our product spans multiple data centers, and we don’t trust our LAN networking, our primary goal is that all log entries hit disk on their origin machine first. Some people have been using UDP or HTTP for their first level logging, and I believe this is a mistake. I believe having a disk default that consistently works is critical in a logging system. Once our messages have been logged locally, we stream them to an aggregator which then back hauls the log entries to various collection and aggregation tools.

Since all of our services run under runit, our programs simply log their JSON to stderr, and svlogd takes care of getting the data into a local file. Then we use a custom tool written in Node.js that is like running a tail -F on the log file, sending this data to a local Scribe instance. The Scribe instance is then responsible for transporting the logs to our log analyzing services.

For locally examining the log files generated by svlogd, we also made a tool called gelf-chainsaw. Since JSON strings cannot contain a newline, the log format is easy to parse, you just split up the file by \n, and try to JSON.parse each line. This is useful for our systems engineers when they are on a single machine, trying to debug an issue.

Collecting, Indexing, Searching

Once the logs crossing machines, there are many options to process those logs. Some examples that can all accept JSON as their input format:

• Perl Scripts (Hah! Did you think Perl will ever go away?)
• Graylog2 (open source)
• LogStash (open source)
• Loggly (SaaS)
• Splunk (Proprietary Software, can do JSON with an extra tool)

For Rackspace Cloud Monitoring we are currently using Graylog2 with a patch to support Scribe as a transport written by @wirehead.

Bellow is an example of searching for specific txnId in our system in Graylog2:

While this example is simple, we have some situations where a single txnId spans multiple services, and the ability to trace all of them transparently is critical in a distributed system.

Conclusion

Write your logs for machines to process. Build tooling around those logs to transform them into something that is consumable by a human. Humans cannot process information in the massive flows that are created by concurrent and distributed systems. This means you should store the data from these systems in a format that enables innovative and creative ways for it to be processed. Right now, the best way to do that is to log in JSON. Stop logging with format strings.

My wife and I love Doctor Who, so for this Christmas I wanted to do something slightly different for her. River Song is one of our favorite characters (along with Captain Jack) so I wanted to make River’s famous blue book.

Book cover was made from cardboard bound with faux leather and painted with several layers of blues to build up the highlights.

Pages are simply stained with coffee to age them. Images and text were a combination of images printed and worked into with ink, paint and pencil and straight up drawings done by hand.

The cover could use a little more work but it turned out pretty well and my wife loved it (combined it with a Tardis Key) which is the important thing.

I’m familiar with cars for graduation. Christmas, not so much.

Here’s one to point out:

returning my kindle fire and getting an iphone…this is bullshit.

The English in most of the other posts is worse.

It’s my favorite because it reminded me of Louis C.K.’s “Everything’s Amazing & Nobody’s Happy” interview with Conan:

I was on an airplane, and there was high-speed Internet on it. That’s the newest thing that I know exists. And I’m sitting on the plan and they go “Open up your laptop; you can go on the Internet”. It’s fast, and I’m watching YouTube clips; I’m in an airplane! …and then it breaks down. And they apologize: “the Internet’s not working”.

The guy next to me goes: “Pfft, this is bullshit.”

Like, how quickly the world owes him something he knew existed only ten seconds ago.

What may have seemed over the top in Louis’s skit seems a lot more real now.

(via reddit)

Memories of My California.

I will be back.

Merry Christmas.   Happy New Year