Daniel Shiffman

People occasionally ask me how to get started with procedural generation.

When someone asks me how to learn how to program, everything else being equal, I generally suggest that they start with Processing, because it’s an accessible platform with instant visual feedback and tons of resources.

For learning procedural generation specifically, I’ve pointed people to various resources in the past. But I think what I should be recommending to most people is Daniel Shiffman’s creative computing resources.

Daniel is the author of Nature of Code, a book about using natural principles to create emergent software and has created hundreds of videos of creative coding with processing. You can learn Poisson-disc sampling, A* pathfinding, genetic algorithms, strange attractors, and dozens of other useful things.

All of the stuff he talks about are great building blocks for generating your own creative projects, and even for experts there’s a lot here to learn from.

Flight404

The work of Robert Hodgin, aka flight404, has been influential in creative computing. To the point where his Magnetosphere was added to iTunes as a visualizer (with porting assistance of Andrew Bell).

His Vimeo account has recently been restored, with videos of projects going back over a decade. A few examples:

Solar, with lyrics.

Magnetic Ink, Process video

Magnetosphere, revisited (from 2007)

Murmuration, 40,000 starlings

Fox Horror, by Nervo and TBG 

Much of his early work was done in Processing (early in the history of that project) and he later went on to help with Cinder (where Andrew Bell was and is the lead architect).

Here’s a talk Robert gave Eyeo 2014 that manages to cover his working methods, simulating the planet, the rules of flocking, stacking things, walking in New York, murmuration, and giving a talk without a title.

Flow 

An interactive, generative art installation by Mathieu Le Sourd that translates real-time data about water temperature and sea level into this undulating visualization.

In an interview with the artist, he talks about giving the system “enough intelligence and randomness” so that the visuals become “organic and unpredictable.” This, I think, is one of the great challenges for making interesting visualizations, finding the balance between incomprehensible chaos and uninteresting stasis.

Note also the use of the unusual inputs, translating the numbers in the database into movement and sound that humans could absorb. As a group, we’re very bad at understanding masses of numbers, but very good at comprehending the general sense of them once they’re translated into a form that we can directly experience.

While this is for an art installation, there are also more practical uses: a monitoring system that produces warnings that humans humans can more easily detect dangerous changes is a tool that can make us safer.

https://www.youtube.com/watch?v=CX0fd8E1Q2c

Just because you can produce an infinite number of content doesn’t mean that you can show all of them to the player. Instead, the player will fail to see any difference between your generated content and will get bored fairly easily. Craig Perko calls this issue the Content Swamp and states that No Man’s Sky suffers from it, especially with its random planet generation.

In this video, Craig Perko discusses possible solutions to the Content Swamp: reduce the occurrence of the content to make it seem more rare and special, reuse the content by allowing people to carry on a long-term relationship (allow people to take possession of the “content” or to revisit the “content” regularly), and recycle the content by allowing the player to change the generated content to thereby produce their own human-generated content.

Procedural generation is just a tool. It’s important to know its limitations and be able to work around them…

A reader submission by Igor Horst, talking about a Craig Perko video. Both people come highly recommended by me. (Check out Craig’s huge list of videos and blog posts!) And I can definitely get on board with procedural generation just being a tool and a starting point.

The Content Swamp is clearly related to Kate Compton’s Bowls of Oatmeal Problem, something a lot of people are working on finding ways to address.

Space Plan Generator

Speaking of CAD and architectural uses of procgen, here’s a presentation by Shubhajit Das of research into using Autodesk Dynamo to find optimal floor plans for things like hospitals. 

As a practical problem, it’s a good candidate for algorithmic design: when  you’re designing something that needs to use the space in particular ways, like hospital beds or office departments, its much faster for the machine to suggest ways to lay out the space. Instead of trying out one or two possibilities in detail, the architects can rapidly review a huge number of plans that fit the specifications, allowing them to spend more time on the intangible, non-quantifiable parts of design.

The Space Plan Generator works off a spreadsheet of requirements, feeds it into the Dynamo visual programming, and output suggested floorplans for laying out rooms, corridors, and entire departments within the building.

You might notice some resemblance to roguelike building generators. I suspect it is because good level design and good building design have a lot of overlap. Though there’s also a bunch of stuff here worth thinking about the next time you need to write a corridor placement algorithm.

https://www.youtube.com/watch?v=ujG2Kans1UA&feature=youtu.be&t=40m50s

Tweet 2 Form

A bot by Andrew Heumann that performs parametric geometric operations when someone tweets at it. It understands eleven different formal operations, which is more than enough to produce a wide variety of results. It uses the tweet ID as a seed for the exact parameters it uses, so identical input texts will still produce different outputs.

I gather that the back end is powered by Grasshopper definitions, though I’m not too sure of the details. Still, it’s a good reminder that the CAD/industrial design world has their own tools for generative modeling, and that there’s many opportunities for cross-pollination between that and the more aesthetic, entertainment focused generative community.

https://twitter.com/Tweet2Form

Experimental music video made with neural face generator 

Mario Klingemann has been working on a face-expression neural net face generator that uses pix2pix to translate simple facial expressions into generated images. And then he fed a music video into it.

The resulting hallucinatory video has its moments of lucidity when the lip-syncing almost works as well as bits of more bizzare imagery. But, although the data makes many of the movements jumpy, the space itself is fairly consistent: completely different faces, but matching expressions. (And, as Mario points out, more open mouths would make the lip-sync work better.)

My expectation for the near-term artistic use of neural nets is something along this line, at least conceptually. Given a data space (faces, in this case) you introduce a structure (the face markers) and a concept (the expressions extracted from the song).

I expect that the visual aesthetics will be fairly varied, since different techniques can have radically different styles. But the framework behind them will generally fall along those lines.

Meanwhile, we have this. And I have to admit, it’s less disturbing than Mario’s earlier pose-generation tests.

https://www.youtube.com/watch?v=-elV0lT5-H4

Universal Adversarial Perturbations

Now for the recent developments in adversarial image mis-classifications: fixed patterns that can cause most images to be mis-categorized.

This research by Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard has lead to the discovery that there are patterns that can perturb nearly every image enough to cause an image recognition neural net to fail to identify it most of the time.

Earlier research into adversarial patterns required that you solve an optimization problem for each image. The universal patterns don’t need to do this: they can just add the pattern to every image, which saves a lot of time and effort.

Interestingly, despite the universal patterns being slightly different between different neural net architectures, they are still universal enough that one pattern works pretty well on all of the neural nets.

So if you want to fool, say, a facial recognition system, but you don’t know what neural net it uses, you can just deploy one of these patterns and expect that you’ll likely trick it.

It also tells us more about what is going on under the hood of these neural networks. Understanding why these universal patterns work can help us build better neural networks. The researchers think that the explanation might be because there’s a correlation between different regions of the decision boundary, so the geometry of the decision boundary is what they plan to research in the future.

https://arxiv.org/abs/1610.08401

Adversarial Machines

There’s been some interesting developments recently in adversarial training, but I thought it would probably be a good idea to first talk about what adversarial images are in the first place. This Medium article by @samim is an accessible explanation of what’s going on. It references this talk by Ian Goodfellow, asking if statistical models understand the world.

Machine learning can do amazing magical things, but the computer isn’t looking at things the same way that we do. One way to exploit that is by adding patterns that we can’t detect but that create enough of a difference in the data to completely fool the computer. Is it a dog or an ostrich?

There’s been quite a lot of research into finding ways round this problem as well as exploiting it to avoid facial recognition or other surveillance. And, like I said, there’s been some interesting recent developments that I hope to talk about here.

https://medium.com/@samim/adversarial-machines-998d8362e996#.n7j43766v

Tweet Game of Life

Something simple for today: a twitter bot by Avy Faingezicht, tweeting GIFs of a Game of Life sim. You can tweet at it, and it’ll give you a pattern based on it. And the source code is on GitHub.

https://github.com/avyfain/conway

https://twitter.com/tweetgameoflife