Neural Doodle and Pixel Art

A little while ago, @santiontanon suggested trying to use neural-doodle to upscale pixel art. Alex J. Champandard added some tweaks to improve the support for the process, and now we’re off to the races.

I’ve been running my own experiments, as you can see above. It doesn’t work for every texture, and there’s still a few occasional issues, but the combinations that do work are pretty astonishing.

NeuralDoodle generally takes a content image, and a style image, and translates the content into the style. You can also give it just the style image and seed data and have it synthesize a new image based on what it sees in the original style image.

Here’s a blue rock texture I made with Tilemancer getting interpreted as vines:

And here’s some more textures that used that same blue rock image as a seed. These were done with just a style image (the photographs, in this case) and a seed image (the pixel art blue rock), but no content image. 

Also using a content image results in a final output that more strongly resembles the content image, though with more discontinuities if there’s too great a mismatch between the content and style. You can see how this one doesn’t quite work:

These, on the other hand, came out much better:

As you can see, if there’s a huge mismatch between the content and the style, you may have trouble seeing the content in the result (though playing with the settings can sometimes find a better blend). But they still make for some pretty good images anyway.

Now, there’s a virtuoso pointillism to the practice of pixel art, so this isn’t going to replace all pixel art. Pixel art has already found its evolutionary art niche; new technology replaced all of the pixel art that had technologic reasons a long time ago. (Though we’re likely to get some Minecraft texture packs out of this.) 

But as part of a larger workflow–sketch out a rough idea of the texture you want, let the neural network dream up the details–this is going to be an interesting tool in the toolbox.