Neural-doodle and style transfer

I’m still playing with neural-doodle. It’s a bit slow, due to my insistence on using larger image sizes for nice results. I’d probably get further with smaller images iterating faster on the GPU before rendering a full resolution, but one of the things I’m currently experimenting with is finding which settings get the best final results.

Said results, so far, are a mix of astonishing and a noisy mess, depending on which style image I use and how it’s marked-up semantically. The algorithm is very good at understanding the edges of the branches and leaves, so marking too close to them can sometimes have a worse result.

The scale of the replacement image also matters. I’m using Gigapixel scans of public domain artwork for most of my style images, so I have the resolution to crop out a small section of the painting. This works better than trying to use the full-scale painting. I’m not sure yet if that is because the details are too small once I scale down the larger painting, or if it is because the subject of the cropped image has more similarities with the target content. The algorithm appears to be very sensitive to the scale of the features in the image; using a larger image produces a different result.

I think that of all of the artistic applications of neural networks, style transfer algorithms have the most potential for artists in the near term. Taking a photo or a sketch, applying a style image that you painted, and getting a usable result that can be put into whatever game, film, comic, or other project you’re working on will be a handy ability for some future artists. Whatever default settings it ships with will probably be overused and we’ll have a rash of van Gogh-styled photo filters, but for working artists it’ll probably take on the status of something like Z-Brush or Mudbox: a tool that you use to sculpt a result that would have taken you hundred of hours to do by hand.

It will, of course, also raise questions about copyright, intellectual property, and just who exactly is responsible for what part of the result, which is one reason why I’ve been using my own art and public domain paintings as the source images for my experiments. Having public domain data with clear licenses for this kind of stuff is going to be critically important for keeping the artistic possibilities of these kinds of tools open to everyone to experiment with.