Hotdogs

A game about drawing the weirdest thing that an AI (CLIP) still thinks is a dog.

Unfortunately, the game is not up anymore :( It's not easy to host it for free like some of my other projects because you need to run the CLIP model on a server somewhere. It might still be doable, but the game also had some design flaws that I'm not sure how to fix (see below).

The idea

Your goal is to draw a dog that will reach a certain "dogness level" - how similar the AI thinks your drawing is to the platonic ideal of a dog drawing - while maximizing the uniqueness level, which measures how different your dog is from the ones that other people have drawn.

The premise was that we wanted to create a visual version of Gandalf, the prompt injection game I designed at Lakera and then put together with my colleagues. Gandalf went viral, earning Lakera a lot of attention, which was something we naturally wanted to repeat.

For Hotdogs, the design that Aashna Majmudar and I came up with during another company hackathon was to make a game about fooling CLIP, the moel that allows you to compute similarity between pairs of text and images (or pairs of text, or pairs of images). The idea (draw the strangest thing that still looks like a dog to the model) is anologous to prompt injection because in both cases you're looking for adversarial examples: inputs on which the network behaves differently than we humans would expect.

To compute uniqueness, we have a set of base dog drawings and compute the average CLIP similarity of the image to the dog drawing dataset. I collected these by going around the Lakera office and asking everyone to draw a dog. The quality was... mixed.

What happened?

Unfortunately, Hotdogs never took off, and after working on it for about two weeks, we were told gently (and then less gently) to stop it and go back to working on prompt injection detection models. Fair enough.

Looking back, the game had a few issues.

Preventing cheating

Because Lakera is a company about hacking AI models, of course they would try to break the game as well. Since the backend endpoint just accepts an image and doesn't check whether it was actually produced by a person drawing it by hand, you can submit dogs like this:

Obviously, a photo of a dog is way more dog-like than whatever you can draw, and it also looks very different than the other submissions.

By the way, just writing the word "dog" in the image was also accepted by CLIP as being dog-like enough, which is what topped the weirdness leaderboard until Aashna patched it with some prompt engineering.

Moderation

We wanted to have a leaderboard with the best weird dogs and show them publicly. However, giving people a place where they can express their creativity publicly on the internet is generally a bad idea. Or more bluntly, if they can draw dicks, they will draw dicks. We could mitigate this with more computer vision or community moderation (or not?), but it'd be extra work.

It's not fun

The fact that you need to reach a minimum level of dogness means that you might end up spending a lot of time on drawing a beautiful dog, only to have CLIP reject it because it doesn't consider it dog-like enough. This could be solved by not requiring a fixed minimum dogness, but rewarding people who are on the Pareto frontier of the dogness-uniqueness curve.

But beyond that, the game felt more gimmicky than Gandalf. We let people play it and they always liked the initial idea, but rarely stuck around to draw more than just one dog.

We were thinking about doing something Wordle-like, where every day you would be given a different animal or object to draw, and you could repeatedly challenge others. One issue there is how to compute the weirdness: do you compute similarity to all existing entries? Then submitting later gives you an advantage. Or do you wait for the end of the day and only then compute similarity? Then you don't get the immediate feedback about how well you did until tomorrow, and the player will certainly forget about the game until then.

It could still work

I do think the basic principle of Hotdogs is fun and there could still be something there, though it would require a substantial redesign. For example, my friend recently sent me this site, which has a vaguely similar idea: https://drawafish.com/