A pelican-on-a-bicycle SVG is a meaningful benchmark of overall AI reasoning

What's this about?

People disagree about whether a pelican riding a bike in an SVG, a code-based drawing, tests all AI reasoning.

The test may show one small skill, but it cannot show the full range of thinking.

What supporters say

• The task tests whether AI can combine objects and place them in the right link.
• An SVG can show errors that words may hide, such as poor shape, place, or object links.

What critics say

• One drawing cannot test all forms of reasoning, such as plans, doubt, or new tasks.
• A good result may show skill with drawing code rather than deep thought.

How to read this

The number of points on each side does not show who is right; the strength of the proof matters more.

The bottom line

This test works as a small check of how AI turns words into a clear, linked picture.

But the proof does not show that it measures overall reasoning, and some success may come from drawing skill.

A pelican riding a bicycle may look like a playful test, but some researchers see it as a window into how AI understands relationships between objects. The evidence supports that narrower use, but not the broader claim that one such SVG can measure overall reasoning.

The case for

The strongest argument is that the task tests compositional visual understanding, not just the ability to describe an image in words. An AI must represent both a pelican and a bicycle, then place them in the required relationship. That involves coordinating several objects in one structured image, including their hierarchy, position and meaning. Research on compositional SVG generation treats these abilities as relevant to structured visual output. (see Figure 3) 1

The test can also reveal mistakes that a text-only answer might hide. A model could correctly say that a pelican is riding a bicycle while producing an image in which the two objects are separate, poorly placed or visually incoherent. Requiring an SVG creates visible constraints on geometry, object relationships, code and rendering. (see Figure 2) 2

That makes the task a credible narrow diagnostic. It can test whether a model translates a relationship expressed in language into a structured visual artifact. But this conclusion depends on using clear scoring rules, rather than judging the result only by whether it looks appealing.

The case against

The central objection is that one artifact cannot measure overall reasoning. Broad reasoning includes many abilities, such as abstraction, planning, dealing with uncertainty, strategic interaction and generalizing to unfamiliar situations. A single drawing directly tests only a small part of that range. 3

Success on the pelican task therefore does not, by itself, show that a model would reason well in nonvisual or unrelated settings. The available evidence does not report a validated link between performance on this one prompt and independent measures of general reasoning. It also does not compare models across repeated variations or test whether the skill transfers to abstract, uncertain or social problems.

Performance may reflect graphics and SVG-writing skill as much as reasoning. Producing a valid file requires knowledge of markup, geometric representation, rendering conventions and style. A model might fail because it struggles with vector graphics, or succeed by using a familiar drawing pattern, without demonstrating broad reasoning ability. (see Figure 2) 4

There is also a risk that a distinctive prompt could be solved through memorized examples, templates or familiar visual associations. Stronger testing would use different animals, vehicles, poses and spatial relationships, along with paraphrased instructions and new combinations. It would also score semantic accuracy, visual quality and the structure of the SVG separately.

The bottom line

The evidence does not support calling a single pelican-on-a-bicycle SVG a meaningful benchmark of overall AI reasoning. That conclusion is backed more strongly than the broader claim because the task covers only a limited slice of reasoning and has not been shown to predict performance elsewhere.

The evidence does support treating a carefully designed, repeatedly tested SVG suite as a potentially useful benchmark of compositional visual generation. Its value would depend on explicit scoring, varied and held-out examples, and safeguards against memorization. In short, the task is a plausible narrow test, but the claim that it measures overall reasoning is not established.