ResearchUpdated

Building Rothko, Plnty's creative agent

Requirement 1

To learn the intuitive

Understanding how to communicate the intuitive elements and subjective preferences that characterise Yambo Studio's films, such as timing and camera movement choices, is a challenge.

Over the course of dozens of product films, Yambo Studio has developed certain unspoken formulas for camera and animation work. The movement speeds up as it arrives at a cut, carrying through to the next shot; it then slows to linger on the product.

You can see examples of this formula with the foil package below. When attempting to translate the same motion principles to the kids camera film, words alone failed to achieve a good result. However, using the foil film as a motion reference, the new film began to behave more in line with my preferences.

At Plnty, we performed an optical flow analysis of the video to measure movement between frames and compare the films. This data was then used to establish a default behaviour for Rothko, an applicable guideline for camera motion that dictates the type of camera motion expected in this given context. Then you can make it faster or slower using language.

How fast the picture movesThrough an average shotYambo Studio Motion Referencesampled from over 100 filmswith signature motionKids camera, words only1.0x faster at the cuts (no change)Kids camera, with the foil film ref5.0x faster at the cutsKids camera, finished4.5x faster at the cuts036912151821 sstart of shotendthe shot's average
Camera movement over time for each film. Thin grey vertical lines mark cuts. Dotted horizontal lines mark each film's average camera speed. The reference helped the camera film recover the fast cuts and slower middles. Multipliers (e.g. 5.0x) indicate the average factor of camera speed increase at each cut versus the middle of each shot.
How fast the picture movesHeadphones3.1x faster at the cuts036912151821 sThrough an average shotcamera speed, measuredthe sun, one steady reveal(the formula, drawn)start of shotcut
The headphone film uses the same default camera behaviour, no longer requiring the motion reference to match the behaviour. Intuition has been codified and taught so that Rothko can recall my preference prior to an initial generation. The small graph is the formula, the camera easing between cuts while the sun keeps one steady reveal. The camera's curve is measured; the sun's line is drawn, the steady pace the formula asks for.

Requirement 2

Then, a creativity ladder

If we at Plnty are to equip an agent with the capacity to not only create, but to direct creativity, then we need a way to approximate a designer's ability to judge the appropriateness of a solution compared against a brief and its context. This judgement is informed largely by ineffable qualities such as 'mood'. For example, certain beauty brands might favour an avant-garde product composition where others require something more conventional. If Rothko cannot determine the suitable response to a brief (on a spectrum from 'playing it safe' through to 'tear up the book'), then output will be uninformed and arbitrary. The solution: a 'Creativity Ladder'. The further up the ladder, the more extreme the ideas. The ladder offers six levels of freedom over composition, scale and staging.

These early tests are built on the same prompt at each level. At the low end of the ladder, Rothko writes a plain brief. From the middle up, it draws on the Plnty team's taste, with bolder staging at the top. Your brief still sets the boundaries and allows an override for projects requiring a departure from studio convention.

Four ideas, six levels of creativity. In the edits, the changes are subtler until the high levels bring the camera lower and closer. Imagery created by Sasha, Plnty's designer.
Six stepsVery lowLowMediumMedium highHighVery high
From wordsPerfume launch"a launch picture for a new perfume called North"
A clear glass perfume bottle on a black rock in the snow, at Very low.
The North bottle on a black rock in a snowfield with mountains behind, at Low.
A tall glass bottle on a black rock in the snow, at Medium.
The North bottle with a black cap on a rock in the snow, at Medium high.
The bottle seen closer and from below on its rock, at High.
The bottle seen from very low, towering over the rock against the sky, at Very high.
EditDance festival poster"turn this into a poster for a dance festival", on Sasha's dancer
Sasha's dancer in a handstand on a lilac tiled block beside a tiled wall, at Very low.
The dancer in a handstand on the block against a peach sky, at Low.
The dancer in a handstand beside the tiled wall, at Medium.
The dancer in a handstand on the block, the wall rising behind, at Medium high.
The dancer seen closer from below on the block, at High.
The dancer seen from low, large against the sky, at Very high.
EditPerfume campaign"make a campaign picture for this perfume", on Sasha's bottle
Sasha's blue and amber perfume bottle with a white lily, standing in rippling water, at Very low.
The bottle in the water with a warm sunrise behind, at Low.
The bottle in rippling water under a pale blue sky, at Medium.
The bottle in calm water with soft reflections, at Medium high.
The bottle seen lower and closer, the water lit by the sun, at High.
The bottle seen from the waterline, large against the sky, at Very high.
From wordsRunning shoes launch"a launch picture for new running shoes"
A pair of grey-blue knit running shoes on a plain grey studio floor, at Very low.
One pale grey running shoe with a blue sole, close up in the studio, at Low.
Black and red running shoes set on starting blocks on a red track at dawn, at Medium.
A runner's foot in a white running shoe, lifting off a white starting block, at Medium high.
Black and red running shoes on the crest of a dark banked track against a blue sky, seen from below, at High.
White and blue running shoes on top of a giant loop of blue running track, on cobalt, at Very high.

Requirement 3

Creative skills

A creative skill in Plnty is a working process, or series of tools combined into a workflow, that Rothko can recall on command and apply to any chosen context. The skill is a pre-configured set of steps which remain in a fixed order. Choosing and developing output in a new context using a skill allows for interesting results with an element of control depending on the intended remit of the skill.

Select something (or many things) on your board to open Rothko, and select Run a creative skill. That starts the whole job, with three seconds to cancel using Esc before it spends credits. You can stop a run at any time. Turn on Spending limits in Settings, under Creative Agent, to require approval above your limit.

Owen, a designer on our team, made this film with Made by hand, the creative skill now on Plnty's home page.
Transcript

No words. The sounds of the kitchen for the first twelve seconds, the flour and the eggs, then quiet to the end.

Requirement 4

Creating Rothko's 'brain'

Rothko uses separate models to plan, write instructions and judge results. At Plnty, our corrections and examples guide the planning and writing. The judge compares pictures against your brief and the criteria for that stage. Bringing our taste more directly into those judgments is work we still have to do.

In a session recorded for this paper, Rothko kept three of eight first results, one from each of three directions. It combined these selects into another picture, then began developing all four. We still need to measure how often those choices lead to work a team actually uses.

Until the processes can be connected so that decisions are informed by other phases of the decision-making process, the workflow is going to be scattered.

From Yam

A lifetime of creative work teaches you what to explore, what to keep, and when to change direction. Over the past few months, I've been pouring that experience into Rothko, giving it a sense of how creative decisions get made, and what makes an idea worth pursuing.

Yam Ben AdivaCEO and co-founder of Plnty

Automating the Creative Workflow

At Plnty, we've written about this in 'Automating the Creative Workflow'. The paper describes what we've built, the decisions we've recorded and what we still need to test. It also looks beyond today's skills, toward an agent that can choose which stage the work needs next.

The draft is confidential for now. Sign in and request access; we'll email you when it's available to your account.

More hands for a small team

I want a small creative team to have more room to explore an idea before committing to it. Rothko can take on the combinations, variations and first clips, leaving the team more time to choose what deserves attention as it moves towards a finished product.

Sources and methodology

Rothko's practice draws on the works below, as cited in Plnty's research paper Automating the Creative Workflow (version 1.1.8, a confidential draft). Every picture and film in this article was made in Plnty, and the motion reference is sampled from Yambo Studio's films.

  • J. L. Lush. Animal Breeding Plans. Iowa State College Press, 1937.
  • D. S. Falconer and T. F. C. Mackay. Introduction to Quantitative Genetics, 4th edition. Longman, 1996.
  • Design Council. The Double Diamond. Framework and account of its use.
  • W. R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3–4), 1933.
  • D. A. Schön and G. Wiggins. Kinds of seeing and their functions in designing. Design Studies, 13(2), 1992.
  • D. Kirsh and P. Maglio. On distinguishing epistemic from pragmatic action. Cognitive Science, 18(4), 1994.
  • M. A. Boden. Creativity and artificial intelligence. Artificial Intelligence, 103(1–2), 1998.
  • G. Wallas. The Art of Thought. Jonathan Cape, 1926.
  • J. P. Guilford. Creativity. American Psychologist, 5(9), 1950.
  • D. T. Campbell. Blind variation and selective retentions in creative thought as in other knowledge processes. Psychological Review, 67(6), 1960.
  • D. K. Simonton. Creativity and discovery as blind variation: Campbell's (1960) BVSR model after the half-century mark. Review of General Psychology, 15(2), 2011.
  • A. Koestler. The Act of Creation. Hutchinson, 1964.
  • D. A. Schön. The Reflective Practitioner: How Professionals Think in Action. Basic Books, 1983.
  • G. Goldschmidt. The dialectics of sketching. Creativity Research Journal, 4(2), 1991.
  • A. Clark and D. Chalmers. The extended mind. Analysis, 58(1), 1998.
  • R. E. Beaty, M. Benedek, P. J. Silvia and D. L. Schacter. Creative cognition and brain network dynamics. Trends in Cognitive Sciences, 20(2), 2016.
  • M. Polanyi. The Tacit Dimension. Doubleday, 1966.
  • T. M. Amabile. Social psychology of creativity: A consensual assessment technique. Journal of Personality and Social Psychology, 43(5), 1982.
  • Anthropic. The Model Context Protocol. Specification and documentation, November 2024.
  • E. Cherry and C. Latulipe. Quantifying the creativity support of digital tools through the Creativity Support Index. ACM TOCHI, 21(4), 2014.
  • Yambo Studio. Films and motion design, 2014 to 2026.
  • ComfyUI contributors. ComfyUI. Open-source node-based environment.
  • S. Yao et al. Tree of Thoughts: Deliberate problem solving with large language models. NeurIPS, 2023.
  • Z. Wang et al. GenArtist: Multimodal LLM as an agent for unified image generation and editing. NeurIPS, 2024.
  • J.-B. Mouret and J. Clune. Illuminating search spaces by mapping elites. arXiv, 2015.
  • J. Lehman and K. O. Stanley. Abandoning objectives: Evolution through the search for novelty alone. Evolutionary Computation, 19(2), 2011.
  • L. Zheng et al. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. NeurIPS Datasets and Benchmarks, 2023.
  • L. Gao, J. Schulman and J. Hilton. Scaling laws for reward model overoptimization. ICML, 2023.
  • R. A. Finke, T. B. Ward and S. M. Smith. Creative Cognition: Theory, Research, and Applications. MIT Press, 1992.

About the author

Yam Ben Adiva

Yam Ben Adiva is the CEO and co-founder of Plnty.

Run a skill on your own moodboard

Select a few pictures on a board, open Rothko's chat and press Run a creative skill.

Try Rothko in Plnty

Ideas worth opening.

Hear from Plnty by email, starting with when access opens.

By subscribing you agree to hear from Plnty by email. We send one email first to confirm your address, and you can unsubscribe at any time.