Designer

Verified against v1.3.0

The Designer is the native canvas design editor at /media/designer. It is built on react-konva (Konva.js) and reads input from and writes output to the Files library; it never stores its own assets outside /files.

DesignerDoc contract (server-authoritative)

The DesignerDoc JSON is the agent interface, and the server owns it. There is one zod-based source of truth in libraries/nestjs-libraries/src/media/designer-doc/designer-doc.schema.ts; the frontend store and the server renderer both import type from it, while the dependency-free runtime helpers (migrateDoc, createBlankDoc, limits) live in designer-doc.migrate.ts and stay out of the client bundle. The contract discriminates on mode (image ⇒ outputs: DesignerOutput[], video ⇒ outputs: VideoOutput[]; an image output's children is page 1 and optional pages[] hold further pages — video outputs reject pages) and carries version + migrateDoc for forward-compatible upgrades.

DesignerDocService provides validate (lenient, clamps out-of-shape docs), validateStrict (rejects unknown keys), applyOps (mode-aware document transforms), and assignIdsAndNormalize (CSPRNG id minting across children/tracks/clips). DesignService validates every persisted design and template, reconciles Design.width/height from doc.outputs[0], and exposes instantiateTemplate and placeAsset for agent use.

Frontend architecture

All components live under apps/frontend/src/components/media-tools/designer/.

• designer.store.ts — a per-mount Zustand store created by createDesignerStore(w, h, attribution, fetch?). No module-level singleton (it resets on unmount). It holds the shared DesignerDoc (mode + outputs[], one output per format; elements are text | image | shape | icon | path | raster | group | fill | adjustment | symbol), plus selectedIds, currentOutput, currentPage, zoom/viewport, labelled undo/redo history, clipboard, editFormatOnly (linked vs format-only edits), and playheadMs / selectedClip for video mode.
• canvas.tsx — the Konva Stage. Handles multi-select (shift/⌘ + marquee), group-aware selection, snapping/alignment guides, a custom Transformer with a dimension HUD, wheel-zoom, space-drag pan, and drag-and-drop drops from panels (payload key application/x-designer-element). actions.ts is the single registry behind the menu bar (menu-bar.tsx) and the ⌘K command palette.
• elements.tsx / layer-render.tsx — element renderers (text/image/shape/…), gradient fills, crop, flip, layer effects (shadow, glow, stroke), and entrance animations.
• Panels (panels/) — templates (with org template folders), my designs, layers, pages, version history, icons, brand, AI (gated on an active org AI provider via useAiActive), plus the per-type inspectors.
• controls/ — reusable control primitives. fonts.ts — curated OFL fonts + ensureFontLoaded; org custom fonts are uploaded through POST /media/fonts/upload (listed by GET /media/fonts, removed by DELETE /media/fonts/:fileId).
• video-timeline.tsx — the video-mode track/clip timeline; MP4/WebM/GIF/animated-WebP exports render server-side through POST /media/designs/render-video.
• export-dialog.tsx — PNG / JPEG / transparent-PNG / WebP / PDF / SVG plus animated formats, high-res pixelRatio, multi-output export, and a create-post flow (export → one draft per selected channel → the composer). Exports are saved to the Files library and the done step also offers per-file download links.
• collaboration.ts — opt-in real-time collaboration (useCollaboration): a Yjs document synced over the backend's /collaboration WebSocket (apps/backend/src/services/collaboration/collaboration.gateway.ts), with peer cursors from collaboration-cursors.tsx.

Cross-origin canvas

Konva's toBlob/toDataURL taints on cross-origin images. Element images load with crossOrigin="anonymous"; for object storage you must enable CORS (see the operations storage guide) or route through the same-origin image proxy (GET /media/designer/proxy).

Backend

Layering is Controller → Service → Repository (only repositories touch Prisma).

• Models: Design, DesignVersion (version-history snapshots of the whole doc; restore takes an auto safety snapshot first), DesignTemplate, DesignTemplateFolder (org folder tree for templates; nullable DesignTemplate.folderId = unfiled).
• CRUD: DesignController / DesignTemplateController — /media/designs, /media/design-templates. Version history lives under /media/designs/:id/versions (list/create, GET/DELETE /:vid, POST /:vid/restore); template folders under /media/design-templates/folders (CRUD) with PUT /media/design-templates/:id/move. DesignerProxyController — GET /media/designer/proxy (org-bound, safeFetch, fail-closed on non-image, size-capped).
• Server-side render (DesignRenderService, node-canvas): POST /media/designs/render → PNG or PDF (pdfkit). Text blocks auto-fit and honor verticalAlign; a curated-font load failure retries after a 15-minute TTL (font-loader.service.ts) instead of blacklisting the family for the process. Bulk generation (DesignBulkService): POST /media/designs/bulk-generate substitutes per row and renders a batch. Both endpoints validate body.doc before rendering.
• Video render (VideoRenderService): POST /media/designs/render-video enqueues a timeline render; GET /media/designs/render-video/:jobId returns status/artifact. Gated on media:create and video-exports:create.
• Document ops / validation: POST /media/designs/validate (lenient, media:read) returns { valid, errors? }; POST /media/designs/apply-ops (strict, media:create) returns { doc } after applying a strict-parsed op sequence. Two op behaviors changed with the AI Designer release: updateElement accepts an optional scope ('shared' propagates the patch to linked elements across outputs, 'format-only'/absent keeps the previous single-element behavior), and addOutput seeds the new output from the primary output's children (reflowed; originId backfilled on the primary) instead of appending an empty white canvas — matching the manual Designer's linked-by-default behavior. addOutput also accepts an optional seed: false to keep the pre-change empty-canvas behavior, and appends an unseeded canvas when the doc has no outputs at that point in the sequence (e.g. after removeOutput).
• AI ops (AiUserController, throttled, org-authenticated, credit-checked via AiMediaService): POST /ai/media with operations remove-background, inpaint, upscale, image, video, audio, avatar, bg-remove, tts, stt, alt-text. These delegate to AiMediaService, which routes through the per-org Media provider system. The Designer's generate dialogs pick a provider from GET /media/generation-providers?operation= (enabled + credentialed providers for that generation type plus the org's resolved default; media:read), and its editor AI actions also call POST /media/outpaint, POST /media/video-segment-transform and POST /media/voiceover-script.
• Agent seam: /copilot/agent now carries the acting user in the Mastra requestContext so user-attributed tools can fill createdById. The designerDesign Mastra tool (libraries/nestjs-libraries/src/chat/tools/designer.design.tool.ts) creates/updates designs from a DesignerDoc, template, or op sequence and persists an image preview when mode === 'image'.
• Doc changes need no migration. Design.doc and DesignTemplate.doc are Prisma Json fields; the DesignerDoc schema is enforced at the service boundary (DesignVersion and DesignTemplateFolder are additive tables with their own migrations).
• Brand kit: AIBrandProfile.logoFileIds / palette / fontFamilies are read/written through the brand profile API.

AI Designer chatbot (/media/ai-designer)

The AI Designer is a chat-first, server-rendered design assistant built on top of the Designer foundations. Sessions are single-mode: an all-video-preset channel selection derives format: 'video' on the session (mixed image+video selections are rejected). Video sessions compose and QC with the image pipeline, then the conductor converts each delivered doc to a video-mode doc (util/image-doc-to-video.ts) and, when the org has a video default, generates a scene video per variant's primary output that lands on a video track above the converted still.

• Realtime transport: bespoke Socket.IO namespace /ai-designer registered in apps/backend/src/main.ts. Cookie JWT + CSRF handshake, session rooms (session:<id>), monotonic seq ordering, client ack, and Redis-backed nonce idempotency for every mutating event (start/message/form:submit/accept:plan/revise); a nonce consumed by a rejected operation (budget/guardrail/limit) — or by a handler that fails unexpectedly, which also emits internal_error with the nonce — is released so a retry with the same nonce isn't locked out for the TTL. The gateway also enforces per-user rate limits on every mutating event (buckets keyed by user id, so opening fresh sockets doesn't reset the window) plus a per-IP budget on connection attempts (checked before any JWT/DB work — neither the HTTP throttler nor per-event limits cover the handshake itself — and keyed on the transport address: the forgeable X-Forwarded-For header is never trusted, matching the HTTP throttler's posture), caps stored sessions at 100 per (org, user), re-checks membership/RBAC/billing every 60s for long-lived sockets (the NOT_SECURED dev toggle — honoured only when NODE_ENV=development, like every other NOT_SECURED relaxation — bypasses only the handshake CSRF check, never RBAC or billing), validates session ownership before any handler-side write, bounds form:submit values (32 KB serialized / 5 levels deep), and runs all user free-text — chat messages, form values (strings at any nesting depth), and revise instructions — through the org's input guardrail chain (@reaatech/guardrail-chain via the governance GuardrailService) exactly once, at the gateway, before it is persisted or dispatched. Errors are emitted as { code, message, nonce? }; progress as { kind: 'progress', agent, phase, pct, note }; user-message echoes carry the client nonce for optimistic reconciliation. Deployment note: the gateway uses Socket.IO's in-memory adapter and the conductor keeps its pipeline mutex / prompt correlation / circuit breakers in-process — run the backend as a single instance (or behind sticky sessions) until a Redis adapter + shared stores are added. Setting COLLAB_SINGLE_INSTANCE=false without COLLAB_REDIS_ADAPTER emits a startup warning for both /collaboration and /ai-designer.
  • Reverse proxies: by default (TRUST_PROXY_HOPS unset) the connect-rate bucket keys on the socket peer address, so all clients behind a reverse proxy share one bucket. Set TRUST_PROXY_HOPS to the exact number of XFF-appending proxies to key the bucket on the Nth-from-right entry of X-Forwarded-For instead — overestimating lands in attacker-controlled left-most XFF entries and makes the buckets spoofable. The same setting also drives the HTTP throttler's per-IP buckets (login/register/enterprise/public-report) and the MCP rate limits.
  • Stuck sessions: if a session is in planning or executing and untouched for longer than AI_DESIGNER_STUCK_SESSION_MINUTES (default 15), reconnecting rolls it back to awaiting_plan so the user can retry.
• Session model: AiDesignerSession + AiDesignerMessage (additive Prisma tables). State machine: intake → planning → awaiting_plan → executing → delivered → revising. Sessions are pruned by the daily retention-purge cron after AI_DESIGNER_SESSION_RETENTION_DAYS (default 90; messages cascade), and users can delete their own via DELETE /ai-designer/sessions/:id. deleteUser tears sessions down explicitly (the userId FK is RESTRICT). Intermediate files minted for a session (asset imports stamped metadata.aiDesigner = 'asset' + sessionId; reference imports stamped 'reference') are hidden from the default /files listing and soft-deleted on session delete/retention purge unless a delivered design doc still references them.
• Agents: six in-process agents registered via agent-mesh v-next (libraries/nestjs-libraries/src/ai-designer/agent-mesh/). The agent registry is bundled TS data (agent-registry.data.ts, validated against AgentRegistrySchema at boot) — not a YAML asset, which would not survive nest build — with AI_DESIGNER_AGENT_REGISTRY as an optional directory-of-YAML override (mapped to the package's AGENT_REGISTRY_DIR at import time). The mesh session/breaker stores default to Redis; AI_DESIGNER_MESH_STORE=postgres opts into the package's Postgres stores (runs its own DDL on a second connection pool — deliberately not the default, and it requires a dedicated AI_DESIGNER_MESH_DATABASE_URL: never the Prisma DATABASE_URL, where third-party DDL would fail the CI schema-drift gate). agent-mesh-env.shim.ts (first import in every agent-mesh-importing file) tames the package's import-time env handling: it seeds placeholder GOOGLE_CLOUD_PROJECT/API_KEY (Postmill uses only the in-process transport; the package's LLM classifier is never invoked), forces the package's global circuit breaker off (it is keyed by agent id only, so one tenant's provider failures would open it for every org — the conductor's per-(org, agent) breaker is the only breaker; the flag is z.coerce.boolean(), so the empty string is the only disabling value), and stash-and-restores unrelated env values the package's strict schema would otherwise process.exit(1) the backend on (e.g. LOG_LEVEL=verbose, NODE_ENV=staging, a malformed OTEL_EXPORTER_OTLP_ENDPOINT). The restore is synchronous with zero exposure window: agent-mesh-env.stash.ts stashes, the shim then imports the mesh package (whose env parse runs at import), and restores in the same tick — no other module ever evaluates with the stashed vars deleted. Mesh module setup is non-fatal: a failure degrades AI Designer with a logged error, never blocks backend boot.
  • Conversationalist — elicits intent conversationally (extracting brief fields every turn; intent must keep the user's concrete specifics verbatim — event names, offers, schedules — and verbatim copy like coupon codes lands in fixedCopy) and parses natural-language revisions. Once the brief is complete it recaps what it understood (including fixedCopy) exactly once and waits for an explicit green light — the recap is enforced deterministically via the server-owned brief.recapShown flag (not by the classifier), so planning never starts before the user confirms the recap.
  • Art Director — routes the brief through a skill registry (skills/design-skill.registry.ts: the meme, advertisement, greeting-card, product-promo and announcement genres, a reference-clone meta-skill that wins whenever the brief carries reference cues, and commerce/announcement/social-native/authority/social-proof/seasonal/vertical skill families); each skill carries layoutHints (formatTemplates + slotSchema), richer prompts, and requiredBriefFields wired into intake, and a low-confidence route asks the user to pick a skill (brief.preferredSkill) instead of planning silently. Emits plan-schema-v2 DesignPlan[] variants — plan/slot styleId, slot kinds (text/image/cta-button/badge/accent-shape), and per-slot texts (the actual headline/subhead/CTA/badge copy, written to the skill's copy rules and naming the brief's real event/offer; brief.fixedCopy appears verbatim) — varying style presets across variants unless the user picked one. Plans describe the primary format only (first selected channel, else first custom size); other formats are adapted later by the conductor, not planned. Custom sizes are keyed uniformly as custom-${w}x${h}, and a custom-sizes-only start is allowed.
  • Copywriter — fills text slots per plan/brand voice. Slots the user approved on the plan card arrive as lockedTexts: returned verbatim and never rewritten; only open slots are written (all-locked skips the model call entirely).
  • Asset — generates/stocks imagery per output aspect (slotId:aspect keys; square/wide/tall size mappings for gpt-image and replicate/flux, stock orientation passthrough, layout-aware prompt suffixes that reserve space for copy); every generated-image prompt also carries a "no text, no words, no letters, no typography, no watermark, no logo" suffix so assets never bake in copy (stock searches don't get it). When the session has a reference image, the conductor stamps it onto background/hero needs (sourceFileId/sourceUrl) and the agent tries the org's media/image-to-image default first, falling back image-to-image → text-to-image → stock (only transient failures earn a retry); a run cannot start without a media/text-to-image default (AiDesignerDefaultsGate). Any stock stand-in is disclosed in the delivery notes unless stock was deliberately requested; the last resort is a generated gradient fallback.
  • Composer — builds validated DesignerDocOp[] using server-side seedCopy/smartReflow/applyLinked; honors the plan's style preset (palette, curated font pairing, typeScale, treatments) and brand fonts through a 6-template layout gallery (hero-fullbleed, split-panel, top-bottom, badge-burst, editorial-sidebar, minimal-centered; older ids alias in) and renders CTA slots as shape+text buttons. Imagery is always edge-to-edge (full-bleed or full-width/full-height bands — minimal-centered renders its image as a top band, never a floating framed inset). Per-role type floors keep copy legible at feed-thumbnail size (headline ≥ 6%, subhead/body ≥ 3.2%, badge/CTA ≥ 2.8% of min(w,h), clamped up in _typeScalePx), and badge/burst labels auto-fit inside their shape's inner safe area (centered both ways, never spilling). It composes the primary format only; the conductor then seeds each remaining format with addOutput (linked to the original via shared originIds) and re-resolves per-aspect assets (applyPerOutputAssets). applyFixes accepts an optional targetOutputs list that pins every fix to one format and forces format-only scope, so per-format critique fixes never propagate onto other outputs. Its revise-op vocabulary covers fontFamily/align/verticalAlign/textStroke/textShadow/addElement/removeElement/setOutputBackground (the revise prompt summarizes each output's current background, and the op replaces an image background when the instruction asks for a color/gradient), and a shared-scope fontSize fix is scaled proportionally per output. After compose and after reviseByInstruction/applyFixes, a conservative font-size clamp shrinks any flat text whose estimated wrap would overflow its box (same 60%/8px floor as the renderer's own shrink-to-fit, logged when it engages), and a deterministic overlap guard separates text-on-text collisions and re-clamps text spilling outside its containing shape (nudge/shrink with a logged degradation note — it never throws). Uses @reaatech/structured-repair-core to coerce raw LLM ops and falls back to a safe layout.
  • Vision Critic — holistic contact-sheet critique with tiered escalation to full-res per-output review; also interprets reference/brand images. Reviews the original before variants are seeded (the beauty gate: up to MAX_QUALITY_PASSES = 3 passes, the last one critique-only; a variant still flagged for an aesthetic criterion is held back unless it is the only result), then reviews each seeded format variant separately (up to two critique→fix passes per format). Every skill's rubric gets base criteria appended centrally: text_fit (no text overflows its band or the canvas edges), no_baked_in_text, no_framed_imagery, feed_legibility (each output is listed with its concrete 25% feed-scale pixel size), text_accuracy (checked against the expected copy per slot, passed as plans[].texts), and text_alignment (text centered within its containing shape, no text-on-text collisions, scrim over busy regions).
• Style presets (styles/): a registry of 8 presets — bold, editorial, minimal, neon, retro, neobrutalism, corporate, refined — each with palette sets, curated font pairings (families must come from the renderer's curated catalog), typeScale, and treatments. The start form has a Style select ("Let AI decide" default); unpicked, the Art Director varies styles across variants.
• Conductor (ai-designer-conductor.service.ts) runs a deterministic first-design pipeline and an agentic revise loop. The plan message is multi-select with accept-all as the default, and each plan's copy slots render as inline-editable inputs — the user fixes the headline/CTA/badge text before approving, and accept:plan carries the edits (validated server-side against the stored plans: known variantIds and copy-slot ids only, bounded, guardrail-checked) so execution renders exactly the approved copy; every accepted variant executes (plans are never padded with identical duplicates) — the per-variant compose, quality and format-expansion loops run with bounded concurrency (VARIANT_CONCURRENCY = 3, order-preserving, per-variant failure isolation) — plan-stage revise works from awaiting_plan and merges activeDesignIds. Backgrounds and focal-point imagery cover-crop per output instead of warping, and asset/composer degradation notes are surfaced to the user in a single chat message. Every agent dispatch carries a BudgetService check, a per-dispatch timeout (AI_DESIGNER_AGENT_TIMEOUT_MS, default 120s), and a per-(org, agent) circuit breaker that half-opens after 60s with a 10-minute failure-count window (one tenant's broken provider never disables other orgs, and stale failure counts decay/prune). Plans shown to the user are persisted on the session brief (lastPlans) so accept executes exactly what was presented; server-owned brief keys (lastPlans, lastDeliveredDesignIds, skillId, pendingReviseTarget, questionsAsked, referenceCues) are stripped from form:submit values before the brief merge (conductor/brief-values.ts, which also drops delivery-form control values like action/variantId so they never pollute the persisted brief), the executed plan list is clamped to the session's requested variants (≤10) regardless of what the stored brief contains, plan-requested asset generation is capped at 8 deduped asset needs per run, vision-critique findings are capped at 10 per pass (plans/critiques are LLM-shaped JSON — the caps bound the image-generation and note-fix LLM fan-out; typed fixes without a target slot are skipped rather than applied output-wide), and the persisted brief is bounded (64 KB serialized, sliding questionsAsked window). A per-session mutex rejects concurrent pipelines across all phases (intake/planning as well as accept/revise); cancel aborts the in-flight pipeline via a per-session AbortController (checked at every step boundary and raced against in-flight dispatches — a cancel never trips the circuit breaker, and a cancelled planning run can never write awaiting_plan afterwards). accept:plan executes only from awaiting_plan, a vision-critique failure degrades to delivering the un-critiqued variants (rendered work is never rolled back), and other failures roll the session back to a recoverable state with a sanitized chat message. The composer returns docs without persisting — AiDesignerSaverService is the single Design writer (revise re-saves update the same Design row rather than orphaning one per fix pass). A "change the background" revise on an image-backed design is classified kind: 'imagery' and regenerates the background image (the plan's image background, else the primary output's full-bleed hero image element) through the asset agent rather than composer doc ops; a revise whose ops are all filtered out tells the user the change is unsupported instead of re-delivering an unchanged design.
• Delivery: rendered previews are written to /files via AiDesignerSaverService (the contact sheet is a transient QC artifact for the vision critic — written to storage but deliberately NOT persisted as a File row, so it never appears in the org's Files library); Design rows are created and can be promoted to DesignTemplate. The target folder is resolved once per run: a client-supplied saveFolderId counts only when the folder belongs to the org, else the config's savePath (e.g. /campaigns/summer-launch) is resolved find-or-create per segment via FileService.resolveFolderPath, else the /files root. Delivery is conversational — there is no delivery form: the media message (with an Open in Designer handoff to /media/designer?designId=<id> per item) is followed by a plain prompt to say what to change or "looks good" to finish. A whole-message accept phrase ("looks good", "perfect", "done", …, optionally with the "no template" / "don't save" opt-out clause) is matched deterministically before the conversationalist dispatch — an instruction-bearing message ("looks good but make it darker") always goes to the classifier. Chat accept auto-saves each design of the latest delivery as a reusable template (server-owned brief.lastDeliveredDesignIds, set where delivery completes — superseded revisions stay active but are not re-saved); any other free text in delivered/revising is classified by the conversationalist and applied as a revision, honoring an extracted target design ("…on variant 2").
• Live progress: every phase transition emits a live agent:progress event (persisted progress rows survive reload; the frontend also shows an instant "Thinking…" bubble the moment the user sends anything, and a Cancel button inside the progress bubble while a run is in flight — it emits the socket cancel event and the conductor rolls the session back to its pre-run state). The pipeline composes one original in the primary format per accepted plan, then _expandVariants seeds each remaining format via addOutput (auto-seeded from the original, shared originIds), re-resolves per-aspect assets, and runs up to two per-format vision-critic passes (applyFixes pinned to that format) — all persisted to the same Design row.