Designer

Verified against v1.0.0

The Designer is the native canvas design editor at /media/designer. It is built on react-konva (Konva.js) and reads input from and writes output to the Files library; it never stores its own assets outside /files.

DesignerDoc contract (server-authoritative)

The DesignerDoc JSON is the agent interface, and the server owns it. There is one zod-based source of truth in libraries/nestjs-libraries/src/media/designer-doc/designer-doc.schema.ts; the frontend store and the server renderer both import type from it, while the dependency-free runtime helpers (migrateDoc, createBlankDoc, limits) live in designer-doc.migrate.ts and stay out of the client bundle. The contract discriminates on mode (image ⇒ outputs: DesignerOutput[], video ⇒ outputs: VideoOutput[]) and carries version + migrateDoc for forward-compatible upgrades.

DesignerDocService provides validate (lenient, clamps out-of-shape docs), validateStrict (rejects unknown keys), applyOps (mode-aware document transforms), and assignIdsAndNormalize (CSPRNG id minting across children/tracks/clips). DesignService validates every persisted design and template, reconciles Design.width/height from doc.outputs[0], and exposes instantiateTemplate and placeAsset for agent use.

Frontend architecture

All components live under apps/frontend/src/components/media-tools/designer/.

• designer.store.ts — a per-mount Zustand store created by createDesignerStore(w, h, attribution). No module-level singleton (it resets on unmount). The document model is:
  • DesignerDoc { version, width, height, pages: DesignerPage[], attribution?, durationMs? }
  • DesignerPage { id, background, bg?: DesignerBackground, children: DesignerElement[] }
  • DesignerElement — text | image | shape with geometry, opacity, locked, hidden, groupId, flipX/Y, crop, text styling + textShadow/textStroke, fillGradient, and an optional entrance animation.
  • State also tracks selectedIds, currentPage, zoom, undo/redo history, clipboard, and previewTime (animation playback clock).
• canvas.tsx — the Konva Stage. Handles multi-select (shift/⌘ + marquee), group-aware selection, snapping/alignment guides, a custom Transformer with a dimension HUD, wheel-zoom, space-drag pan, the keyboard-shortcut matrix, and drag-and-drop drops from panels (payload key application/x-designer-element).
• elements.tsx — element renderers (text/image/shape), gradient fills, crop, flip, and entrance-animation interpolation driven by previewTime.
• Panels (panels/) — Templates, Text, Elements, Icons, Photos, Uploads, Background, Layers, AI (gated on an active org AI provider via useAiActive), Brand, plus the selection Inspector.
• controls/ — reusable control primitives (color swatch, slider, segmented control, stepper, font-preview picker). fonts.ts — curated OFL fonts + ensureFontLoaded.
• timeline.tsx — per-element entrance animations with live preview and WebM export via MediaRecorder + canvas.captureStream (no ffmpeg dependency).
• export-dialog.tsx — PNG / JPEG / transparent-PNG, high-res pixelRatio, multi-page carousel export, and "Save & Post". Reuses SaveToFilesModal for the destination folder.

Cross-origin canvas

Konva's toBlob/toDataURL taints on cross-origin images. Element images load with crossOrigin="anonymous"; for object storage you must enable CORS (see the operations storage guide) or route through the same-origin image proxy (GET /media/designer/proxy).

Backend

Layering is Controller → Service → Repository (only repositories touch Prisma).

• Models: Design, DesignTemplate (additive, nullable/defaulted — db-push-safe).
• CRUD: DesignController / DesignTemplateController — /media/designs, /media/design-templates. DesignerProxyController — GET /media/designer/proxy (org-bound, safeFetch, fail-closed on non-image, size-capped).
• Server-side render (DesignRenderService, node-canvas): POST /media/designs/render → PNG or PDF (pdfkit). Text blocks auto-fit and honor verticalAlign; a curated-font load failure retries after a 15-minute TTL (font-loader.service.ts) instead of blacklisting the family for the process. Bulk generation (DesignBulkService): POST /media/designs/bulk-generate substitutes per row and renders a batch. Both endpoints validate body.doc before rendering.
• Video render (VideoRenderService): POST /media/designs/render-video enqueues a timeline render; GET /media/designs/render-video/:jobId returns status/artifact. Gated on media:create and video-exports:create.
• Document ops / validation: POST /media/designs/validate (lenient, media:read) returns { valid, errors? }; POST /media/designs/apply-ops (strict, media:create) returns { doc } after applying a strict-parsed op sequence. Two op behaviors changed with the AI Designer release: updateElement accepts an optional scope ('shared' propagates the patch to linked elements across outputs, 'format-only'/absent keeps the previous single-element behavior), and addOutput seeds the new output from the primary output's children (reflowed; originId backfilled on the primary) instead of appending an empty white canvas — matching the manual Designer's linked-by-default behavior. addOutput also accepts an optional seed: false to keep the pre-change empty-canvas behavior, and appends an unseeded canvas when the doc has no outputs at that point in the sequence (e.g. after removeOutput).
• AI ops (AiUserController, throttled, org-authenticated, credit-checked via AiMediaService): POST /ai/media with operations remove-background, inpaint, upscale, image, video, audio, avatar, bg-remove, tts, stt, alt-text. These delegate to AiMediaService, which routes through the per-org Media provider system.
• Agent seam: /copilot/agent now carries the acting user in the Mastra requestContext so user-attributed tools can fill createdById. The designerDesign Mastra tool (libraries/nestjs-libraries/src/chat/tools/designer.design.tool.ts) creates/updates designs from a DesignerDoc, template, or op sequence and persists an image preview when mode === 'image'.
• No migration required. Design.doc and DesignTemplate.doc are already Prisma Json fields; the new schema is enforced at the service boundary.
• Brand kit: AIBrandProfile.logoFileIds / palette / fontFamilies are read/written through the brand profile API.

AI Designer chatbot (/media/ai-designer)

The AI Designer is a chat-first, server-rendered design assistant built on top of the Designer foundations. It is image-only in this release; video is out of scope.

• Realtime transport: bespoke Socket.IO namespace /ai-designer registered in apps/backend/src/main.ts. Cookie JWT + CSRF handshake, session rooms (session:<id>), monotonic seq ordering, client ack, and Redis-backed nonce idempotency for every mutating event (start/message/form:submit/accept:plan/revise); a nonce consumed by a rejected operation (budget/guardrail/limit) — or by a handler that fails unexpectedly, which also emits internal_error with the nonce — is released so a retry with the same nonce isn't locked out for the TTL. The gateway also enforces per-user rate limits on every mutating event (buckets keyed by user id, so opening fresh sockets doesn't reset the window) plus a per-IP budget on connection attempts (checked before any JWT/DB work — neither the HTTP throttler nor per-event limits cover the handshake itself — and keyed on the transport address: the forgeable X-Forwarded-For header is never trusted, matching the HTTP throttler's posture), caps stored sessions at 100 per (org, user), re-checks membership/RBAC/billing every 60s for long-lived sockets (the NOT_SECURED dev toggle bypasses only CSRF, matching HTTP), validates session ownership before any handler-side write, bounds form:submit values (32 KB serialized / 5 levels deep), and runs all user free-text — chat messages, form values (strings at any nesting depth), and revise instructions — through the org's input guardrail chain (@reaatech/guardrail-chain via the governance GuardrailService) exactly once, at the gateway, before it is persisted or dispatched. Errors are emitted as { code, message, nonce? }; progress as { kind: 'progress', agent, phase, pct, note }; user-message echoes carry the client nonce for optimistic reconciliation. Deployment note: the gateway uses Socket.IO's in-memory adapter and the conductor keeps its pipeline mutex / prompt correlation / circuit breakers in-process — run the backend as a single instance (or behind sticky sessions) until a Redis adapter + shared stores are added. Setting COLLAB_SINGLE_INSTANCE=false without COLLAB_REDIS_ADAPTER emits a startup warning for both /collaboration and /ai-designer.
  • Reverse proxies: by default (TRUST_PROXY_HOPS unset) the connect-rate bucket keys on the socket peer address, so all clients behind a reverse proxy share one bucket. Set TRUST_PROXY_HOPS to the exact number of XFF-appending proxies to key the bucket on the Nth-from-right entry of X-Forwarded-For instead — overestimating lands in attacker-controlled left-most XFF entries and makes the buckets spoofable. The same setting also drives the HTTP throttler's per-IP buckets (login/register/enterprise/public-report) and the MCP rate limits.
  • Stuck sessions: if a session is in planning or executing and untouched for longer than AI_DESIGNER_STUCK_SESSION_MINUTES (default 15), reconnecting rolls it back to awaiting_plan so the user can retry.
• Session model: AiDesignerSession + AiDesignerMessage (additive Prisma tables). State machine: intake → planning → awaiting_plan → executing → delivered → revising. Sessions are pruned by the daily retention-purge cron after AI_DESIGNER_SESSION_RETENTION_DAYS (default 90; messages cascade), and users can delete their own via DELETE /ai-designer/sessions/:id. deleteUser tears sessions down explicitly (the userId FK is RESTRICT).
• Agents: six in-process agents registered via agent-mesh v-next (libraries/nestjs-libraries/src/ai-designer/agent-mesh/). The agent registry is bundled TS data (agent-registry.data.ts, validated against AgentRegistrySchema at boot) — not a YAML asset, which would not survive nest build — with AI_DESIGNER_AGENT_REGISTRY as an optional directory-of-YAML override (mapped to the package's AGENT_REGISTRY_DIR at import time). The mesh session/breaker stores default to Redis; AI_DESIGNER_MESH_STORE=postgres opts into the package's Postgres stores (runs its own DDL on a second connection pool — deliberately not the default, and it requires a dedicated AI_DESIGNER_MESH_DATABASE_URL: never the Prisma DATABASE_URL, where third-party DDL would fail the CI schema-drift gate). agent-mesh-env.shim.ts (first import in every agent-mesh-importing file) tames the package's import-time env handling: it seeds placeholder GOOGLE_CLOUD_PROJECT/API_KEY (Postmill uses only the in-process transport; the package's LLM classifier is never invoked), forces the package's global circuit breaker off (it is keyed by agent id only, so one tenant's provider failures would open it for every org — the conductor's per-(org, agent) breaker is the only breaker; the flag is z.coerce.boolean(), so the empty string is the only disabling value), and stash-and-restores unrelated env values the package's strict schema would otherwise process.exit(1) the backend on (e.g. LOG_LEVEL=verbose, NODE_ENV=staging, a malformed OTEL_EXPORTER_OTLP_ENDPOINT). The restore is synchronous with zero exposure window: agent-mesh-env.stash.ts stashes, the shim then imports the mesh package (whose env parse runs at import), and restores in the same tick — no other module ever evaluates with the stashed vars deleted. Mesh module setup is non-fatal: a failure degrades AI Designer with a logged error, never blocks backend boot.
  • Conversationalist — elicits intent conversationally (extracting brief fields every turn; intent must keep the user's concrete specifics verbatim — event names, offers, schedules — and verbatim copy like coupon codes lands in fixedCopy) and parses natural-language revisions. Once the brief is complete it recaps what it understood (including fixedCopy) exactly once and waits for an explicit green light — the recap is enforced deterministically via the server-owned brief.recapShown flag (not by the classifier), so planning never starts before the user confirms the recap.
  • Art Director — routes the brief through a skill registry (meme, advertisement, greeting-card, product-promo, announcement); each skill carries layoutHints (formatTemplates + slotSchema), richer prompts, and requiredBriefFields wired into intake, and a low-confidence route asks the user to pick a skill (brief.preferredSkill) instead of planning silently. Emits plan-schema-v2 DesignPlan[] variants — plan/slot styleId, slot kinds (text/image/cta-button/badge/accent-shape), and per-slot texts (the actual headline/subhead/CTA/badge copy, written to the skill's copy rules and naming the brief's real event/offer; brief.fixedCopy appears verbatim) — varying style presets across variants unless the user picked one. Plans describe the primary format only (first selected channel, else first custom size); other formats are adapted later by the conductor, not planned. Custom sizes are keyed uniformly as custom-${w}x${h}, and a custom-sizes-only start is allowed.
  • Copywriter — fills text slots per plan/brand voice. Slots the user approved on the plan card arrive as lockedTexts: returned verbatim and never rewritten; only open slots are written (all-locked skips the model call entirely).
  • Asset — generates/stocks imagery per output aspect (slotId:aspect keys; square/wide/tall size mappings for gpt-image and replicate/flux, stock orientation passthrough, layout-aware prompt suffixes that reserve space for copy); every generated-image prompt also carries a "no text, no words, no letters, no typography, no watermark, no logo" suffix so assets never bake in copy (stock searches don't get it); degrades to a generated gradient fallback.
  • Composer — builds validated DesignerDocOp[] using server-side seedCopy/smartReflow/applyLinked; honors the plan's style preset (palette, curated font pairing, typeScale, treatments) and brand fonts through a 6-template layout gallery (hero-fullbleed, split-panel, top-bottom, badge-burst, editorial-sidebar, minimal-centered; older ids alias in) and renders CTA slots as shape+text buttons. Imagery is always edge-to-edge (full-bleed or full-width/full-height bands — minimal-centered renders its image as a top band, never a floating framed inset). Per-role type floors keep copy legible at feed-thumbnail size (headline ≥ 6%, subhead/body ≥ 3.2%, badge/CTA ≥ 2.8% of min(w,h), clamped up in _typeScalePx), and badge/burst labels auto-fit inside their shape's inner safe area (centered both ways, never spilling). It composes the primary format only; the conductor then seeds each remaining format with addOutput (linked to the original via shared originIds) and re-resolves per-aspect assets (applyPerOutputAssets). applyFixes accepts an optional targetOutputs list that pins every fix to one format and forces format-only scope, so per-format critique fixes never propagate onto other outputs. Its revise-op vocabulary covers fontFamily/align/verticalAlign/textStroke/textShadow/addElement/removeElement/setOutputBackground (the revise prompt summarizes each output's current background, and the op replaces an image background when the instruction asks for a color/gradient), and a shared-scope fontSize fix is scaled proportionally per output. After compose and after reviseByInstruction/applyFixes, a conservative font-size clamp shrinks any flat text whose estimated wrap would overflow its box (same 60%/8px floor as the renderer's own shrink-to-fit, logged when it engages), and a deterministic overlap guard separates text-on-text collisions and re-clamps text spilling outside its containing shape (nudge/shrink with a logged degradation note — it never throws). Uses @reaatech/structured-repair-core to coerce raw LLM ops and falls back to a safe layout.
  • Vision Critic — holistic contact-sheet critique with tiered escalation to full-res per-output review; also interprets reference/brand images. Reviews the original before variants are seeded, then reviews each seeded format variant separately (up to two critique→fix passes per variant). Every skill's rubric gets base criteria appended centrally: text_fit (no text overflows its band or the canvas edges), no_baked_in_text, no_framed_imagery, feed_legibility (each output is listed with its concrete 25% feed-scale pixel size), text_accuracy (checked against the expected copy per slot, passed as plans[].texts), and text_alignment (text centered within its containing shape, no text-on-text collisions, scrim over busy regions).
• Style presets (styles/): a registry of 8 presets — bold, editorial, minimal, neon, retro, neobrutalism, corporate, refined — each with palette sets, curated font pairings (families must come from the renderer's curated catalog), typeScale, and treatments. The start form has a Style select ("Let AI decide" default); unpicked, the Art Director varies styles across variants.
• Conductor (ai-designer-conductor.service.ts) runs a deterministic first-design pipeline and an agentic revise loop. The plan message is multi-select with accept-all as the default, and each plan's copy slots render as inline-editable inputs — the user fixes the headline/CTA/badge text before approving, and accept:plan carries the edits (validated server-side against the stored plans: known variantIds and copy-slot ids only, bounded, guardrail-checked) so execution renders exactly the approved copy; every accepted variant executes (plans are never padded with identical duplicates), plan-stage revise works from awaiting_plan and merges activeDesignIds. Backgrounds and focal-point imagery cover-crop per output instead of warping, and asset/composer degradation notes are surfaced to the user in a single chat message. Every agent dispatch carries a BudgetService check, a per-dispatch timeout (AI_DESIGNER_AGENT_TIMEOUT_MS, default 120s), and a per-(org, agent) circuit breaker that half-opens after 60s with a 10-minute failure-count window (one tenant's broken provider never disables other orgs, and stale failure counts decay/prune). Plans shown to the user are persisted on the session brief (lastPlans) so accept executes exactly what was presented; server-owned brief keys (lastPlans, lastDeliveredDesignIds, skillId, pendingReviseTarget, questionsAsked, referenceCues) are stripped from form:submit values before the brief merge (conductor/brief-values.ts, which also drops delivery-form control values like action/variantId so they never pollute the persisted brief), the executed plan list is clamped to the session's requested variants (≤10) regardless of what the stored brief contains, plan-requested asset generation is capped at 8 deduped asset needs per run, vision-critique findings are capped at 10 per pass (plans/critiques are LLM-shaped JSON — the caps bound the image-generation and note-fix LLM fan-out; typed fixes without a target slot are skipped rather than applied output-wide), and the persisted brief is bounded (64 KB serialized, sliding questionsAsked window). A per-session mutex rejects concurrent pipelines across all phases (intake/planning as well as accept/revise); cancel aborts the in-flight pipeline via a per-session AbortController (checked at every step boundary and raced against in-flight dispatches — a cancel never trips the circuit breaker, and a cancelled planning run can never write awaiting_plan afterwards). accept:plan executes only from awaiting_plan, a vision-critique failure degrades to delivering the un-critiqued variants (rendered work is never rolled back), and other failures roll the session back to a recoverable state with a sanitized chat message. The composer returns docs without persisting — AiDesignerSaverService is the single Design writer (revise re-saves update the same Design row rather than orphaning one per fix pass).
• Delivery: rendered previews are written to /files via AiDesignerSaverService (the contact sheet is a transient QC artifact for the vision critic — written to storage but deliberately NOT persisted as a File row, so it never appears in the org's Files library); Design rows are created and can be promoted to DesignTemplate. The target folder is resolved once per run: a client-supplied saveFolderId counts only when the folder belongs to the org, else the config's savePath (e.g. /campaigns/summer-launch) is resolved find-or-create per segment via FileService.resolveFolderPath, else the /files root. Delivery is conversational — there is no delivery form: the media message (with an Open in Designer handoff to /media/designer?designId=<id> per item) is followed by a plain prompt to say what to change or "looks good" to finish. A whole-message accept phrase ("looks good", "perfect", "done", …, optionally with the "no template" / "don't save" opt-out clause) is matched deterministically before the conversationalist dispatch — an instruction-bearing message ("looks good but make it darker") always goes to the classifier. Chat accept auto-saves each design of the latest delivery as a reusable template (server-owned brief.lastDeliveredDesignIds, set where delivery completes — superseded revisions stay active but are not re-saved); any other free text in delivered/revising is classified by the conversationalist and applied as a revision, honoring an extracted target design ("…on variant 2").
• Live progress: every phase transition emits a live agent:progress event (persisted progress rows survive reload; the frontend also shows an instant "Thinking…" bubble the moment the user sends anything, and a Cancel button inside the progress bubble while a run is in flight — it emits the socket cancel event and the conductor rolls the session back to its pre-run state). The pipeline composes one original in the primary format per accepted plan, then _expandVariants seeds each remaining format via addOutput (auto-seeded from the original, shared originIds), re-resolves per-aspect assets, and runs up to two per-format vision-critic passes (applyFixes pinned to that format) — all persisted to the same Design row.

Not yet implemented

• Real-time multi-user collaboration — deferred. Requires adding a WebSocket platform (@nestjs/websockets + an adapter) and a CRDT layer (Yjs) for conflict-free editing. Tracked in dev/MEDIA_PHASE_2.md.